Chipi - A (House) Automation for Common Lisp

!!! I’m running the core framework in production 24/7 for many months. But still considered beta. !!!

Motivation

I use home automation in my home. Currently utilizing openHAB. While openHAB has many features and supports many devices and so on, it is also a heavy beast.

So my aim was to eventually replace openHAB. There are a few things still missing to do this. The current state of the Chipi project lacks the following for me to be able to replace openHAB:

• support for KNX bus (finished)
• a REST interface (in progress, implemented, documentation missing)
• a UI to interact with the system

See plugin section below.

What does Chipi currently do

Chipi consists of the following components:

• item: items are a primitive to represent the temperatur of a sensor, the state of light in your house, the state of your window being open or closed, etc.
• itemgroup: items can be organized in groups
• binding: bindings are a means to update an items value or distribute a value change to somewhere
• persistence: persistences allow to persist and load the state of items. Currently two types of persistences are supported:
  • simple-persistence that just stores the current state to file and can retrieve it
  • historic persistence with a support for influxdb to store each changed item value in influxdb and also retrieve values for a specific time range to calculate averages, peaks or whatever.
• rules: rules are kind of scripts that are run on certain ‘triggers’. Triggers may be a change of an item value, or a cron timing.

How does it work

Chipi is largely based on Sento, a Common Lisp Actor Framework where features like thread-safety, queuing and an event-bus are very handy for Chipi.

How it works is to define and combine all the mentioned primitives in a file. This file is a Common Lisp source file. The definition of the primitives (item, binding, …) is done using a defined DSL (domain specific language) which largely consists of a few macros. But it is also possible to define your own macros, functions or variables.

A full production example that defines items, bindings, persistences and rules can be seen here. This script, very much based on Common Lisp, is a full example configuration of Chipi. At the end of the script evaluation of all items, persistences and rules will start their work.

This particular example uses Common Lisp libraries like py4cl (to talk to ina219 analog 2 digital converter for a pressure sensor in a cistern plugged to the GPIOs of a Raspberry PI 4), Drakma to query data of a Shelly power switch or cserial to communicate with some other device to retrieve temperatur sensor data.

But let’s go slowly

The top of the Chipi standalone script should include this (eval-when) in order to make sure that dependencies are loaded properly. But this is not mandatory if you find another way of loading dependencies.

(eval-when (:compile-toplevel :load-toplevel :execute)
  (ql:quickload :chipi))  ; at least this if you don't have more

Next thing is that you might want to define a separate package using defpackage, it allows to inherit symbols using :use or :import-from that you want to use in the script.

Defining the global environment

The first thing Chipi specific that should be defined is the ‘config’. This is done by (defconfig). This expression should be before items, persistences and rules because it defines and starts the underlying actor system and sets up some other necessary structures.

Defining persistences

Persistences are used to persist the value of an item and to retrieve it back. The influx persistence also allows to retrieve values in a certain time range.

Simple persistence

To define a ‘simple’ persistence you do:

(defpersistence :simple
  (lambda (id)
    (make-simple-persistence id :storage-root-path #P"some-path")))

The :simple here is the persistence id passed along to the lambda factory function.

The only information simple-persistence requires is where to store the item values. Each item is stored in its own file and just stores the last value. The value plus a timestamp (GET-UNIVERSAL-TIME) is stored in JSON format like this:

{"value":66.19999694824219,"timestamp":3907315980}

Under normal circumstances you don’t need to deal with simple persistences directly. To retrieve an item value you’d call ITEM:GET-VALUE. Though it is possible to call PERSP:FETCH and retrieve the current value from the persistence directly.

Influx persistence

A persistence that allows retrieving item values of a time range, i.e. to build daily or weekly average values, is influx. Currently only influx v2 is supported. Define such persistence like this:

(defpersistence :influx
    (lambda (id)
      (make-influx-persistence
       id
       :base-url "http://your-host:8086"
       :token "your-token"
       :org "your-org"
       :bucket "your-bucket")))

To retrieve a range of values you can call PERSP:FETCH with an optional range instance. Currently only RELATIVE-RANGE exists.

Bindings

Bindings are a means to interact with sources or targets, meaning they allow interactivity with the item value. Bindings are defined as part of an item definition and not on toplevel. A basic binding definition looke like this:

(binding :initial-delay 5
         :delay 60
         :pull (lambda () 0) ;;pull value from somewhere
         :transform (lambda (value) (1+ value))
         :push (lambda (value)) ;; push to somewhere else
         :call-push-p t)

This binding uses the pull function to retrieve a value, which is passed on to the item value. When to pull is determined by :initial-delay and :delay in seconds where the former is an ‘initial delay’ and the latter a repetetive delay. :call-push-p actually defines whether the push function is called when the value was updated. The push function can be used to push the value elsewhere if required. Both pull and push are optional. Though one of the two should be used, otherwise the binding doesn’t make much sense. What is transform? It is optional but can be used to transform the value retrieved with pull. transform should return a transformed value.

Thinking further, I’d like to have bindings that are specific to pulling from http, serial, or whatever, and allow to be specified in that way. The pull, push functions are very generic but may require repetition and are not enough specialized.

See next how to define and attach bindings on items.

(Also see below for bindings available as plugins.)

Defining item groups

(defitemgroup 'group1 "Group1")

This defines a group 'group1 with label “Group1”. See below for how to add items to groups.

Defining items

The simplest form to define an item is:

(defitem 'myitem "My Item" 'integer)

This defines a plain item that can hold a value. You could manually use SET-VALUE function to give it a value or GET-VALUE to retrieve its value. In some cases this is useful in ‘rules’. See later.

The three parameters define an id of the item (for easier lookup), a label and a type hint. The type hint is not necessary (can be specified as NIL) unless you want to use influx db where under the hoods it is necessary to bring the value in the right format based on what type the value is in. Checkout influx persistence for which types are supported. However, even if not required it might be a good idea to define the type for clarity.

Usually you’d want to at least define an initial value. You can do so by:

(defitem 'myitem "My Item" 'integer
  :initial-value 0)

Items can be defined for group membership. It can be done like so:

(defitem 'myitem "My Item" 'integer
  :initial-value 0
  :group 'group1)

Define and attach bindings

In many cases you want to retrieve the item value from somewhere and maybe also want to push it somewhere else once it was set. This is where bindings come in. There can be more binding definitions on an item but this only really makes sense if you plan to push to more places. An item definition with binding looks like this:

(defitem 'myitem "My Item" 'integer
  :initial-value 0
  (binding :initial-delay 0.1
           :delay 30
           :pull (lambda () (do-some-http-get))
           :push (lambda (value) (do-some-http-post))
           :call-push-p t))

Attaching persistences on the item definitions

Persistences, as defined above can now be ‘attached’ to the item like this:

(defitem 'myitem "My Item" 'integer
  :initial-value 0
  (binding :initial-delay 0.1
           :delay 30
           :pull (lambda () (do-some-http-get))
           :push (lambda (value) (do-some-http-post))
           :call-push-p t)
  :persistence '(:id :simple
                 :frequency :every-change
                 :load-on-start t)
  :persistence '(:id :influx
                 :frequency :every-20s))

It is possible to attach multiple. In the case above both have different purposes. The :simple persistence is used to just store the latest value and can recover from it when told so using :load-on-start.

The :influx persistence will just store every value change to the database.

The :frequency defines how often the value is stored. :every-change will store the value to the persistence on every change of the item value. :every-20s (form example) stored the value every 20 seconds recirring. The notation here is :every-N<s|m|h> where N is the number, s (seconds), m (minutes) and h (hours).

Defining rules

Rules are scripts that are run on certain triggers. Triggers are the change of one or more item values or one or more cron definitions. Example:

(defrule "My Rule"
  :when-item-change 'my-item
  :when-cron '(:minute 0 :hour 0)
  :do (lambda (trigger)
        (case (car trigger)
          (:item
           (let ((item (cdr trigger)))
             (format t "Item changed: ~a~%" item)
             ;; asynchronously do something with the value
             (future:fcompleted (item:get-value item)
                 (value)
               (do-some-action-with-value value))))
          (:cron
           (format t "Cron triggered: ~a~%" (cdr trigger))
           (do-some-action))))

Rules can be triggered by item value changes. To subscribe to certain items one has to use :when-item-change with the item id of the item definition. Use multiple :when-item-change to subscribe to multiple item changes.

The other trigger is cron. The lowest granularity is minutes. Specify cron triggers with :with-cron. Also multiple triggers can be defined. The cdr of the trigger variable is the cron expression.

When a cron trigger is specified as ’(:boot-only t) then this means that the rule is called immediately after initialization, but only once.

Redefining persistences, items and rules

Those elements can be redefined, meaning re-evaluated to update a changed configuration. The re-evaluation usually happens in Emacs in the same way as functions are re-evaluated using C-c C-c or by reloading the whole script.

Caveeat: if a persistence is changed and re-evaluated also the items where it is attached have to be re-evaluated.

Logging

The project uses log4cl. You can change the log level for ’(chipi) to suite your needs of granularity. The underlying Sento actor framework will log extensively on :debug level so a good idea is to silence this via (log:config '(sento) :warn).

Available binding plugins

KNXNet/IP binding: see README