Refactoring All The Parts

[ninewise]

Current Setup

We don't want to stop anyone from participating in this discussion, so I'll start by sketching the current setup of Count-Von-Count and Boxxy. After this, I'll propose a new setup, roughly created by @iasoon and myself during the night after 12UL 2015.

Schema of current setup

 .-------.                                           .--------------.
 | Gyrid |>----.                            .------->| Boxxy Client |
 '-------'     |                            |        '--------------'
               |                            ^
 .-------.     |  .-------.    .--------------.      .--------------.
 | Gyrid |>--. '->|       |>-->| Boxxy Server |>---->|     ...      |
 '-------'   '--->| Count |    '--------------'      '--------------'
                  |  Von  |
 .-------.   .--->| Count |    .--------------.
 |  ...  |>--' .->|       |>-->|     ...      |
 '-------'     |  '-------'    '--------------'
               |
 .-------.     |
 | Gyrid |>----'
 '-------'

Current Components (and evaluation)

Gyrid

Gyrid is a python program, developped and maintained by the Geography department of UGent. It runs on small Arduino machines, mainly used because we can use the ones the Geography owns for free. Note that we call the machines (and not just the program) Gyrids.

To detect runners, or rather the bluetooth chips they unknowingly carry arround, we place several of these Gyrids along the circuit. When powered on, they start sending messages to a configured IP on a configured gate.

Where to look for details?

When you've acquired a gyrid, you'll want to look at:

• /etc/gyrid/gyrid.conf for the configuration (set the CVC address and port with network_server_host and network_server_port).
• /usr/share/gyrid/ for the launch scripts (executed on boot) and some python middleware code.
• /usr/lib/pymodules/python2.7/ contains a gyrid directory with the installed gyrid library and a package information file.

When you don't have a gyrid, the (no longer updated and entirely updated) git repository can be found here. Perhaps we should put the recent code online somewhere ourselves... (#114)

Gyrid to Count Von Count

Before 2014

A CSV-stream was sent through a socket. Part of one such a stream (generated by the Gyrid simulator):

00:24:01:eb:c2:68,_,20:11:02:15:01:77,0
00:24:01:eb:c2:68,_,20:13:01:24:00:28,0
00:24:01:eb:c2:68,_,20:11:02:15:01:77,0
00:24:01:eb:c2:68,_,20:13:01:24:00:28,0
00:24:01:eb:c2:68,_,20:11:02:15:01:77,0
00:24:01:eb:c2:68,_,20:11:02:15:01:93,0
00:24:01:eb:c2:68,_,00:12:02:01:07:82,0
00:24:01:eb:c2:68,_,20:13:01:24:00:28,0
00:24:01:eb:c2:68,_,20:11:02:15:01:77,0
00:24:01:eb:c2:68,_,20:11:02:15:01:93,0

Part of a real stream (as recorded during the 2013 run):

00:24:01:62:6C:B3,14,20:11:02:15:01:67,14,-69.0
00:24:01:62:6C:B3,12,20:13:01:24:00:28,12,-79.0
00:24:01:62:6C:B3,12,20:13:01:24:00:28,12,-76.0
00:24:01:62:6C:B3,1,20:13:01:24:11:15,1,-80.0
00:24:01:62:6C:B3,14,20:11:02:15:01:67,14,-68.0
00:24:01:62:6C:B3,14,20:11:02:15:01:67,14,-71.0
00:24:01:62:6C:B3,12,20:13:01:24:00:28,12,-77.0
00:24:01:62:6C:B3,14,20:11:02:15:01:67,14,-74.0
00:24:01:62:6C:B3,12,20:13:01:24:00:28,12,-81.0
00:24:01:62:6C:B3,3,20:13:01:31:03:09,3,-80.0
00:24:01:62:6C:B3,15,20:11:02:15:01:42,15,-73.0
00:24:01:62:6C:B3,11,20:13:02:20:16:95,11,-80.0
00:24:01:62:6C:B3,2,20:13:01:30:03:69,2,-76.0
00:24:01:62:6C:B3,15,20:11:02:15:01:42,15,-73.0
00:24:01:62:6C:B3,5,20:13:01:24:10:46,5,-76.0
00:24:01:62:6C:B3,3,20:13:01:31:03:09,3,-72.0
00:24:01:62:6C:B3,3,20:13:01:31:03:09,3,-79.0

What we see here are lines containing in order:

• The (bluetooth) mac address of the gyrid.
• A small positive integer we ignore entirely, of which the purpose is unknown to me, and is not even included in the simulator.
• The (bluetooth) max address of the passing chip.
• The RSSI value, a small negative double indicating the strength of the bluetooth-signal. Also not used in the simulator.

Since 2014

The Protocol Buffers library was used. This greatly complicated the communication over the same socket. Where before, the gyrids just started sending data, we now have to configure the messages we want to receive upon socket creation. I'll leave you to reading the code to find out the details.

Count Von Count

CVC receives the updates from all these gyrids, and determines the position of runners and the number of laps they ran.

Current tasks:

• Parse gyrids protocols
• Combine events from all gyrids into one stream
• Interpret this stream to the lap counts and positions
• Provide a web-interface for adding laps, changing batons and monitoring.
• Forward counting decisions to the configured boxxy servers.
• Write a replay log, so the input of CVC can be replayed.

Count Von Count to Boxxy Servers

Count Von Count sends HTTP Put requests with JSON bodies to currently 4 paths on the boxxy server. Examples:

PUT /ping HTTP/1.1
{}

Just to see if the boxxy server is up. The body is empty.

PUT /state HTTP/1.1
{ 'laps': [ { 'timestamp': '2015-04-06T21:53:21.029Z'
            , 'count': 1
            , 'reason': None
            , 'team': '1'
            , 'id': '2615'
            , 'total': 154
            }
          , {'timestamp': '2015-04-06T21:53:20.030Z'
            , 'count': 1
            , 'reason': None
            , 'team': '15'
            , 'id': '2614'
            , 'total': 157
            }
          , ... // SOme more recent laps
          ]
, 'teams': { '11': { 'station': None
                   , 'laps': 159
                   , 'name': 'Politeia'
                   , 'id': '11'
                   , 'updated': None
                   }
           , '1': { 'station': None
                  , 'laps': 154
                  , 'name': 'KVHV'
                  , 'id': '1'
                  , 'updated': None
                  }
           , ... // One more for each team.
           }
, 'stations': { '1': { 'name': 'gyrid1'
                     , 'id': '1'
                     , 'position': 45.0
                     }
              , '2': { 'name': 'gyrid2'
                     , 'id': '2'
                     , 'position': 120.0
                     }
              , ... // And more for each gyrid
              }
}

The state of the race. Count Von Count sends this when it detects the boxxy server going up.

PUT /position HTTP/1.1
{ 'station': '2'
, 'timestamp': '2015-04-23T23:27:48.145Z'
, 'team': '10'
}

Put by CVC when it determines a team has gone to a new position.

PUT /lap HTTP/1.1
{ 'timestamp': '2015-04-23T23:32:10.977Z'
, 'count': 1
, 'reason': None
, 'team': '7'
, 'id': '2618'
, 'total': 156
}

Put when CVC decides a team has completed a lap.

Boxxy

A boxxy server is a CVC client, and a server for boxxy clients. The boxxy server receives the state/laps/positions from CVC, and forwards this to his clients. By letting the boxxy clients listen to the boxxy server, in stead of directly to CVC, we can limit the trafic on the local network at the 12UL.

A boxxy server uses, towards its clients, the same protocol as CVC uses towards the boxxy server. The main difference is that it sends the JSON data to each client which connects to the server with a websocket, while CVC PUTs the JSON's to statically configured boxxy servers.

Proposed new setup

We would propose to split the whole schema into more modules, as drawn in the following new schema:

                                  .-------.   .-------.
                                  | Gyrid |   | ..... |
                                  '-------'   '-------'
                                      v           v
                                      |           |
                                      v           v
                             .------------.   .------------.
                             | Translator |   | .......... |
                             '------------'   '------------'
                                      v           v
                                      |           |
                                      '-----O-----'
                                            |
                                            v
                                      .-1---------.
                                      | FilterMap |
                                      '-----------'
                                        v   v   v
                                        |   |   |
                    .---------O---------'   O   '---------O---------.
                    |                       |                       |
                    v                       v                       v
              .-2---------.           .-1---------.           .-2---------.
              | FilterMap |           |    Map    |           |    Map    |
              '-----------'           '-----------'           '-----------'
                    v                       v                       v
                    |                       |                       |
                    v                       v                       v
          .-1-----------------.   .-2-----------------.   .-3-----------------.
          |     Multiplex     |   |     Multiplex     |   |     Multiplex     |
          |         |         |   |         |         |   |         |         |
          |     .---'---.     |   |     .---'---.     |   |     .---'---.     |
          |     |       |     |   |     |       |     |   |     |       |     |
          | Arbitrator ...    |   | WatchDog   ...    |   | WatchDog   ...    |
          |     |       |     |   |     |       |     |   |     |       |     |
          |     '---.---'     |   |     '---.---'     |   |     '---.---'     |
          |         |         |   |         |         |   |         |         |
          '-------------------'   '-------------------'   '-------------------'
                    v                       v                       v
           .-----.  |                       |                       |
           |     |  |                       |                       |
           |     v  v                       |                       |
           |   .---------.                  |                       |
           |   |   Log   |                  |                       |
           |   '---------'                  |                       |
           |        v                       |                       |
           |        |                       |                       |
           |        v                       v                       v
           |      .---------------------------------------------------.
           |      |                Admin Web Interface                |
           |      '---------------------------------------------------'
           |        v                       v
           |        |                       |
           '--------'                       |
                                           ...

Fifo pipe

A O in the schema represents a fifo pipe. These are used to decouple
components. For instance, the FilterMap marked 1 can be terminated
and replaced with another, while the fifo's before and after it will
buffer.

Translator

This component ensures decoupling between the Gyrids (or whatever
we're listening to) and the rest of the network. The input protocol
doesn't matter for the network, but could be CSV of Protobufs. The
output consists of a line for each "detection". One line contains a
JSON object:

{
    when: timestamp,     # the time of the detection, as accurate as possible
    who: baton id,       # the id of the detected baton
    where: station id    # the id of the station where the baton was detected
}

FilterMap

Uses the configuration files to filter and map a JSON stream.

FilterMap 1

Output:

{
    when: timestamp,     # copy of `when` in input
    who: team id,        # the team the `who` in the input is assigned to
    where: station id    # copy of `where` in input
}

This map filters JSON input where the who does not have a mapping in
the configuration file.

FilterMap 2

Output:

{
    when: timestamp,     # copy of `when` in input
    who: team id,        # copy of `who` in input
    where: double        # part of the lap the detection took place, [0,1[
}

No filtering.

Map

Simple component to select part of the JSON attributes. One of the
maps will select the when and who, the other will select the
when and where.

Multiplex

The multiplex will be configured with a JSON attribute and a component
to run in subprocesses. For each value of the given attribute
encountered in the inputstream, a subprocesses will be started and
given all JSON input objects with that same value (with the configured
value filtered out).

Multiplex 1

Will multiplex its inputstream on the who attribute and thus start
an Arbitrator for each team. To the Arbitrator, a JSON of the
following structure will be given:

{
    when: timestamp,    # copy of `when` in input
    where: double       # copy of `where` in input
}

The output of the Arbitrator is joined with the who attribute from
the input of the Multiplex and outputted as actual output of the
Multiplex.

Multiplex 2 and 3

Will multiplex its inputstream on either the who or where
attribute, depending on which one was selected in the Map. This means
the WatchDog will receive JSON's of the following structure:

{
    when: timestamp    # copy of `when` in input
}

The output from the WatchDog will again be joined by the who or
where attribute from the input and sent to the output of the
Multiplex.

Arbitrator

The Arbitrator receives where and when its team was detected and
decides when a team has completed a lap. It outputs a JSON structured
as follows:

{
    when: timestamp    # when the lap was completed
}

WatchDog

The WatchDog receives when its target was seen and remembers when
it was last seen. Once this is longer past then the span it's
configured for, it outputs this last seen time as a warning.

{
    when: timestamp    # when the target was last seen
}

Log

Logs the decisions made by the Arbitrators and manual bonus laps made
through the admin web interface. This is used on the startup (or
restart) of the admin user interface to start/continue the counts.

Admin Web Interface

Monitor stuff and add bonus laps.

Also rewrite boxxy.

[iasoon]

I'm in favor of dropping the multiplexers. It's not really more complex to write a program that tracks multiple teams/batons/... compared to a program that only tracks one. That multiplexer, on the other hand, would be complex and error-prone.

How will we make sure the data streams won't break? What do we do if they do?