2nd March 2016

Capturing Cube Touch Data (TUIO) and Storage Using Splunk

Touch REST service visualisation example

The SensingSEC platform provides access to touch data that is captured from across all of the multi-touch screens that make up the QUT Cube installation (http://www.thecube.qut.edu.au). Each screen outputs touch data in TUIO format (Tangible User Interface Objects – http://www.tuio.org). The TUIO data is captured and archived using Splunk (https://en.wikipedia.org/wiki/Splunk), which is a platform that specialises in indexing massive amounts of data in a way to make it searchable and retrievable. TUIO touch events are produced at a very high rate (up to 100Hz for each finger), and the data sizes grow quickly with increasing simultaneous users. For this reason it was decided that Splunk was a logical storage solution for archiving such large amounts of data, while still having the data searchable (e.g. by date range).

TUIO is an open framework defining a common protocol and API for multi-touch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and other tangible object states. The protocol defines 3 distinct types of events:

cursor events are used to track finger touches. This is the most common event type, and the primary type considered by the SensingSEC platform. The fields captured from cursor events are as follows:

_time, screen_id, tuio_type, tuio_action, tuio_id, session_id, x, y, motion_speed, motion_accel

object events are for tracking fiducial markers. Object event fields:

_time, screen_id, tuio_type, tuio_action, tuio_id, session_id, x, y, angle, motion_speed, motion_accel, rotation_speed, rotation_accel

blob events are used to track generic untagged objects/shapes. Blob event fields:

_time, screen_id, tuio_type, tuio_action, tuio_id, session_id, x, y, angle, width, height, area, motion_speed, motion_accel, rotation_speed, rotation_accel

In order to log the data into Splunk, a Splunk Universal Forwarder is used, which is running on central machine and accessible by all touch machines over a network. The Splunk forwarder is a lightweight, dedicated version of Splunk that contains only the components required to forward data to a main Splunk indexer instance (for more information, see http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Introducingtheuniversalforwarder)

Each Cube machine controls 2 touch screens, and when users interact with these touch screens, TUIO touch data is generated for every finger touching the screen (touch on, touch release and touch movement). This data is used as input to control the various software applications running on the Cube, such as the Virtual Reef or The Cube Globe, for example. Each machine is running a custom process called the Daemon Tool which transparently intercepts the TUIO data stream, logs the data using Splunk, and then forwards the data onto the receiving application. The intercepted touch data is reformatted by the Daemon Tool into a comma-separated value (CSV) syslog message logging format (https://en.wikipedia.org/wiki/Syslog) and sent directly to the Splunk forwarder over a UDP network connection. This method allows the touch data to be intercepted, forwarded and archived without any expense to the performance of the main running touch application.

Example touch event syslog record produced by the Daemon Tool:

Jan 28 10:28:48 fe80::8071:b7ff:bd61:1f5a%12 2016-01-28T10:28:48.087+10:00,cursor,update,0,0,4,0.5887326,0.4824866,3.993752,-28.30107

Splunk Configuration

The Splunk forwarder is configured to listen on UDP port 514, which is the default port used for syslog messages. The configuration settings for the forwarder input are shown below:


inputs.conf

[udp://514]
index = sec_touch
source = tuio_syslog
sourcetype = sec_touch_tuio

The touch events are forwarded as raw syslog events to the central Splunk indexer (in a CSV format as shown above). In order to extract the individual column values to named fields, field extraction entries are setup in the Splunk indexer (via the online dashboard) which use regular expressions (regex) to extract the individual fields. This enables the fields to be individually searched and used in search filters and conditions.

For example,the cursor touch type requires the following regex entry to extract the fields:

(?cursor),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),(?[^,]+),?(?[^,]+),?(?[^,]+)

A separate field extraction entry is required for each of the three different touch types (cursor, object, blob).
After the touch data is forwarded to Splunk for indexing and the fields have been extracted, the data appears in the Splunk dashboard as shown below.

Splunk touch events

Example touch events as archived in Splunk and displayed in the dashboard, with extracted fields.

REST Queries for Accessing Touch Data:

An example REST query to retrieve touch data (cursor) from all zones, specifying a time range of 1 second:
http://sensingsec.qut.edu.au/api/touch/all/cursor?from=2016-01-27T15:00:01&to=2016-01-27T15:00:02


A subset of JSON data output from a REST query specifying a range of 1 second:

{
  "sources":[
    {
      "screens":[
        {
          "touches":[
            {
              "ts":"2016-01-27T15:00:01.576+10:00",
              "screen_id":5,
              "tuio_type":"cursor",
              "tuio_action":"add",
              "tuio_id":0,
              "session_id":1089342,
              "x":0.7202784,
              "y":0.13205,
              "speed":1.0,
              "accel":8.0
            },
            {
              "ts":"2016-01-27T15:00:01.593+10:00",
              "screen_id":5,
              "tuio_type":"cursor",
              "tuio_action":"update",
              "tuio_id":0,
              "session_id":1089342,
              "x":0.7202823,
              "y":0.1320703,
              "speed":0.00189533,
              "accel":0.1894099
            },
            {
              "ts":"2016-01-27T15:00:01.609+10:00",
              "screen_id":5,
              "tuio_type":"cursor",
              "tuio_action":"update",
              "tuio_id":0,
              "session_id":1089342,
              "x":0.7202967,
              "y":0.132114,
              "speed":0.004591069,
              "accel":0.2693987
            },
 
            ....
 
          ],
          "host":"131.181.25.13",
          "screen_id":5
        }
      ],
      "zone":"QUT-GP-P4-Cube-Zone1"
    }
  ],
  "from":"2016-01-27T15:00:01",
  "to":"2016-01-27T15:00:02"
}

Over just one second, there can be a large number of touch events available, depending on the amount of activity on the touch screens. In the case above, the JSON data has been reduced and ellipses “….” represent the data that has been left out for brevity.
Note, even though Splunk excels in ingesting and indexing massive amounts of data, we still encountered problems with certain queries and returning this data in a reasonable amount of time. This is due to the order in which Splunk performs the search and packages the results.

Unbounded queries can take minutes (and even longer) to complete, which is generally unacceptable. A simple solution is to restrict queries to relatively small time ranges (up to 30 minutes, for example), but a more permanent solution is being investigated that engages some more obscure Splunk search features. This will be discussed in a future post.

Main REST query format:

http://sensingsec.qut.edu.au/api/touch

http://sensingsec.qut.edu.au/api/touch/range

Example query variations:
All zones over all time – currently not recommended as too much data is returned – see note above

Visualisation

The image shown below is an interactive visualisation of the archived touch data, which is shown on the SensingSEC Explore page (http://sensingsec.qut.edu.au/Explore/Touch) as an example usage of the REST interface. After selecting a date/time range and zone on the form, clicking Search displays touch events on the panels below. In this example, each rectangle represents a touch screen on Zone 1 of The Cube, and the colored dots represent finger touches, with each new color representing a distinct finger interaction. This tool utilizes exclusively the REST interface, with the REST query string being constructed from the user specified form values, and the JSON data returned is then parsed to produce the interactive graphical representation.

Touch REST service visualisation example

An example interactive visualisation that harnesses the touch REST service.

- Allan James

Leave a Reply

Your email address will not be published. Required fields are marked *