Skip to content

Real time API

Jump to bottom
Sofian Audry edited this page Apr 19, 2017 · 19 revisions

Rationale

Media artists work with real-time sensory signals all the time. However, few artists (especially beginners) really "know their signals" and how to extract high-level informations from them (such as debouncing, smoothing, normalizing, detecting peaks and regularities). There currently does not exist any simple, widely-adoptable way to do this. There is no accessible, readily-available source of information on the subject.

Consider the following case of learning how to work with a simple photoresistor sensor plugged into an Arduino board on analog pin 0. The code reads as follow:

int value = analogRead(0);

The value read is the raw 10-bits value returned by the ADC and thus reads as an integer between 0 and 1023. But how is that really useful for an artist who wants to use this value creatively. For example, what if one wants to react to a flash of light? Well, one solution is to look at the kind of values we get and set a threshold. For example:

if (value > 607)
  // do something

Nice. But there are two problems with this. First, while it might work under certain light conditions, it will likely stop working if these conditions change, forcing us to make adjustments by hand to the threshold value. Second, and perhaps more importantly, this piece of code does not really expressed what we are after. We don't want to know of the light is above "607": what we really want to know is whether it is significantly above average.

What this example shows is that the way we are teaching and learning about sensor data is ineffective for creative applications. In other words: raw digital data lacks expressiveness. Furthermore, it makes interoperability difficult, often even within a certain environment.

Continuing with our example, consider how one would take the input value and directly reroute it to an analog (PWM) output on pin 3:

analogWrite(3, value / 4);

Why do we need to perform that division by 4? That's because while the ADC gives us 10-bit values, the PWM only supports 8-bits. But again, why is this important to know for an artist? And what exactly does it have to do with our artistic intention, which is just to reroute the signal.

These cases are true not only for Arduino, but for most softwares, languages and platforms used by artists, such as Max/MSP, Processing, or Python. And while there certainly exist tools for some of them to process data better (such as signal processing libraries), these tools are often intended for either music practitioners or scientists. Furthermore, there lacks a common, cross-platform approach on how to "deal with data".

A new standard

As a way to address these issues, we propose to create a general-purpose standard interface for simple, real-time signal processing for media artists. The objectives are as follow:

  1. Allow artists to concentrate on the creative dimension of their work rather than on irrelevant numerical questions, hence also facilitating learning.
  2. Provide artists with simple, accessible, easy-to-understand tools that grasp high-level concepts such as "normalizing" and "detecting peaks" (rather than specific, arcane techniques on "how" to extract these informations such as "FFT", "zero-crossing" or "Chebyshev filtering").
  3. Facilitate team work and interoperability between applications by favouring an easily understandable, cross-platform way of thinking about real-time signals (for example by keeping all signals "in check" between 0 and 1).

The API will have the following characteristics:

  • Real-time (!)
  • Lo-fi (artists can tolerate low precision)
  • Robust (it needs to tolerate changes in the sensory context without breaking down, because media artworks are often presented in environments that are difficult to fully control)
  • Cross-platform (the same concepts and functions should be recognizable across many different tools artists use)
  • Easy to learn (concepts in the API should be chosen carefully to respond to common problems artists face ie. limited to only a few functions that will solve 95% of your problems)

Specs (in progress)

Datatype

  • The basic data unit is a 32-bit float (IEEE 754)
  • Typically the values are either:
    • in [0, 1]
    • in [-1, 1] (for audio-like data)
    • in [-inf,+inf] (centered at zero, for normalized data)

Functions

  • "Smoother": low-pass filtering (exponential moving average)
  • "Normalizer": normalize data to a N(0, 1) in [-inf, +inf]
    • two modes: exponential moving normalization (adaptive) OR standard (full)
  • "Thresholder": "triggers" (0/1) when value goes over or below a certain threshold (can be used for peak detection when combined with normalizer)
  • "Clamper"/"Constrainer": constrains (ie. "crops") value within a certain range (typically [0, 1] or [-1, 1])
  • "Rescaler": rescale/stretch value in [0, 1] based on min/max value
    • (x - min_x) / (max_x - min_x)
    • might be combined with a simple/small median or low-pass filter to prevent outliers

Some ideas (brainstorm)

Tentative list of functions:

  • Smoothing (low-pass filtering)
  • Debouncing (ie. 0/1 signals smoothing)
  • Normalizing
  • Detecting peaks
  • Detecting beats (ie. frequency + phase)
  • Power spectrum

Also, I was thinking one way to convey some of these concepts in a nutshell and encourage the cross-application aspects would be to create a "verb" based on the software name. For example, let's say the API is called "Bob" we could say "bobify your data". Example:

Emma: What's going on nothing is working!!!
Marc: Have you bobified your inputs?
Emma: Of course I have! Have you bobified your outputs?
Marc: ... ...
Emma: (frowning)
Marc: Shit.

References/inspirations

Clone this wiki locally