+ Development environment consisting of MS Visual Studio 11 (2012) and the Eyesweb XMI SDK (5.5.x).
+ Creation of blocks for use in the Eyesweb DE during the Gear Shift series of performances
held at Melbourne's Loop Bar throughout 2014.

From November 2014 a series of blocks are being developed that will eventually be released under a single .dll catalog file known as OSL_light with appropriate levels of datatype checks to allow use outside the current Object_State fixed performance requirements.

A performance patch has gone through several iterations and is at two stable and usable versions (7.x, 8.x),
however the development machine runs them at an FPS matching the input camera (25), while the performance laptop runs them at 3-6 FPS.

Several blocks are being implemented to simplify the signal path and reduce the overall number of blocks that the signal passes through. As the Eyesweb DE has gone through several major revisions since 3.3.x it has taken a while to get the build environment working and the new SDK understood.

The first block (BusMixer) is mixer for two video (IImage) datatypes that are matching in ColorModel, PixelModel and size. A parameter is present to provide a double value between 0.0 and 1.0 for a ScalarArithOpDouble operation that is performed on one source and is inverted for a similar operation on the other source. An IArithOp::Add is performed on the two changed image streams and then sent to the output.

A second block is called DiffFilter that subtracts a snapshot image from an internal buffer taken from the source in. This then has a threshold operation performed and then a final median filter. Most of the key variables are presented as parameter pins for changes during performance. The source video is internally converted to ecmBW if needed.

The third block (Ghost), also forcing to ecmBW if needed, alters an input image to create a ghosting effect using an internal bus system to loop the image stream and subtract in to black. A simple gain effect was added to increase the brightness of the image.

A fourth block called Flow uses a pixelwise comparator to split a single channel buffered image list of 3 frames size and colour it with values for red, blue and green. This block represents one of the last needed to replicate and simplify the latest Gear Shift performance patch. Additional blocks may be created to provide new effects for signal processing depending on time available prior to the next Gear Shift performance.

A fifth (BWConv) is a simple colour to black and white conversion block that was implemented mainly for the sake completeness in the Gear Shift patch. It takes the camera input that is ecmBGR and converts it to ecmBW for use in later signal processing that requires single channel images.

Sixth is a block (AudioBang) that uses an audio buffer to compare against a threshold value and output a trigger if it passes over it within a set time period. A simple bypass is in place to allow a frame rate based timer to run and send a trigger when a set range is reached.

DelayColor is the seventh block that takes a black and white source, copies to RGB buses with a delay based upon frame count if wanted and then combines into a BGR output via a colour channel mixer. A few memory leaks were encountered with the buffers but they were caught during the testing phase. Installing the OSL_Light catalog onto the performance laptop and running tests is next. Update: testing prior to next Gear Shift performance gets 7 fps...

Some notes from developing with the Eyesweb SDK: Somewhat of a lack of implementation documentation, some use of cut and paste code notations and the occasional use of Italian made it a bit difficult when accessing the interfaces made visible. Unfortunately some of the more interesting libraries seem as yet inaccessible, such as classes from the Intel Integrated Performance Primitives and OpenCV, though some access was provided by the EywSDK. The life-cycle of a block and the correct implementation for use within the Eyesweb Engine are still to be mastered but otherwise, the Eyesweb SDK provides a fascinating entry to computer vision.

A link to the github repo for the Object_State Light blocks (and Visual Studio sln) is found here: OSL repo


Test patch using OSL coded block for mixing two video streams into a final blended output.


Final Gear Shift 2014 performance patch using all current OSL coded blocks (in blue).


Final Gear Shift 2014 performance patch showing the controller and monitor section.