Real-time Interactive Tracks

From SenSei to EboSuite

We managed to integrate most concepts of SenSei, the audio-visual instrument I created with EboStudio for my shows and productions. The audio-visual effects, the 3D video mix and real-time motion graphics engine, the flexible AV players, the EboLyzer and AGT, and much more. This way we translated the prototype software of SenSei into release worthy code, a big step towards the release of EboSuite.

Cinekid app

For the Cinekid festival we developed the EboTracks app and installation. With the EboTracks app children (7-14 yr.) easily make visual music with video recordings. The app guides them through the creative process. All creative features work in real time. This way, kids can play, experiment and improvise with music and video in a  intuitive, playful way. They use audio-visual effects, create audio-visual rhythms, basslines, harmonies and melodies and mix videos and graphics in real time, in 3D!

Children make short audio-visual compositions (‘Blocks’) that they stitch together in the Block Editor to form an extensive audio-visual composition. Blocks are template compositions that children personalise by adding their own video recordings and using creative tools.

To make a composition children first select a Block. After selecting a Block, it starts playing immediately in a loop, but without video recordings. Placeholders are visble where the video recordings will be. Children use two different screens to personalise the Blocks. One screen to make video recordings and another screen to change the composition.

On the first screen they make video recordings. All neccessary recordings are listed with instructions and buttons to start recording, so they don’t have to leave the screen to make a recording. A recording is used in the Block immediately. Part of a visual rhythm, bassline or a melody appears. An audio-visual analyser analyses the recording to make it fit into the composition automatically. Children are free to make a recording at any time.

At any time they can move to the second screen. This screen contains controls to change the audio-visual effects, the visual rhythm, bassline and melody patterns, the 3D video mix, the background music and animations. All functions work in real time. 

With the XY pads children change the audio-visual effects. Moving the dot over the square space changes two parameters of the effect: one parameter mapped to the X-axis and one parameter mapped to the Y-axis. The movements on the square are recorded to store the changes of the effect over time. The Block is looping, so children hear their recordings in the next loop and can change it (partly or entirely) at any time.

With buttons children change the patterns of the rhythms, basslines and melodies. With each press of a button another pattern is selected from the database. Each Block has many patterns to choose from. Combining the patterns creates new unexpected grooves.

With another button children choose a video mix from the database. These video mixes make the video recordings move together in a virtual 3D space with graphics, color effects and a supporting animation. For this we used SenSei’s 3D video mix and real time motion graphics software (and translated it into C code). The supporting animation can be changed with another button.

At any time children can move the Block to the Block Editor to combine it with other Blocks and then share it on social media.

CineKid installation

For the CineKid festival the app was part of a physical installation on the festival site. The installation provided the children with many objects to sample and to dress themselves up. Their compositions were projected above the installation on a large video screen live. For this each shared video was added to a database connected to a media server with a video projector. After the festival the installation toured through the Netherlands. 

Face Dance app

We also made a prototype for a simplified version of the EboTracks app. It has less, but similar creative functions and doesn’t have the Block Editor. It was meant for branding campaigns. We used Laidback Luke’s track ‘Bea’, for the prototype. 

This app also has two screens: one to record videos and another screen with creative functions to personalise the track. Moving the dot over the blue square chages the audio-visual effects. With buttons different rhythms and audio-visual effects can be selected.

Finally this app was not sold to a client for a branding campaign. A pitty, because it is a great (working) app and concept. Selling this app was not my priority though, because the end goal of interactive tracks was the development of EboSuite, which was launched using this technology in 2017. We did sell this technology for non-real-time apps like TricksMix and Fame Game 2. 

SenSei

The templates (rhythm, bassline and melody patterns, audiovisual effects and video mixes) are made with Ableton Live and SenSei. In the same way as for the other Interactive Tracks apps, like SelfideoClip. Because the technical concepts of the EboTracks app is the same as SenSei, a composition for EboTracks is made in the same way as a regular SenSei track (like for a live show) and imported into the app. 

Senna

The workflow and many features of the EboTracks and FaceDance apps are inspired by Senna, the audio-visual instrument for children we created in 2010. Read more about Senna on the project page.

Team

Produced and developed by Eboman and EboStudio

• Bas van der Graaff
• Jeroen Hofs
• Timo Rozendal
• Nenad Popov