Space dragonflies. A computer vision system for gesture and sonic interaction

Space Dragonflies is a computer vision project within a hybrid mixed reality space. The audio-visual installation aims to reflect on the relationships between humans, nature and digital information. These relationships are sometimes based on control, usually a human intention to subordinate nature, other times causal or even random.

The installation proposes a space for interaction, an immersive experience where the user can modify the behavior of a swarm of dragonflies and along with it, create a symphony.

Technical setup

Space butterflies is a modular installation that requieres different types of technological solutions and tools.

  • Computer vision: a patch created in PureData capable of using a camera to perform blob detection. The patch will subtract foreground and background and provide information in the form of X-Y coordinates regarding the motion of the blob.

  • 3D Graphics: The swarm of dragonflies has been created in Unity 3D software. Each individual insect has its own motion but is part of a larger entity which is the swarm. This swarm of dragonflies has a nest to one of the upper corners of the screen and a set of assigned behaviors.
    The dragonflies follow the upper part of the detected blob providing a sense of agency in the user. These insects will continuously follow the motion of the user but will also decide by themselves if they want to stop following and return to the nest. The user will have to catch their attention again by moving in any given direction so as to create a playful experience.

  • Sound mixer: a patch in PureData allows us to trigger and mix the different sounds that the user interacts with. The motion detection patch delivers information regarding not only the XY values for position of the blob but also speed of the movement, size of the blob, height and width. All these parameters are mapped into our sound mixer allowing for the sonic representation of the user’s motion. Since the behavior of the dragonflies is mapped with that of the user, the sonification has a strong relationship with the behavior of the former.

    With the described setup we intend to create an integral immersive experience that provides the user with a strong feeling of interaction and an ambivalent perception of control. Sometimes she feels like her behavior leads to the control of the audiovisual outputs but sometimes this control is lost and the digital representation of nature makes its own way across sound and movement.

    We created this project with Diogo, Giovanni, Aditya, Daniel and Josep.

About Mara Balestrini

Mara Balestrini is Human Computer Interaction (HCI) expert and technology strategist.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: