abstract

This paper explains the importance of transparency in the context of interactive digital art, using a particular project to investigate some of the different problems.

The project takes the form of a grid of pressure sensitive pads that act as the input to an audio/visual system.

1.   Introduction

The reason that this project has been undertaken is due to the general lack of transparency in a vast majority of interactive digital artworks. It is felt by the authors that the issue of transparency, or more importantly the control of transparency, is of fundamental importance when planning an interactive artwork.

1.1.      Transparency

Transparency can be taken in this context as the audience/participant’s ability to surmise the general operation of the interactive artwork. It is not necessary for the audience to actually understand the inner workings of the artwork, but instead to generalise how their actions influence the output of the artwork.

A simplified feedback system is shown below (fig.1) showing the interactive artwork with both input and output. The input can come from any source, but in the interest of interactivity the input should be modifiable by the viewer. The output can take many forms, most commonly using some form of visual or auditory output. The input and output are the initial concern in assessing the function of the artwork. ‘What can I control?’ And ‘What am I changing?’

The most important part however, once the input and output of the artwork is established is determining how the input is being translated into the output. This is where the user will potentially try a series of different inputs and will (subconsciously at least) work out what the artwork is doing. It is this stage that is most important to the transparency of an artwork. If the viewer cannot determine in any way what the artwork is doing to the input, then the system is to all extents either completely random or opaque. If the viewer can clearly distinguish what the artwork is doing, even if they do not understand how it is doing it, then the artwork is effectively transparent.

Fig. 1 Feedback system between Artwork and Viewer.

1.2.      The role of transparency in the traditional arts

Traditionally, the arts are relatively transparent. This could be mainly attributed to artworks either being static, or if dynamic (such as the performing arts) they involve humans in the dynamic role. In more detail, if a human takes the place of the artwork in fig.1, then the viewer can instantly know the inputs and outputs of the system if they are familiar with an average human.

1.3.      Electronics and Transparency

When the Artwork involves the use of electronics, including computers, the function of the artwork is immediately hidden away from the viewer. Thus, it could be claimed that the electronic or digital artwork is in fact inherently opaque.

However, in contrast to this, the designer or artist involved in the construction of the electronic artwork should have a very clear idea of how the artwork functions. This may cause the problem that the artwork is completely transparent to them and they feel no need to convey this transparency to the casual viewer. The result of such endeavours is usually an artwork that only appeals to the technically informed.

A second common problem, is that the artist feels that complexity is lost if the artwork is made transparent.

The problem then is how can an electronic artwork be made transparent without sacrificing complexity?

2.   The project

The project involved creating an audio-visual artwork which depended on the participants interaction, exploration and understanding.

2.1.      Hardware

Twenty pressure sensitive mats are used to form a four by five grid approximately three metres square. These mats work as switches, so that when a person is standing on them the switch is closed. These switches are then wired into a voltage-to-MIDI converter, so that when the mats are switched on, a MIDI note-on message is generated.

A prototype of the floor mats was created with twenty push-button switches (fig. 2). Although this prototype was never meant to be the primary input, it has been found to be an equally valid for a single user to interact with the system.

The output of the system is a standard video projector and four speakers. Ideally the video projector would be mounted on the ceiling and project onto the input pads, however due to the lack of a permanent space for the project this has not been tested yet. The speakers are positioned on each corner of the grid allowing quadraphony.

Fig. 2 Push-button Pad

2.2.      Visual Software

The software used to generate the visuals is Processing [1], a java based language that has been made specifically as an introduction to programming visual art. It was decided to use this software for the visuals for three reasons; firstly it offers an extensive drawing library, secondly because it is relatively fast at performing mathematical calculations, thirdly because it can communicate with the chosen audio software [2].

2.3.      Audio Software

The audio software used is Max/MSP [2], a graphical programming language designed specifically for audio experimentation. This program was chosen both for its sound manipulation capabilities and also its ease of interpreting MIDI data.

2.4.      Mapping & Interpretation

The first step in mapping the input to the output was to simply display which pads were activated on the screen. The next small step from this was to join all the activated pads together by a line (fig. 3). Both of these steps were purely visual and completely transparent to the user.

Fig. 3 Early prototype.

The next stage of development involved adding a sound element to the system. This is the point in which it would be possible to loose transparency rather easily, so we investigated various methods of relating sounds to the input. The most obvious method is one that is employed by a majority of traditional instruments, which is to map every input to a different sound. So to avoid the simplicity of this one-to-one mapping, whilst still maintaining the simplicity of concept, we decided to use the length of the line between two points as the input rather than the point itself. Various ideas were proposed in how to map the length of the line to a sound. These included using the length to control the length of a sample, control the pitch of a sample or control the pitch of a generated waveform. After much experimentation it was decided to use the length of the line to control the pitch of a sine wave. This choice was also brought about by the fact that due to the uniform grid, interesting ratios between multiple sine waves were occurring.

Although the sine waves added a sound element, the system was found to be far too static to hold the attention of the user. For this purpose, we decided to add a rhythmic element. A simple, and potentially transparent, way to implement this was to have balls shuttling back and forth along each line (fig. 4) triggering a sound event every time they hit the end of the line. The sound events were then quadraphonically panned so that as the ball hit the representation of the mat on which a participant was standing, the sound appeared to come from that area on the grid.

At this point in the development of the project it had to be decided how many levels of abstraction were going to be present in the finished piece. Initially, to remain true to the idea of transparency, we considered only displaying only one abstraction (i.e. just the sine waves, or just the balls) at one time. We decided instead to attempt to use both processes at the same time whilst maintaining a high level of transparency.

Fig. 4 Prototype showing red balls moving back and forth along each line.

A problem with using the sine waves is that it is not immediately obvious as to how they are being generated. A solution to this is to draw a representative sine wave between each point (fig. 5). This also led to the development of lowering the pitch of the sine wave as the line is lengthened and likewise raising the pitch as the line is shortened. This then means that the both the visualisation and the sounds produced are in concordance.

Fig. 5 Prototype showing the addition of a sine wave between each point.

A late addition to the rhythm system is the rotation through a set of samples. A particular number of samples is chosen (typically four) that can be rotated through on each hit of the ball. Each square holds its current position in the rotation, which is then triggered when a ball hits the square. When the square is off the current sample of the square is returned to the first sample. This, alongside the quadraphonic panning of the samples, allowed a complex polyrhythm to emerge. An important aspect of this is its strict determinism. The same input will always output exactly the same rhythm and visuals. This is a very important part of transparency, and is not to be overlooked.

3.   Results

The project resulted in a piece that was fully functional and at a stage of development where it could be shown in a gallery environment.

3.1.      Visual

The final visual output (fig. 6) was changed dramatically from the initial prototypes, mainly as a consequence of using the projector. The background tones have been subdued substantially and the active elements have been brought to the front by the use of orange tones.

Through testing it was found that the final visualization was adequate in itself in explaining the sound-generation process to the participant.

Fig. 6 Final visualisation.

3.2.      Audio

The exploration of building a transparent artwork determined most of the important decisions with regard to its audio composition. For example the sine waves were chosen because, due to their lack of harmonics, they can be easily perceived as separate sounds when heard at the same time and are also easily displayed graphically.

The quadraphonic panning of the samples was another addition that was made directly as an aid to transparency.

It was found that the aesthetic of the audio almost entirely depended on the choice of the rhythm sounds. There was an important choice to be made at this point whether these sounds were to be produced by synthesis or by sampled audio files. After experimenting it was found that non-pitched sounds complemented the sine waves well and although the synthesis of non-pitched sounds would have been possible, it was decided that samples would be more flexible.

3.3.      Hardware

It was that by running MaxMSP on one laptop and Processing on another, both the audio and visuals could be made to run smoothly and in synchronisation with one another [3]. Also, by separating the project into two distinct halves the speed of development was increased, because both parts could be updated simultaneously.

4.   further work

One of the main improvements would be to project the visualisation directly on to the floor, so that the link between sensor and visual output is even stronger. This would only become possible though if this work was installed, as it would require installing a projector above the pads.

A similar improvement would be to cover the pressure pads. This would serve to protect them from viewers and also to improve the overall aesthetics of the completed installation. This would also be necessary if the proposed idea of floor projection is used.

In regards to the visual aesthetic, one method of improving the visualisation would be to show the composition of all the sine waves that are currently playing. Thus giving another direct link from the sound to the visuals.

It is also felt that further work is needed in refining the audio content, and also in improving the sound generation program.

Finally, the piece will be developed into a single-user instrument utilising the push-button box as opposed to the floor pads. This will allow the piece to be displayed more readily without the need for floor-space and a projector.

5.   conclusion

By undertaking this project it was hoped that a greater understanding would be gained of the role of transparency in digital artwork. Although some artistic considerations were sacrificed in order to explore this area, the project has formed a solid basis on which to create further works of greater complexity but equal transparency.

6.   reference

[1]     “Processing” http://processing.org/

[2]     “Max/MSP” http://www.cycling74.com/products/maxmsp.html

[3]     “JKmaxLink” http://jklabs.net/maxlink/

(All internet address are correct on the 20^th May 2004)