KaleidOk test ‘emotional speech visualiser’ binnen Observe

Door op

Kimberley Bianca, mede-oprichter van KaleidOk, is onlangs bij het lectoraat afgestudeerd rondom het project Observe. Zij heeft eigen ontworpen content (emotional speech visualiser) getest op publieke schermen. In deze blogpost is te lezen wat haar onderzoek inhield en welke bevindingen zij heeft gedaan. Voor een impressie van haar werk kan onderstaande video bekeken worden:

Most of the time technology inhibits emotional experience and deep social connection. This can limit our communication in almost all contemporary situations, especially in public spaces or in the sharing of collective information. There is a demand for engaging, interactive content for public screens, so we present an emotional spoken word image retriever and visualiser made with the goal of enhancing the emotional expression had with a display. We employed a method of applied research, which involved system development, artistic practice and user studies. Through this journey we found that by displaying affective visual feedback of speech people are more than willing to express themselves emotionally and, therefore, have a more meaningful experiences with the interactive content than is currently widely available.

This user study was undertaken in the context of a public interactive screen inside two publicly accessible venues and two locations outside in the public space. The four research questions investigated relate to participants use the system, emotional experience with the interaction, the perception of the content, and of a greater social interest in the project. The results demonstrated that most importantly kaleidOk proves to enhance the emotional experience with an interactive public display, however with much room to improve on in regards to user experience design.

1. Enschede, Epy Drost Foyer: Saxion University of Applied Sciences
Date: 1st October 2015


2. Hengelo, City Centre: Innocent Subcultural Youth Centre
Date: 10th October 2015


3. Berlin, Cinema: Panke Cultural Centre
Date: 21st October 2015

4. Berlin, City Centre: Platoon Kunsthalle
Date: 23rd October 2015


Proximity and Profile:
About half of people who came to the screen and interacted were alone. Fewer people came in groups. However, pairs and groups typically stayed longer interacting or mingling by the screen. At Saxion, there was the unique case in which a whole class was sent down to have a look. Other than this there was the man in Hengelo who brought some of his friends from the neighboring bar, although they did not stick around. Onlookers in Berlin seemed to be on their own more so than in Enschede or Hengelo.

At Saxion all participants spoke English as a second language, however with near bilingual proficiency. In Hengelo there was more trouble with speaking English, however there were two participants who tried to use kaleidOk in Dutch and in French. At both venues in Berlin, people were mostly speaking in English, although, having kaleidOk only working in English did make an elderly couple too shy to try it out, and they walked away.
People are most often quite shy or modest. Some of these people noticeably begin to open up with use of kaleidOk. There was one person at each intervention that was more outgoing tried to encourage others to participate.
The Hengelo intervention generated the most social atmosphere of the four. People who did not want to use kaleidOk were staying around to see what was going on or how it works. People even left and came back bringing their friends to have a look.
Something noticeable about all experiments is that the discussions surrounding kaleidOk were mostly of a philosophical nature, between friends and strangers.

Most users through all venues hesitated at first. Either they are not sure if they want to participate, or after they have already decided to they begin to freeze up. This hesitation usually passes quite soon after speaking and getting into a flow. With a very brief explanation, people seemed to comprehend quite easily how kaleidOk worked and what the interaction form involved. Some people had freeform emotional speech come to them naturally, and others needed time to think about what they intend to say. As there was no time constraint and no pressure on the users to perform or not, the word’s always came. People were often laughing and smiling at themselves, or at their loved ones who were speaking about them. People stayed much longer with kaleidOk than I had anticipated, approximately five minutes per user, but this may be only one minute of speech interaction, and the rest of discussion and reflection of the imagery.

The number of interactive participants cannot determine the effectiveness of kaleidOk within our research questions at such a small scale of studies. However it is useful to see that with each venue, over a 2.5-hour period, there wasn’t such a big difference with the amount of interactions. What was significantly different was the amount of onlookers and people at the screen (although not interacting). Both outdoor locations in Hengelo and Berlin attracted between 20 and 40 onlookers, but less people in close proximity to the screen. The indoor venues attracted fewer onlookers from afar and more people standing at the screen, or by the participant to see what was going on.

Use of System:
None of the participants so far have used something like this before. However, one onlooker said he had seen something like this before, in the late 90’s in France. A little more than half of the users found it “a bit difficult” to use, this was skewed as the system was still buggy. Users who used the system without bugs said it was rather easy. Some users needed to be guided, after that they were able to understand the interaction easily. The comfort of speaking in public space depends on per person basis.

Emotion Evocation:
All users so far have said it has evoked an emotional response to some degree. This includes bringing out new emotions, which most said- happiness. And in highlighting existing emotions most said- happiness, frustration and nostalgia. Every user questioned has said the use has encouraged (even forced) them to be more emotional.

Content Perception:
The most common remark on the perception is that it was hard to focus on kaleidOk while speaking. Most users only really looked at the final result. Even with just the final result, most users said they saw their emotional content and word use reflected in the imagery. Others subjectively saw their emotional content reflected. Some users were unsure what they were seeing but enjoyed it anyway. And some users with no emotional words said they saw no emotional reflections. All users except for one were “surprised” with what they saw, and someone reported being “shocked”, the one exception saying that there’s nothing surprising left in Berlin. All users thought it was aesthetically pleasing and interesting.

All participants said they would bring people to try it out, such as their friends, family and classmates. However, the exception I gave them in a discussion being that “if kaleidOk was in a more user-friendly stage of development”. Most participants had suggestions for usage of kaleidOk in other contexts, i.e., in their city or community. These potential uses are recorded as- in education, rehabilitation, cultural centres, and in learning English. Nearly half of the people think it would work well in public spaces, and with the ones that do, they think it can make a good icebreaker or a positive intrusion into “small talk” people typically have in public.

Further Work:
In our further work, we will address aspects of emotion detection from the voice, and updated developments on emotion in speech recognition. Firstly we will employ an alternative speech to text service (currently experimenting with CMU Sphinx). It would be a great enhancement to use analysis of tonal features to contribute to the colour search and texture generation, so not only speech but also singing and other vocal expressions can enhance the interaction and results. It is most desirable that we use an alternative image search index, and one that can be accessed and contributed to by a local community audience. User studies will need to address other factors such as user experience design and interface design. We will also find a research methodology to test emotional accuracy, and for reproducible outcomes.


Deel deze pagina