“Did You Hear That?” - A Post-Mortem Analysis on Building Aural Game Experiences

I worked on a project in May of 2017 called “Did You Hear That?,” a completely audio-based experience made with the use of 3D audio technology and the Kinect 2. This technology allowed the player to feel as if he/she were actually in the audial worlds that were being provided to them, which was a truly unique experience.

Our initial plan for the project was that the player would only identify sounds, but after playtesting and receiving user feedback we ended up adding a story to the experience. After being given a pair of headphones and being blindfolded, the player would be presented with a variety of auditory stories to select from: fantasy, western, haunted house, or abandoned coal mine. Once the player selected their story, they would hear narration and several sound cues and would need to identify where the sound was coming from by pointing to it.

The role of the Kinect was to track the player’s movement during this process. If the player successfully identified the sound, they would progress in the story. If not, they would hear the sound cue again to try and identify where the sound was coming from. Because the sounds were recorded with a 3D microphone, I was able to have sounds come from all directions rather than just from the left and right. The storylines we created were intended for children, so many adults found the scripts to be humorous and seemed to have a great time.

Creating the game wasn’t incredibly challenging other than dealing with the Kinect API for the first time. The game itself was created in Unity, but with Microsoft’s Kinect example project as a backbone. What was nice was the fact that Microsoft had already had set up the skeleton and several different cameras, so when the Kinect was plugged in Unity could immediately begin tracking the player.

What ended up being a challenge later on was the fact that the Kinect automatically detected multiple people, so several skeletal instances would spawn. What made this problematic was how the game was set up. When a skeleton would be instantiated, the hands were immediately tagged so that we could determine if the player collided with the sound (when the player collided with the spawned sound the story would progress). We also had a game manager that spawned 3D game objects to represent the sounds (to have something for the skeleton to collide with). Those who were not playing the game could watch the back-end framework being displayed on a computer screen, where the Kinect skeleton and Unity obstacles would collide.

The challenge wasn’t necessarily that a new skeleton would spawn, but that several skeletons could spawn causing confusion for the 3D game objects. The reason we couldn’t handle multiple skeletons was because the 3D game objects were being spawned central to the player’s position. If there were multiple players, the game manager would not know where to spawn the object.

Another major challenge we faced was figuring out how to get the Kinect to not spawn multiple skeletal instances or how to get it to differentiate the different player skeletons. The solution we ended up with was just to prevent other people from walking in front of the camera. It wasn’t a very elegant solution, but it worked — and since this was for a temporary interactive exhibit rather than a shipped game, it was a more reasonable solution than trying to figure out a solution via code.

Even with the obstacles that we had to overcome in creating this experience, it was very rewarding to see so many people play the game. The event attendance that day was between two and three thousand people, and many stopped by to experience what we had created. If I could go back and change something, I would want to add more omnidirectional sounds and try to push the limits of the Kinect’s abilities further.

Gaming technologies can be used to create all sorts of wonderful interactive experiences, I’d encourage anyone to try experimenting with them and see what you can make!