Weeks 8-11 Spatial Audio for VR

The Workflow Challenge:  Ambisonics or Stereo?

Spatial audio helps to enhance the experience of immersion in VR. Despite its importance, designing spatial audio for VR is not easy as there is no unified workflow from the stages of recording, postproduction to the final delivery to the target platform. The quality of sounds also varies greatly between audio engines as I will discuss later.

Spatial audio is primarily used in game and cinematic VR. The audio in cinematic VR has a strong emphasis on the timed playback and synchronisation with video, and less dependent on user interaction except the head tracking in rendering. On the other hand, VR game has higher demand for interaction and less time-based. As the project is a semi-cinematic VR app made with a game engine, I preferred to make room for interactivity that can be easily controlled in Unity instead of mixing down a pre-baked 360 soundscape in a digital audio workstation that currently has so little support for the two major ambisonic formats, namely the traditional FuMa B-format or the popular AmbiX B-format (ACN/SN3D, also known as the first-order B-format).

Enda has proposed four types of audio for the app earlier, including voiceover, music, UI tones and sound effects in general. He took care of the narration and produced music and UI tones. I focused on recording and processing ambience, and implementing spatial audio in Unity with third-party sound engines. The narration, music and UI tones throughout the tour are not supposed to be spatialized. The other audios in our app respond to the head movement, and the audio playbacks depend on user interaction. Enda had asked me to prepare ambience background for each static locales. I discovered later that even as subtle as ambience sounds can have HRTF enabled and respond to the head rotation if done properly from the beginning.

The workflow for designing spatial audio starts from recording. With the ZOOM H2n we can have sounds recorded in a 4-channel AmbiX format, or having regular stereo audios with perfect mono compatibility. A major problem for spatial audio comes in postproduction. Currently only Pro Tool, Reaper and Nuendo that have plugin supports for ambisonics. Mainstream media player such as Quicktime Player, iTune don’t have playback functionality for ambisonics. By contrast, stereo files don’t have such support problems in playback or postproduction. After mixing in a DAW, the audios will be imported into Unity and mapped with head rotation data in the background for real-time rendering of directional sounds. Unity has native support for ambisonics, but the sound quality is far from ideal compared to other third-party audio engines. The Google VR SDK enhances audio spatialisation with the Gvr Audio Soundfield APIs that aims specifically to support ambisonics.The Gvr Audio Soundfield API was mentioned in the Google I/O talk in May 2016 . However this feature was not released until 4 August in the v.0.9.0 SDK. The official documentation was updated the next day. If the GvrAudioSourcefield APIs was available when I made the spatial audio demo in mid-July, I would probably record the audios in the ambiX B-format, have them processed in Reaper and configure them in Unity with the Gvr Audio Soundfield APIs. Due to the time constraint, I don’t have time to swap the ambience with the GvrAudioSourcefield APIs in Unity. Instead, I only made a Youtube demo to show that we have the ability to implement ambisonics in Unity with Google VR SDK.

In light of the compatibility and support problems for ambisonics, I chose an easy workaround for the audio spatialisation. I have the audio recorded in stereo and processed in Audacity. Then the audio clips were imported in Unity. These audios were sprinkled as invisible game objects in the scenes to give a sense of specialisation when users interact with them. As the Gvr Audio Source APIs supports stereo and mono audios and has better spatialisation than the native audio engine in Unity, it was used for the sound effects that have directions in our app.

Recording and Editing Ambience

The first two weeks in August was primarily spent on recording and editing ambience. I recorded the ambience with the ZOOM H2n recorder. It has 5 built-in microphones, with two arranged as an XY pair and the other three configured in an MS pattern. The ambience was recorded in 4-channel mode, with signals from the XY mics and the MS mics recorded onto two separate stereo tracks. The output is two synchronised stereo audios for each locations in the tours. I also upgraded the software to Firmware v2.0. After the upgrade, the ZOOM H2n in the lab can produce first-order ambisonic surround sound and can write the 4-channel audio onto one track in AmbiX B-format. Due to the limit of time, I didn’t have time to re-do the ambience recordings in AmbiX B-format, but settled with the traditional stereo recordings I made in early August. With the GoogleVR SDK for Unity, the rendering of audios can adapt to the head movement in real time.

The ambience recording and editing took more time than expected, spanning from 19 July to 13 August. I first recorded the ambience during the daytime with lots of human activities and conversations. The panorama in the July demo was also packed with people. However, the formal 360 photos and videos was shot in an early morning when the campus was quiet and empty. It sounded very strange to have human activities in the ambience for the serene morning scenes. So I did the field recordings again in the early mornings on 4, 5 and 6 August. As the environment was very quiet, I turned up the Gain levels, fearing to miss the bird tweets and rustling leaves at the background. It turned out to be another mistake, as increasing gain in the recorder also introduced noises, which was difficult to remove in postproduction. Jill suggested not to turn up the Gain level during the recording, but only bring it up in the postproduction when necessary. So I went for another early morning field recording on 13 August and have the Gain level set at 0 or 1. This batch turned out to be very quiet, with the RMS at -70dB in general. I normalised the soundtracks to somewhere between -28dB and -36dB before putting them in Unity. The recordings in August formed a large part of ambience ready for use. The recordings taken in July was mostly discarded. Also, in the postproduction stage, Enda reminded me to remove the low-frequency rumbling in the ambience with high pass filter, which was very useful.

Sample recording 1: Regent House ambience with conversation

Sample recording 2: Regent House ambience without conversation

Sample recording 3: Front Square ambience with conversation

Sample recording 4: Front Square ambience without conversation

Building Spatial Audio and Collaboration In Unity

I had proposed to Enda in July that we should use Git for version control and Github for remote collaboration to keep everything in sync. Enda thought Git was not very helpful for Unity. Git is great for handling text-based files like scripts, but might have problems with scenes and prefabs as these are normally stored as a mix of binary and text in Unity. Although we had the option to store the scene and prefab assets as text-based files , they were too important that we could not afford to break them. Also, it would be much easier for Enda to integrate my works in Unity if they were packed as prefabs and exported as Unity package. We eventually went with the prefab way of collaboration. A minor problem in this approach was that Unity did not support the update of nested prefab. A nested prefab would appear missing and had to be adjusted every time a new set of prefabs were imported into the project. As Enda finished a mature prototype in Unity with photospheres, videospheres, navigation markers and multimedia placeholder in early August, I started to build on his master copy of the project. The following is a snapshot of the structure of game objects in a scene of our project.

1. Photosphere (with Narration attached)
    1) Marker
        a) Navigation (NAV)
        b) Point of Interests (POI)
    2) Audio 
        a) Ambience (AMB)
        b) Directional Sound Effects (SFX)
2. Videosphere
3. UI Sounds
4. Music

I took care of everything grouped under the Audio in the above hierarchy. My work in Unity concerned with the placing and the control of ambience sounds and directional sound effects. I used the GvrAudioSource API to trigger ambience and directional sounds with the event trigger component, while Enda used the AudioSource API in Unity to control narration, music and UI sounds as these audios didn’t need to be spatialised. Hence the manipulation over audio in Unity was clearly separated by class and functionality, and my scripts would not break Enda’ work.

The ambience audio was dropped at a location close to the main camera, and the playback would be triggered when the camera entered the active photosphere. At first I used the GvrAudioSource for the ambience sounds, and off-set it slightly from the camera to increase a feeling of spatialisation. If the location of the ambience audio overlapped with the camera to which the GvrAudioListener was attached, the spatial illusion would not work anymore. The GvrAudioSource would only create a sense of spatialisation if the audio was played somewhere away from GvrAudioListener. Also, the Google VR SDK for Unity tended to attenuate the audio level and muffled the ambience which was very subtle already. After I swapped the ambience with better quality sound tracks, I suddenly realised the degradation of sound quality in Unity might not come from the audio alone, but caused by the rendering algorithm in Google VR SDK for Unity. Considering the overkilled spatialisation and the tendency of Google VR SDK to muffle the original soundtracks, we preferred to use the native audio engine in Unity for the ambiences for a few good reasons. First, the output from the native audio engine in Unity seemed to be much louder than the playback in GvrAudioSource. Second, the output from the native audio engine can be routed to a mixer in Unity, and brought to a desired level in a batch. As the GvrAudioSource sidestepped the mixer workflow in Unity, it also became an advantage in Unity to continue with a mixer approach to adjust audio levels between narration, UI sounds, music and ambience all in one place.

The directional sounds were sprinkled around the scenes and the playbacks were triggered by the gaze-over interaction. Although the GvrAudioSource was incompetent for the ambience sounds in our project, it was suitable for directional sound effects spreading out in the virtual world. Generally there was an empty object in the scene representing the audio source, and an event trigger that decided when the playback should be started. The event trigger could be attached to the object holding the audio source, or separated from it. An advantage of the last approach was that we can have the user looking at one direction but triggering sound from another direction. Also, the object serving as a trigger can be set inactive, its collider can be disabled, or the event trigger component can be disabled after the crosshair exited the colliding zone. This would guarantee the audio would be played only once. In the spatial audio demo in July, the reticle grew immediately after hitting an audio object. In the later build, I disabled the reticle when it entered the colliding zone, and enabled it again upon exit. This wouldn’t affect the interaction between the GvrReticle and the UI markers somewhere else in the scene. As the crosshair stayed unchanged after the collision with audio objects, users needed to explore the soundscapes without the aid of visual cues. 

Another issue was that I needed to be careful with the UI canvas or other game objects that might block the ray-casting from the camera on the audio triggers in the scene. After Enda changed the projection meshes from sphere to octahedron, many audio triggers was laid outside the projected mesh and the transform coordinates needed to be tweaked again. The audio design for each location also changed as more multimedia contents were brought into Unity. For instance, if Enda put an movie clip at the back of the user, I would place an audio trigger at a location that cannot be neglected, such as the area near the forward navigation marker, and to draw attention with directional sound to the position where the movie clip was placed.

GoogleVR, FB360 Rendering SDK (TwoBigEars’s 3Dception)and Compatibility Issue in Unity

In the last week of August, I started to consider implementing the ambience surround sounds with the newly release GvrAudioSoundfield APIs or another third-party plugin for Unity. 3Dception has been highly praised before the company Two Big Ears was acquired by Facebook recently. I had a false impression from the official website that the Two Big Ears ceased support to the Unity plugin. It turned out that the rendering engine was incorporated into the FB360 Rendering SDK, which was free for download. I tried both the FB360 rendering engine (henceforth TwoBigEars rendering engine for the sake of brevity), and the GvrAudioSoundfield APIs.

Here is a brief comparison between the GvrAudioSoundfield in Google VR SDK for Unity and the TwoBigEars rendering engine in FB360 in terms of sound quality, supported audio formats and the ease of use.

The GvrAudioSoundfield, as part of the Google VR SDK for Unity, tends to attenuate the sound levels, and developer had to turned up the Gain from the inspector to have louder surround sounds. But the average sound levels of GvrAudioSoundfield were higher than the GvrAudioSource. The TwoBigEars rendering engine tends to raise the sound to the highest possible levels and let the developer to scale it down in the range between 0 and 1.

The GvrAudioSoundfield have support for mono, stereo and the ambiX B-format audio which has 4 channels in one track. One can simply drag and drop two audios in the Unity inspector and the Google VR SDK would render them with head tracking data.

The TwoBigEars rendering SDK requires encoding in the custom *.tbe format, B-format (FuMa), B-format (ambiX) or quad-binaural format before decoding and rendering the audios in Unity. It has indirect support for the traditional mono, stereo audios in popular DAWs such as Reaper and Pro Tool through a rebranded spatialiser plugin. In Reaper, sound designers can use the FB360 plugin to spatialise mono or stereo sound tracks, and rendered the master 3D track into an 8-channel audio file in Reaper. The 8-channel audio would be encoded in the FB360 encoder before bringing it to Unity. I tried the customised *.tbe encoding and decoded the audio in Unity. The spatialisation was slightly better the GvrAudioSoundfield. However, it has more softwares involved, and the TwoBigEars audio engine in the FB360 rendering SDK only had a few playback parameters exposed in the Unity inspector. The TBE rendering engine doesn’t even provide a loop function in the inspector, and I had to write a simple C# script for looping an audio asset with the TBE rendering engine. The TBE APIs could be queried in C#, but the amount of exposed APIs was limited. In short, the GvrAudioSoundfield has a more easy-to-use interface in Unity while the TwoBigEars has better quality in audio rendering.

Last but not least, the Google VR SDK cannot work together with the FB360 rendering engine in the same project. They both works well with the native audio engine in Unity. Considering all the directional sounds in Unity used the Google VR SDK, it would be a large amount of work to migrate the directional sounds to the TwoBigEars rendering engine. Due to the time limits, Enda and I decided to leave the ambience rendering to Unity’s native audio engine, while sticking with the GvrAudioSource for the directional sound effects in Unity.

Sample ambience soundtrack playback in Reaper

Soundtrack encoded with FB360 Encoder and decoded with FB360 Rendering SDK (TwoBigEars’ TBSpatDecoder) in Unity

Soundtrack playback with Audio Source in Unity

Soundtrack playback with Google VR SDK’s GvrAudioSoundfield

Soundtrack playback with Google VR SDK’s GvrAudioSource

Advertisements

Week 7 (25-31 July, 2016)

Hi, it’s Ying here. It has been quiet a while since my last update, which will be recounted in this post. I will start with the last few week in July after the mid-term presentation first. Then in the following posts I will summarise my work in August.

Spatial Audio Runtime Glitch in Unity

After finishing the spatial audio demo with Google VR SDK in week 6, I found an annoying glitch in Unity. My spatial audio demo had four audio assets – the bell, the chisel, the bike and the ambience. All of them are triggered by the user’s gaze input except the ambience, which is played on awake and always looping in the background. When the clips are playing simultaneously in the runtime, the screen flashes in pink on my macbook. To be honest I don’t know the exact cause of the glitch, probably due to the memory limitation on my mac or that Unity could not handle audio clips at different levels very well. The glitch did not happen after I normalised the individual audio files. I did the normalisation in Audacity, but the task can be done in other DAWs or in Unity as well.

Waveform Analyser Plugin for Audacity

I reported the aforesaid glitch to Enda, and he suggested to bring all the audio clips roughly to the same levels to save the computer from extra calculation. He suggested to me an audio waveform analyser plugin for Audacity < https://gist.github.com/endolith/514077 >, so that I could check the peaks and average levels (RMS) of each soundtracks. This plugin is very useful when I need to know the average decibel of the audio files before putting them into Unity. A solution for the glitch is found. As long as the audios were normalised to the approximate level, the screen would not flash again in Unity 5.4.0b23 in runtime.

Integrating VST in Cubase and Creating UI sounds

I used Cubase for the composition of UI tones. Although my UI sounds are not adopted in the end, I think the process itself worths a bit of documentation. On Tuesday and Wednesday (26-27 July), I was searching for free VSTs and integrating them into Cubase.

Jill preferred to have a percussion called hang for the UI. I could not find a free VST for this wonderful instrument unfortunately. Also, Enda mentioned that he preferred to have a mix of harp and piano for the UI. The notion was inspired by Trinity College Harp, which, according to legend, was once owned by the Irish King Brian Boru. The harp represents the Irish music tradition and aesthetics in many ways. I specifically looked for a harp VST as the HALION Sonic instruments in Cubase are mostly unavailable on the PC I am working with. Also the existing VSTs don’t have a natural sound either for a harp or a piano. By the end of the day I got 10 VSTs installed in Cubase. The names of the VSTs are listed below. I decided to only go with mda Piano and the VS Etherealwinds Harp to make the UI sounds.

mda Piano < http://mda.smartelectronix.com/synths.htm >
VS Etherealwinds Harp < http://vis.versilstudios.net/etherealwinds-harp.html >
Other VSTs: DSKAkoustiK KeyZ, EP-Station, Piano Harp, 4Front EPiano Module, EVM Grand Piano, GlueReeds, Joe’s Jazz Piano I, MrRay 73

UI-COK2

I made two UI tones, one for the highlight effect when a UI marker is gazed upon and the other for confirmation when a UI marker is clicked. They are made with the VS Etherealwinds Harp and the mda Piano. I chose a chord for the highlight and two notes for the confirmation. Both are very simple so that they won’t distract the user from the immersion experience.


 

Research on Irish harp music

This week I also did a little bit of research into the Irish harp music. Turlough O’Carolan (1670-1738) is perhaps the most famous harper and composer in Irish history. He bequeathed generations of Irish musicians with more than 200 pieces for harp performance alone. I selected seven songs from his repertoires in preparation for the video background music in the app.

Captain O’Kane (recorded)
Lady Maxwell (recorded)
Maurice O’Connor
Blind Mary
Lady Anthenry
Mrs Judge
Sheeberg & Sheemore

Many songs created by Turlough O’Carolan were not published during his life time, and they were supposed to be learned by ears. When I searched for the sheet music for the above pieces, the scores I found only provided the main theme for the right hand. Performer needs to improvise on the left-hand accompaniment, which has unlimited possibility of variance between performers. Due to constraint of time and the limit of my talent, I only recorded the main theme for Captain O’Kane and Lady Maxwell on Friday, 29 July. I passed the Captain O’Kane piece in MIDI to Enda. He found a better arrangement for that piece in the end.


 

Finalising Ambience Sound Design Document

I also finalised the ambience sound design document on Thursday, 28 July so that Enda and others could have an idea what kind of sounds would be put into Unity by locations. The design document followed the naming conventions Enda had outlined in Unity. The test recordings I took on 18, 19, 20 July were also taken into account, especially the environment and direction where the sounds are coming from. Having the sound design penned down, I took another two field recordings on Sunday, 31 July and Monday, 1 August.

Audio format considerations

All the ambience was recorded in MS and XY directions. There were stereo audio files with 4 channels in two tracks. I did not recorded the ambience in a 4-channel ambisonic format into one track as I was not sure about the editing workflow in DAW and the support for this format in Unity. I thought the directional 4-channel recording would suffice for the postproduction anyway.

Also I decided not to export the audio as MP3 format. There was 0.04 seconds of silence at the beginning of the sound track, which was unavoidable as long as it was exported in mp3. It would result in a sudden pop, which was bad when the track was looped in Unity. I would rather export the audio file into .wav and let Unity to compress it after the assets were imported.

Weeks 4-6 (4-24 July)

Week 4-5 (4-17 July)

Transition Experiment in Unity

I continued working with Unity for most of the time in week 4, especially using C# to try out different transition effects. I experimented with the skybox in Unity to create the same illusion in our VR project (Fig.1) instead of applying an equirectangular image to a sphere. One advantage of manipulating the skybox is that there is a script ready for sky transition online, which can be used as a transition between photospheres. A limitation is that it drained more resources than mapping a photosphere with images and hence adverse to optimisation.

Fig. 1 Skybox Mapping in Unity

Unity-TextureTest01-02

In fact, I came across this skybox transition when I was looking for a “fade-into-black” effect. We want to have either a crossfade or fade-into-black effect before moving the main camera from one photosphere to another in Unity. The lerp method did not do the job well as the textures we intend to use are not pure colours. I shelved the transition challenge as it is a bonus but not crucial for the project. I would check with the UI CrossFadeAlpha in Unity if time allowed to see whether it can be used for the desired transition effect.

 

Collecting Sound Clips

I also started to collect audio clips since week 4. They were primarily ambience recordings from website such as <www.freesound.org> and some sound effects from different sound libraries, which can be used in sound design at the later stage. The collection contains c. 200 audio clips in total and are categorised as follows. The list might expand in the future after the sound design document is mapped down.

(1) bird ; (2) tree ; (3) bell; (4) clock; (5) wind chime; (6) people & conversation; (7) footsteps; (8) bicycles; (9) cars; (10) ambience; (11) special sound effects; (12) chisel; (13) shovel; (14) gun; (15) metal; (16) yelling; (17) animals (cow, dear, donkey etc.)

 

Comparing Sound Engines For Unity

In week 5 I did a bit research on sound engines that had support for Unity. Fortunately I found a good article with an app demo (Skeletons vs 3D Audio VR) that saved me loads of time.

This article compared 3Dception from Two Big Ears (now acquired by Facebook), AstoundSound RTI from (€150), Phono 3D from Impulsonic (1-month evaluation license), RealSpace 3D from VisiSonics, and Oculus Audio SDK. My first choice would be 3Dception as it has been used in the Into Kilmainham: A VR Story project in the previous year. After the Facebook buyout in May 2016, the Two Big Ears ceased to offer the plugin for Unity. The AstoundSound was available in the Unity Asset Store but it’s way much expensive than we could afford. The remaining options are generally inferior in quality, or not compatible with the Google cardboard (Oculus Audio SDK).

I decided to go with the spatial audio module in the Google VR SDK for Unity because it is free. Google did not provide good documentation for the spatial audio in early July as they were probably busy with the release of the 0.85 version of the Google VR SDK for Unity. As the documentation was finally updated in mid July and with the help of some Youtube tutorials, I managed to get the demo scenes working in the Google VR SDK for Unity v0.8 (Fig.2).

Fig 2. Audio Demo in Google VR SDK for Unity

This slideshow requires JavaScript.

Week 6 (18-24 July)

Field Recording & Audio Editing 

On Monday, 18 July, I recorded the ambience sounds in the Museum building and the Long Room with the Roland CS-10EM binaural microphones and the Zoom H2N recorder. On Tuesday, 19 July, I went to take field recordings at the Front Gate, the Front Square, the New Square, the Gate of the Berkeley Library and the Museum Building in the order of mapping soundscapes for these locations. I forgot the Rubrics and the Library Square which I will make it up later.

Building Audio Demo in Unity

On Wednesday, 20 July, I spent the entire day to catalogue and edit the field recordings I took two days ago, which is not finished yet. On Thursday, 21 July, I perused the Google documentation and built a spatial audio prototype in Unity (Fig. 3).

Fig. 3 Positioning Box Colliders for the Audio Sources in Unity

Unity-GvrAudio-MB01

Here are the necessary steps to follow to get the spatial audio module in Google VR SDK working in Unity.

  • Choose the “Gvr Audio Spatializer” under the “Edit > Project Settings > Audio” menu
  • Main camera: (a) Physics Raycaster; (b) Gvr Audio Listener
  • An Empty Game Object: (a) Event Trigger; (b) Gvr Audio Source; (c) Gvr Audio Room (this component is optional)
  • Event System: make sure the Gvr Gaze Input is ranked above the other input modules

In the Museum Building demo scene, I used one clip for ambience sounds, and attached a chime audio to the clock in front of the camera,  a shovel audio to the left hand side where the Geology Department is located, a bicycle sound clip to the right hand side which houses the Engineering Department. The ambiance sound is played on awake and set to loop during the runtime. Other sound are only triggered when the user’s pointer/gaze collides with the corresponding box colliders. The demo can be previewed in the video that follows.

Fig. 4 Spatial Audio Demo made with Google VR SDK for Unity  

Weeks 1-3 (13 June – 3 July)

My first few weeks are spent on learning the basics of C#, the Unity Game Engine and the Google VR SDK. The Unity website provides lots of video tutorials, documentation with example codes, which is a good place to start with. I followed the tutorials on Scripting, User Interfaces and Graphics. Enda also recommended lynda.com, and we shared resources on 3D Essential Training, Scripting Unity with C# and the Unity 5 UI lessons.

Interactions for Cardboard VR

The first thing to consider for a cardboard VR project is the types of input. Once the phone is inserted into a viewer, it becomes inaccessible for hand gestures such as swipe, fling, pinch zoom. The remaining options are the simulation of touch, orientation and movement (gyroscope and accelerometer), location services (cellular, GPS, Wi-Fi and bluetooth), connectivity (bluetooth, Wi-Fi), and the camera. Google Cardboard had used a magnet to emulate physical touch on the screen, which was replaced by a capacitive lever connected to a button in 2015. Besides the simulation of touch, a cardboard project can make use of the gyro, the accelerator and the bluetooth to add extra layers of interaction. As we don’t have experience with mobile development, the Google VR SDK for Unity came to our aid. It tracks the head movement so we don’t have to configure the gyro ourselves. It also maps the gaze from the user (GazeInputModule) nicely with the input module in Unity (BaseInputModule).

There is an open-sourced scripts “Cardboard Controls+” that enhance the Google VR SDK for Unity with more input options. For instance, it combines an event-listener with a timer that can be set easily, which is convenient for cardboards that don’t have a button.

Cardboard Controls+
<github.com/JScott/cardboard-controls/tree/master/CardboardControl>

Unity GUI system

The handling of user-driven events in Unity requires a few components to work together.

  • Event System, which monitors the event flow. We need to add the GazeInputModules from the Google VR SDK under the Event System and make sure the Gaze Input Module is ranked above the Standalone Input Module (mouse, keyboard, touch) in the Inspector panel in Unity.
  • Physics Raycaster, which is attached to the main camera of the GvrMain prefabs automatically. As the name suggest, it casts rays from the camera in the virtual world and tells whether the user is interacting with any objects in the scene.
  • Event Trigger, which is added to the game objects, either 3D meshes or UI elements that we wish the user would interact with.

Google Design Lab proposes using light and gaze cues as design principles for VR. Considering the change of lighting would impact on the movie mapping, it is fine to shelve the idea of adapting illumination as part of the UI design. The common HUD (Heads Up Display) is not suitable for VR as it is too close for the user to see. It doesn’t mean that we cannot use the UI in Unity. What we need to do is to change the UI Canvas rendering from Screen Space Overlay / Screen Space Camera to World Space, and spread the UI components to the points of interests in the virtual world. There is a bug in Unity (starting with 5.3.4p2) preventing the World Space GUI to render onto a RenderTexture, which is used for distortion correction by Google VR SDK. The bug is fixed after updating Unity to 5.4.

Unity provides many out-of-the-box event listeners such as PointerEnter(), PointerClick() and so forth, which is great for beginners as it requires no scripting for handling simple events (see video 1). To have more flexible interaction, however, we need to create our own script. For instance, if we want to show or hide an object with a switch without resorting to the toggle box in UI by default, we need to create a script with a boolean to change the current state. The target object can be set active, enabled, rendered, or assigned with a different alpha value depending on the type (see video 2). If we want to swap the textures, we can create an array to hold different materials and step through it whenever the user fires an event (see video 3).

Enda did a great job in customising GUI for the prototype. As he is flying ahead with Unity, we might need to use Git, a version control tool to cooperate in Unity in the future.

Summary

week 0 (starting 6 June) brainstorming the project and fleshing out the workflow
week 1 (starting 13 June) Learning C# and Git
week 2 (starting 20 June) Starting with Unity
week 3 (starting 27 June) Focusing on UI and Interaction
week 4 (starting 4 July) Controlling interaction via scripting, collecting audio clips for sound design
week 5 (starting 11 July) Research on audio engine compatible with Unity