What follows is part 2 of a two-part article on the aesthetic potential of Virtual Reality (VR) storytelling, as extrapolated from what we already know about cinema. Part One can be found here:
http://arnonshorr.blogspot.com/2016/02/the-narrative-vr-rulebook-academic.html
Part 2: Creating the Virtual Dream
In part 1 of this article, we explored the striking parallels between the development of VR cinema and the early history of movies. Now, it’s time to dive into the heavy stuff: using film theory to develop the new language, syntax and grammar of virtual cinema.
For a filmmaker, the most vital thing to understand about movies is their effect on the audience. Film theory has approached this topic from many angles, but I’d like to begin with one particular idea: a good movie is like a dream. Films (at their best) induce a sort of dissociative state in the viewer, a dream-like experience where the viewer gets to identify with another character in another character’s narrative. When the shark attacks Quint’s boat, I know that Chief Brody is scared, but I’m scared, too. Elision of time is another dream-like characteristic of cinema: we’re in one place and time in one scene, then there’s a cut or transition, and suddenly it’s a day later, or a different place. Only in dreams do we have that experience.
But some movies don’t achieve this dream-like effect. Really bad movies, or films that shun the traditional cinematic techniques that we’ve come to expect, throw us out of that dream-like state. Cinema doesn’t automatically hypnotize us - it requires something on the filmmaker’s part to achieve that effect.
The movie-as-dream experience isn’t purely passive. When we’re watching a movie, we have the freedom to look around at anything on the screen. To be drawn into the story, we require expert filmmaking to craft images and sequences that guide and maintain our attention to the narrative. When done right, this keeps us engaged in the dream for the duration of the movie. You might think that this is easier in VR - after all, VR offers an almost automatic immersive experience! In fact, the opposite is true. The more we have to look at, the harder the filmmaker needs to work to shepherd our attention.
For VR cinema to work, it must match or exceed traditional cinema’s ability to capture and guide our dream-like attention. Today, many VR films fail in this regard. We’re in a room with the characters, and they’re doing something, but there’s so much to look at, and there’s so little done by the filmmakers to guide our attention! We may feel like we’re “in the room”, but we don’t feel that we’re “in the story”. Let’s look at some of cinema’s traditional tools, and see how VR pioneers might apply them in a virtual space:
In “flat” movies, the filmmaker’s tools are varied, but include: The Cut (to skip over the less engaging parts of the story); The Close-Up (to make sure the viewer attends to the important detail); Depth of Field (to bring the important detail into focus, or to exclude unimportant details by keeping them out-of-focus); Framing, Staging and Lighting (to draw attention to the important detail); Music and Sound (to help carry us through transitions, to isolate and identify what’s important to hear).
The Cut: A straightforward cut from one VR shot to another can be disorienting if the viewer isn’t guided by the filmmaker as to where to look. A “bad” cut can throw the viewer out of the semi-dream state. Concepts of traditional “invisible” editing are harder to apply in VR, but it’s worth considering what they accomplish: match-on-action (where the end of one shot and the beginning of the next shot show the same moment of the same event) and continuity editing (wherein screen orientation is preserved across cuts) are techniques that prevent our disorientation. In VR, if cuts are necessary, it’s important to keep the viewer’s orientation in mind. If the subject of the viewer’s attention is in one position (we’ll call it “north”) before the cut, the next subject of the viewer’s attention should be within the same field of view after the cut. The last thing you want is for the viewer to have to look around after the cut, to feel ‘lost’ and unsure of what to look at. Some VR pioneers have been employing transitional elements to help guide viewers through a cut. A recent promotional VR video for Disney World utilizes “fairy dust” as a transition guide: the sparkling stuff fills the screen, and when it dissipates, we’re elsewhere, transported, and the pattern of dissipation guides our eyes towards the next thing that we should see. (see: “How to use 360 Starring Goofy”, Dekker Dreyer, Clever Fox). Fades, wipes and other transition techniques might emerge as common pieces of the new VR vocabulary.
The Close-Up: This, in VR, might not be possible. There are technologies emerging that allow viewers to “zoom in” on an object within a scene, but there’s no tool in the VR filmmaker’s toolkit at the moment to “force” the viewer’s attention in this way. If the viewer is “in the space”, that might be all we’ve got to work with, angle-wise. It’s possible that some clever VR pioneer might implement picture-in-picture techniques to highlight a detail, but aside from that, the filmmaker needs to draw the viewer’s attention using other tools. (Interestingly, we do have an interesting lesson in what doesn’t work as a VR equivalent to the close-up. In crafting the animated short “Henry” for Oculus Story Studio, director Ramiro Lopez Dau discovered that placing his character close to the camera doesn’t actually help us emote with that character in the way that it would if it were a close-up in a “flat” film. On the contrary, it feels “too close”, emotionally uncomfortable. He found that sad, intimate moments seem to play best at a bit of a distance. It’ll take more experimentation to figure out what that distance should be, what it means, and how to utilize it to the greatest emotional effect).
Depth of Field: There may be ways to utilize shallow depth of field, or a sort of selective focus, to draw attention within the VR field of view. If the “sides” of the image get blurry, for example, and the “front” remains in focus, it’s a way to force the audience to look forward. It’ll be interesting to see how VR pioneers implement this tool, as I imagine it could backfire if used improperly (it could be annoying, and snap viewers out of their semi-dream, rather than focusing their attention).
Framing, Staging and Lighting: Traditionally, filmmakers staged their characters with either planar (proscenium) staging (where the characters are all equidistant from the camera) or, more commonly, depth staging (where some characters are closer, some farther away.) With 3D production, some filmmakers have experimented with a variant of depth staging that I’ve called “portal staging”, where characters are viewed through a window or doorway, capitalizing on the sense of depth we get with 3D, and narrowing our visual field to guide our attention. Portal staging also draws on elements of traditional framing (wherein foreground objects are placed to “frame” an object or character to draw our attention to it). Traditional framing requires some tweaking before it can work properly in VR, and some consideration has to be paid to whether or not the VR film is 2D or 3D (foreground objects in 2D guide our attention to what’s beyond them, whereas those same objects in 3D become the objects of our attention!) Regardless of the technique, staging and framing are essential components of VR filmmaking, because it’s so much harder to capture and maintain the viewer’s attention. In a VR environment, the arrangement of objects and people is one of the first and most powerful indicators of where we should look.
Lighting in VR is particularly tricky, mostly because the 360 degree virtual space doesn’t allow us to hide our lights! That doesn’t excuse us from paying particular attention to lighting, as it’s probably the best tool in the arsenal for guiding the viewer’s attention in a virtual environment. Constructed sets will benefit here from the possibility of installed “practical” lighting elements. Wall sconces, chandeliers, recessed lighting and other elements will all become important parts of the VR lighting scheme.
On the subject of lighting and framing, a radical strategy presents itself. I draw once again from film history: D.W. Griffith once got in trouble with studio brass for shooting a close-up of his leading lady. Their complaint: ‘we’re paying for the whole actress, not just her face!’ As the story’s told, Griffith leaned in very close and said “can you see my feet?” Of course, they couldn’t. “I’m using what the eyes can see” Griffith explained. Using that reasoning, why not “use what the eyes can see”, and crop the virtual space?
Most people, when they settle down for entertainment (especially for a movie), they sit down, either in a chair, or on their sofa. Sometimes they’re even in bed. They look straight ahead. If they’re in a big movie theater, seated near the screen, they might look a little to the left, a little to the right… but never behind them. In fact, looking back can be an inconvenience - it can snap the viewer out of the dream-like state, can break, rather than preserve, the viewer’s narrative attention! When viewers watch Justin Lin’s “Help”, they tend to look “forward” most of the time, watching characters running away from a monster, or, if they’re standing, or sitting in a swivel-chair, they look “backward” most of the time, watching the monster approach. The constant swiveling back-and-forth doesn’t allow viewers to relax into the narrative, so they pick a side and stick with it.
To solve this problem (as well as a few other technical ones), VR cinema can get rid of that rear-facing camera, can fade the world to black around the edges. For the audience, this provides the immersive experience without uncomfortable effort. A 300 degree field of view feels the same as a 360 degree field of view if the audience never looks at the 60 degrees behind them. Furthermore, if we never have to turn around, we can settle more comfortably into the story. From a production standpoint, it allows a place to hide lights, to place a director, to position camera support and rigging. One of the VR hits from Sundance this year, "Sonar" (Philipp Maas, Dominik Stockhausen, Alexander Maas) already intuited this experience - when you look back, you see their company website - they tacitly acknowledge that there's nothing of the story worth putting there. There’s natural opposition to this idea, of course. The technology gives us 360 degrees to work with - why not use all of it? We’ve hired the whole actress, why just show her face?
Music and Sound: Much like in movie history, VR’s audio technology is a little behind the visual. Eventually, once it catches up, similar techniques to those used in movies will be used by VR pioneers to employ sound as a storytelling tool. Music might be used - much like it is in movies - as a unifying tool, making the pieces of a scene feel like a cohesive whole. Sound, of course, is a critical attention-management tool in movies, and should be used similarly in VR cinema. Eventually, “placed” sounds will be used to draw the viewer’s attention to a particular direction (a sound from the “east” might cause us to look in that direction, for example). If the VR environment is cropped, such that there’s nothing to see behind the viewer, including an ambient soundscape from behind might help solidify the sense of immersion.
The language of VR cinema is still emerging, and there are many questions that we have yet to ask. There are many more elements that can deeply affect and move an audience to either engage or disengage with the dream/story. And there are cinematic concepts that still need to be translated to the VR space. Can there be a “point of view” experience in VR without a shot of the character whose POV we’re about to experience? Should the audience be a “character” in the story, or a sort of disembodied observer? Whether it’s through physical production or academic analysis, I’m looking forward to seeing how these questions are answered, and to answering more of them, myself. We’re creating a new language, developing the syntax and grammar of a new way of telling stories. It’s an exciting time to be a storyteller.
