Physical World News In Virtual Spaces: Representation and Embodiment in Immersive Nonfiction

by Nonny de la Peña

[PDF version]

The use of spatial news and documentary, in which immersive virtual reality constructs of nonfiction are experienced through an embodied digital representation such as an avatar, is a nascent but growing field. By putting the participant into a subjective first-person position and delineating spatially where the audience will encounter editorially-controlled content, these practices raise entirely new questions about the accuracy or impartiality of the nonfiction narrative. Deliberations are certain to grow more heated as this use of the virtual body and immersive virtual reality complicates considerations on what constitutes a subjective relationship to narrative. However, fears over whether the embodied nature of an immersive news piece undermines journalistic integrity because of that embodiment are misplaced. Instead, these virtual reality constructs should be considered in the same light as any documentary or news report, with the relevant factor being the transparency surrounding the sources and research material used to support the factual underpinnings.

The fundamental idea of immersive nonfiction is to allow the audience to actually enter a virtually recreated scenario representing the story. The pieces can be built in persistent online virtual worlds such as Second Life or as a web-based Unity 3D construction. They can also be produced using a head-tracked head-mounted display system, a lightweight helmet with screens that cover the eyes called a head-mounted display (HMD). The helmet tracks head movement to ensure that digital imagery on the screens stays in perspective in order to create the sensation of having a virtual body in a virtual location. Another option is the Microsoft Kinect, an inexpensive camera that follows body movement, released as part of the Microsoft Xbox system. The Kinect has now been linked to avatar movements in multiple 3D environments. Finally, immersive nonfiction can also be constructed in a Cave, which uses full body-tracking technologies in a small room whose walls show relevant video that corresponds to how a user moves their bodies around the space.

Video and audio captured from the physical world are used to reinforce the concept that participants are experiencing a nonfiction story. For example, video triggers at key points in the virtual landscape remind a participant that the computer-generated environment is grounded in a real news story.

Embedded video reinforces the news narrative

Scripted events that create a first person interaction with the reportage can also help create a feeling of “being there.” By using programming to move participants from one virtual location to the next, thus moving the embodied presence along the timeline, the experience employs “embodied edits.”[i]

These experiences blur the line between recreations and cinema verité because audio and video captured live from the physical world (cinema verité material) is embedded throughout the computer-generated environment and encountered in scripted/programmed variations. The computer-generated environment also can act as a replica of an actual space from the physical world, a recreation but with verité intent. Finally, whether visiting the space as oneself or as a subject of the story, immersive constructs aim to afford the participant unprecedented access to the sights and sounds, and possibly, the feelings and emotions that accompany nonfiction stories.

Objectivity/Subjectivity in Nonfiction

When considering a nonfiction narrative, both documentary and news stories purport to represent some version of reality. In John Grierson’s famous definition of documentary as “the creative treatment of actuality,” it is the term “actuality” that most fundamentally describes the creator’s relationship to the content.[ii] Whether that “actuality” is boiled down in a short news piece or depicted at length in a documentary film, there is an attempt to adhere to some version of what the creator considers to be truth. All the formats portray the creator’s view of reality and act as “knowledge-producing and communicative practice[s].”[iii] This shared goal binds the genres when considering how the portrayal of nonfiction might be affected by using virtual reality platforms instead of formats like film, video, photographs, or text.

Moreover, the debate around whether the creator of any nonfiction can ever meaningfully represent truth is decades old. Arguments range from a sense that objectivity can be “operationalized”[iv], to a belief that “truth and objectivity claims of new journalism [are] totalitarian struggles in an ongoing power struggle,” a truth which Dirk Eitzen suggests should always be measured by the question, “Might it be lying?”[v] One can also argue that recent attempts to tell both sides have descended into a sort of tyranny of objectivity that seems to deter critical thinking about the facts presented by either side.

Certainly raw video can offer an extraordinary window onto events, but edits in the footage and the selection process of interviewees are just two obvious ways that the narrative can be manipulated to merely appear to present events fairly. In comparing two Danish documentaries critical of the pesticide company, Cheminova, Olesen claims that even “investigative journalism is a political act within the boundaries of professional journalistic standards.”[vi]  In fact, columns like the one run by the Los Angeles Times from 1936-1941, which supported social eugenics, illustrate the fallacy that journalism has ever been editorially sophisticated enough to work outside of the influences of culture.

The difficulty of holding nonfiction to a purist standard also becomes clear when examining Walter Cronkite’s documentary series You Are There (1953-1957). Cronkite is still considered one of the United States' most reputable journalists and someone who would have adhered to the asserted necessary journalistic tenets of “truthfulness, accuracy, objectivity, impartiality, fairness and public accountability.” However, the You Are There series exposes the weaknesses in any claim of having achieved those tenets, instead reflecting the era in American history in which it was created. For example, his piece on the Alamo addresses the American view on the battle, giving little of the Mexican perspective about the United States’ annexation of Mexican territory that would have made the presentation more balanced.[vii]

Interestingly, actively seeking out an acceptable balance seems to be losing its relevancy for some U.S. news organizations. For example, Salon blogger Nick Leshi writes that a local TV station in New York, FOX-5 (WNYW): 

Doesn’t even bother to call their on-air talent “reporters” or “journalists” anymore—their Website lists them as "personalities," like Anne Craig who covers entertainment in giddy fan-girl fashion. It even fills its airtime with comedians roaming the streets of New York videotaping passersby—no real news value, just laughs.[viii]

Such concessions to the limitations of covering nonfiction are limited, however, and impartiality issues still resonate with most nonfiction content creators. While newspapers in countries like Great Britain sidestep journalistic objectivity issues by offering a variety of papers with editorial viewpoints based on particular political positions, this is not acceptable for the government-funded BBC. The average American city may have multiple news stations, but with only one newspaper, the print publications must purport to be fair-minded and dedicated to presenting well-researched and factual content.

A recently touted solution among leaders in the field, such as Geneva Overholser, who oversees news and documentary film education at USC’s Annenberg School of Journalism, is to shift the culture away from the idea that the individual journalist can act as a frictionless pipe, replacing it instead with a necessary landscape in which the news report offers “transparency.” Where does the information come from? Who are the sources? Such openness is intended to help the audience judge for themselves whether the story and the analysis are valid.

This concept of transparency makes most sense in the linked world of the web, but it becomes a very different challenge for spatial nonfiction that must consider seamless integration of data into landscape and will no doubt add to the controversy that will greet such immersive nonfiction narratives. Moreover, the science behind our connection to our virtual bodies is uncharted ground for journalists and documentary filmmakers. They may understand the compelling nature of a good story and have always tried to make their audience relate to the narrative, but creating that relationship through 3D experience is a unique challenge. The feelings that may be generated through a connection that shares similarities to being in one’s real body can create confusion on how to approach the objectivity/subjectivity debate. Answering Eitzen’s question “Might it be lying?” with an automatic “yes!” simply because the nonfiction exists in virtual reality would be a mistake.

Connection to Virtual Embodiment

Using technology as a medium for human interaction has been in play since at least the invention of the telegraph when operators sent jokes, formed friendships, and even got married down the wires.[ix] The telephone evolved rapidly into a medium for social connection with demonstrable effects on behavior and sentiment. One study found that when males believed they were speaking to an attractive female on the telephone, they became more pleasant. When spoken to in such a manner by the males, the females turned on the charm in return. Similar relationships and personal investment in text-based virtual worlds have been detailed extensively in such books as Sherry Turkle’s Life on the Screen (1995). [x] Perhaps nothing captured this connection to the technological “self” more clearly than when Julian Dibbell published a piece in the Village Voice in 1993 describing the virtual rape of avatars in LambdaMOO. Individuals behind those avatars claimed to feel raped even though nothing had happened to their physical bodies.[xi] 

In 1997, Judith Donath attempted to distinguish between our real world personas and virtual ones. She wrote in “Identity and Deception in the Virtual Community”:

In the physical world there is an inherent unity to the self, for the body provides a compelling and convenient definition of identity. The norm is: one body, one identity. Though the self may be complex and mutable over time and circumstance, the body provides a stabilizing anchor.

Said Sartre in Being and Nothingness, “I am my body to the extent that I am.” The virtual world is different. It is composed of information rather than matter. Information spreads and diffuses; there is no law of the conservation of information. The inhabitants of this impalpable space are also diffuse, free from the body's unifying anchor. One can have, some claim, as many electronic personas as one has time and energy to create.[xii]

However, since Donath wrote her paper, virtual identities that were once constructed solely through text have now become 3D representations. In fact, it seems Donath’s assertions of being free from that “unifying anchor” may have a different truth when considering the relationship to 3D avatars in new virtual worlds, even if users experience a variety of embodiments. Whatever intangible connections were present in previous technologies have been amplified by the visual representations on the computer screen into something visceral.

There are multiple studies that directly connect the sense of one’s physical body to the virtual one. In one experiment, subjects wearing video display goggles saw a virtual rendition of their body in front of them and they were stroked on the back at the same time as they saw their virtual body being stroked. Subjects reported that the sensation made them feel “as if the virtual body was their own body.” The video display was then turned off so it now acted like a blindfold, and subjects who had been moved from their original spot were asked to return to where they had been standing. They invariably moved closer to where they had perceived their virtual selves rather than where they had actually been standing.[xiii]

In another experiment, subjects saw their own back through stereoscopic video display goggles making them feel as if they were sitting behind themselves. When their chests were touched with a plastic rod at the same time as the rod “stroked” the area of their virtual chest, subjects felt the sensation as if their bodies were now located at their viewpoint. This was despite the fact that their real body was in plain view. When a researcher brought a hammer down toward the virtual body, subjects registered a threat that was measured both through skin conductance electrodes and self-reports of feeling anxiety.[xiv] In their minds, they now occupied the space behind their bodies rather than their actual location, since that is where they “saw” themselves to be.

This connection to the virtual body, to what we see rather than where we are, adds another dimension to how we determine “self.” In fact, it seems that we are hardwired to adopt representations of ourselves as real. These studies underscore the importance of our perception about what is happening to our physical representation. They also offer a critical explanation of why an avatar becomes relevant so rapidly to users and why they become invested in the experiences of their particular avatar.

Experiencing a “Camp Xray Cage” in the Virtual Guantanamo Bay Project

Additionally, one of the most remarkable aspects of immersive virtual environments and why it is so applicable to nonfiction is that people tend to respond realistically to virtual situations and events even though they know that these are not real.[xv] Even more surprisingly, this response-as-if-real (RAIR) occurs even though the level of fidelity with respect to everyday physical reality is severely reduced—including visual appearance, the realism of computer graphics rendered scenes, the realization of physics, and above all the representation and behavior of virtual humans.[xvi] This underscores the potential for virtual reality to become an important new palette for generating immersive nonfiction pieces. While story and narrative must be rethought as being “spatial” and the content itself must always designed to be mediated through the avatar or some sort of embodiment, these environments offer a documentary space in which the public can feel that they are “there” when witnessing or experiencing a story.

Finally, what transpires in these virtual spaces occurs in real time and can reflect nonfiction narratives playing out in the physical world. Also, like their physical world counterparts, the digital events can be fleeting.  For example, within days of the Israeli incursion into Gaza in 2008, Doron Friedman, from the Advanced Virtuality Lab at Samy Ofer School of Communications IDC Herliza, logged onto Second Life’s virtual Tel Aviv to test a new graphics card. He was immediately targeted by the shouts of protesters and the sounds of breaking glass as if bottles were being thrown at him. Had Friedman not filmed his experience using machinima, the documentation of his experience would not exist.[xvii] Moreover, his digital representation was under threat and although he knew he could physically be assaulted he says, “It was powerful in the sense that there was definitely the feeling that there were people there who hate you and they are expressing their hate in a nonverbal way.” Interestingly, he was able to ultimately converse with several protesters, something that would have been extremely unlikely in the physical world and a moment worthy of documentation.

Contrast to Documentary Games

Immersive nonfiction and immersive journalism is often compared to news or documentary games because the pieces are typically set in what has been previously solely the terrain of gaming platforms and relies on computer generated graphics. However, there are a number of distinctions, particularly to news games, with the most important being that games work best as systems. Games are better at reproducing the conditions under which events unfold rather than outlining the details of the events themselves. That means linear narrative structures or presentation of multiple specific facts, which can be key to nonfiction, do not work as well with a gaming setup. Often, players advance through the game by passing “levels” that do not necessarily relate to the inherently unchangeable nature of a nonfiction narrative (no matter from whose perspective it has been constructed).

One excellent example is Cutthroat Capitalism (2009), a news game appended to Wired magazine’s story on Somali pirates. Since the audience is asked to become a Somali pirate, a deeper understanding of the economics of the highjack-and-ransom system is achieved through the play. However, it does not attempt to delineate any individual case in particular, and what happens to the player is based on his/her choices rather than reflecting the facts connected to one or more physical world events that have already transpired. Also, in contrast to immersive nonfiction, there is no virtual embodiment.

The difference between immersive nonfiction and documentary games, however, is more difficult to tease out, especially given that there has yet to be a body of research to help determine what constitutes news games versus documentary games. In Tracy Fullerton’s piece on documentary games, she offers a number of examples of games that recreate the events of Pearl Harbor, allow a player to reenact John Kerry’s Vietnam swiftboat tour of duty, participate in a recreation of real Iraqi war scenario in which Saddam Hussein’s sons were killed, or “play” a 9/11 victim.[xviii] As Keith Halper, CEO of KUMA games, notes: “Games allow us to be who we are not, to do what we cannot, to be in places and times we cannot go… Through entertainment and action, we can educate in novel and powerful ways.”[xix]  However, each of these examples has shifting narratives—the player does not necessarily kill Kennedy or follow the exact documented path of any particular victim of 9/11. Even the swiftboat exercise offers a player to variations on the story. This may be a key way to clarify the difference—using an embodied experience in an immersive and unchangeable narrative that allows queries to the environment without changing an individual’s story trajectory is more in keeping with traditional journalistic or documentary practice.

Consider our project using an HMD in which we put individuals in a virtual body of a detainee in a stress position.

Using a head mounted display to create an embodied news experience

The individual in that experience heard an audio of a real transcript of an interrogation read by actors as if it was coming from another room. This was a recreation of a real event and the experience proved to be journalistically accurate—after this immersive piece was complete, the British government released a video associated with the court trial over the death of civilian Baha Mousa.

The video helped establish the veracity of this particular immersive journalism construction, which had relied on Freedom of Information Act reports and International Red Cross descriptions to inform the design. Yet the experience does not allow the user to access the documents or other substantiation that were the factual underpinnings of the design. It leaves the piece open to criticism for not offering a curation system of sources that could reflect the transparency that has become so relevant for traditional journalism even as it attempts to tell the nonfiction story with an accurate relationship to the physical world events.

Final Notes

To confuse a subjective experience with subjectivity would be an error. Such an argument is as specious as to claim that true objectivity currently exists or has ever existed in traditional documentary films or news presentations. As noted in the PRESENCE article, “Immersive Journalism: Immersive Virtual Reality for the First Person Experience of News”:

While we are accustomed to viewing video, images, and audio recordings as faithful duplicates of reality, we know that in many instances they are not. It has now become relatively simple to fake photographic images and even video footage using free software that can be obtained online. Such fakes have been distributed and sometimes even generated by leading media outlets. … [Moreover,] a possible objection to immersive journalism may be that it may strain the credibility of journalistic integrity, undermining the ability to bring the “true facts” to the public. … [W]e claim that, perhaps unintuitively, the opposite may be true. Immersive journalism does not aim solely to present ‘the facts’ but rather the opportunity to experience “the facts.”[xx]

While embodiment offers what may be a new way to encounter the nonfiction story, it in itself does not eradicate the meaningful representation of facts present in a quality news story. Instead, current calls for better transparency in all nonfiction narratives are more relevant and should be the standard upon which criticisms of immersive experiences are based.

Notes

[i] Peggy Weil and Nonny de la Peña, “Avatar Mediated Cinema” ACM International Conference Proceeding Series; 352 (2008): 209-212.

[ii] John Grierson, “The First Principles of Documentary,” in Grierson on Doumentary, ed. Forsythe Hardy (London: Faber & Faber, 1966), 147.

[iii] Mat Ekstrom, “Epistemologies of TV journalism: A theoretical framework,” Journalism 33 (2002): 259–282.

[iv] Jorgen Westerståhl, “Objective News Reporting: General Premises,” Communication Reseach, 10, no. 3 (1983): 403-424.

[v] Dirk Eitzen, “When Is a Documentary?: Documentary as a Mode of Reception,” Cinema Journal,  35, no. 1 (Autumn, 1995): 81-102.

[vi] Thomas Olesen, “Activist Journalism? The Danish Cheminova debates, 1997 and 2006," Journalism Practice, 2, no. 2 (2008): 245 – 263, (247).

[vii] When I have shown a clip of this piece to audiences, they often laugh out loud at the simplicity in Cronkite’s presentation.

[viii] Nick Leshi, “The Death of Objectivity in Journalism,” http://open.salon.com/blog/kikstad/2011/03/11/the_death_of_objectivity_in_journalism.

[ix] Thomas Standage, The Victorian Internet: The Remarkable Story of the Telegraph and the Nineteenth Century’s On-line Pioneers (Walker Publishing Company, 1998).

[x] Sherry Turkle, Life on the Screen: Identity in the Age of the Internet (New York: Simon and Schuster, 1995).

[xi] Julian Dibbel, “A Rape in Cyberspace” The Village Voice (December 21, 1993): 36-42.

[xii] Judith Donath, "Identity and deception in the virtual community," in Communities in Cyberspace, ed. M. Smith and P. Kollock (New York: Routledge, 1998), 29-59.

[xiii] Bigna Lenggenhager, Tej Tadi, Thomas Metzinger, and Olaf Blanke, “Video Egro Sum: Maniplating Bodily Self-Consciousness,” Science 317 (2007): 1096-1099.

[xiv] Henrik Ehrsson, “The Experimental Induction of Out-of-Body Experiences,” Science 317 (2007): 1048.

[xv] Mel Slater, “Place Illusion and Plausibility Can Lead to Realistic Behaviour in Immersive Virtual Environments,” Philos Trans R Soc Lond 364 (2009): 3549-3557.

[xvi] Nonny de la Peña, Peggy Weil, Joan Llobera, Elias Giannopoulos, Ausiàs Pomés, Bernhard Spanlang, Doron Friedman, Maria V. Sanchez-Vives, and Mel Slater. “Immersive Journalism: Immersive Virtual Reality for the First Person Experience of News,” Presence: Teleoperators and Virtual Environments 19, no. 4 (2010): 291 – 301.

[xvii] See “Gaza Protest in SL - conflict resolution in virtual worlds?” http://www.youtube.com/watch?v=qFWZgD-To_A.

[xviii] Tracy Fullerton, “Documentary Games: Putting the Player in the Path of History,” in Playing the Past: Nostalgia in Video Games and Electronic Literature, ed. Zach Whalen and Laurie Taylor (Vanderbilt University Press, September 2008).

[xix] Keith Halper, "Kuma Games Press Release," http://ww.kumagames.com/press.

[xx] De la Peña, et al.,“Immersive Journalism,” 301.

Nonny de la Peña is a Senior Research Fellow at the USC Annenberg School for Journalism and Communication where she has been exploring Immersive Journalism, a novel way to utilize gaming platforms and virtual environments to convey news, documentary and non-fiction stories. Her recent projects include, “Gone Gitmo,” a virtual Guantanamo Bay Prison in Second Life, which was prototyped with funding from the MacArthur Foundation and employs first person experience and spatial narrative. Another project “IPSRESS” investigates the use of head mounted display technology as a means to evoke feelings of presence in reportage. Using her twenty-years of journalism experience, as a former correspondent for Newsweek Magazine and as an award-winning documentary filmmaker, she repeatedly pushes technological boundaries for reporting, including creating the collaborative video remix site Stroome.com.