It’s been a while since I posted anything here, not because it has been quiet following the thesis defense, but because things have not slowed down at all in the last three months.  Today I’m taking a bit of time to catch my breath, and to catch this blog up on the newest scheme concocted in the labs of Team Tanenbaum.  Karen and I have been tossing this idea back and forth since early in the summer, but have recently resolved to try and run with it in earnest (note the copious use of ball-game metaphors).  Below, you will find two versions of the TUNE documents.  The first is a section out of a recent grant proposal I wrote.  It is focused on one aspect of my research within TUNE.  The second is an extended description of the project, which encompasses both my and Karen’s interests more fully.

Summary of the Proposal
Very few intelligent narrative systems have explored the potential of user modeling techniques from the field of artificial intelligence for adapting and personalizing narrative content to the desires of the reader. This research will develop new techniques for adaptive storytelling and games, and will evaluate the impact of those techniques on the experience of the interactor.  The vessel for this research is a fully-realized interactive story environment termed the Tangible Ubiquitous Narrative Environment (TUNE).

TUNE is a story, a space, a game, and a research instrument.  It investigates questions of interactive narrative, player modeling, adaptivity, and tangible embodied interaction. In TUNE, the participant is invited to assume a role in an interactive story through the exploration of a physical space that has been imbued with narrative possibilities through the use of ubiquitous and ambient computing.   As the participant explores the environment, the environment becomes aware of the participant and adapts the experience in response to her interactions and decisions, changing the lighting and sound in the room and triggering new events in response to her choices.  To succeed in TUNE, the participant must solve one of the historical riddles that are hidden within the space by combining objects, answering questions, and transforming the space until the room itself becomes the solution.

Through an iterative design process with ongoing participant studies, TUNE will be used to generate concrete design recommendations for future researchers and authors of interactive narratives and ubiquitous environments.  This research will demonstrate how user modeling techniques can be used to increase immersion in physical games.  These techniques will have practical utility for researchers in game studies and designers in the interactive entertainment industry.

Current level of graduate study:
I am in the first semester of the PhD program at Simon Fraser University’s School of Interactive Arts & Technology (SIAT).  I will be working under the supervision Dr. Magy Seif El-Nasr and Prof. James Bizzocchi, both of whom are prominent in Canadian Game Studies and in international research into games and interactive narrative.  At this stage in my program of study I am in the process of taking courses and formalizing my research trajectory.  I am currently taking a course in Artificial Intelligence, and plan to take courses on Mixed Methods in Design Research and Game Design next semester.  The curriculum at SIAT has afforded me the opportunity to pursue an interdisciplinary course of study that incorporates theory from New Media and Game Design research and techniques from Social Science, Artificial Intelligence, and Design Research.

My thesis proposal emerges from an intersection of the work that I have been pursuing over the last two years at SIAT.  My MA thesis described three analytical lenses for evaluating interactive narrative experiences in games, drawing on ideas from adaptive systems theory, improvisational theatre, and character believability research in AI and psychology.  In addition to my academic research, I have been active in the Vancouver area as an artist and designer.  While finishing my MA, I wrote and directed a number of short films for local and international film competitions.  I also participate in the local new media arts scene as a member of Upgrade Vancouver and DorkBot Vancouver, and this year I was a curator for the Vancouver New Forms Festival’s annual ArtCamp event.  In addition, I am a member of the Canadian Gaming Studies Association.  I have presented my work at conferences and workshops in Canada and abroad for the last two years.  Academically, I have also been involved in the organization of workshops and conferences in Vancouver.  My experiences in the games research and media arts communities have informed my interest in games and narrative experiences that support novel, personalized interactions within a physical space.

Thesis Proposal:
The Intelligent Narrative Technologies community is becoming increasingly interested in the notion of player preference modeling. My research investigates techniques for modeling the preferences of players in order to personalize an interactive narrative or game experience.  These techniques will contribute new analytical and design tools to the field of interactive narrative and games. Thue proposes a player preference modeling technique for interactive storytelling called delayed authoring (Thue et al, 2007); Natkin and Yan describe the use of user models for mixed reality gaming (Natkin & Yan, 2006); and Charles describes a technique for modeling player desire in games (Charles et al, 2005).  My own MA research took a critical look at user modeling systems for game design, and in the spring of 2007 I created a prototype interactive visual storybook that adapted stylistic elements of the narrative in response to a simple model of the user’s emotions.  Most research into this area is currently in the early discovery stages, with a number of prototype systems under development that explore the application of user modeling and recommendation technologies to interactive storytelling.  These include PaSSAGE (Thue et al, 2007) and Mirage (Seif El-Nasr, 2007), both of which attempt to personalize narrative elements by assigning players to a known archetype.  These systems show great promise; however, they are based in virtual environments instead of physical spaces, and have not empirically validated the use of user modeling to increase player immersion in an interactive experience.

I propose to create the Tangible Ubiquitous Narrative Environment (TUNE) as a physical experience, drawing on my background in film and theatre to create an installation space that immerses participants in a real, tactile interactive environment.  By placing the participant in a physical space, TUNE allows me to explore issues of embodiment and tangible interaction, and their relationship to immersion.  A fictional preamble to TUNE might look something like this:

“You are a psychic detective; a master of psychometry.  Objects reveal their stories to you, allowing you to peer into the past.  When you entered this room, the very walls called out to you.  A tapestry of stories lay encoded in the layers of dust on the objects in the room, and you knew that before the day was out you would have to relive the triumphs and tragedies of their long departed owners…”

TUNE will be a fully realized interactive narrative experience that supports immersion in a physical narrative environment, while adapting to the preferences of the user.  Henry Jenkins frames game design as “narrative architecture” and describes how the creation of a narratively saturated space can infuse any game with rich story (Jenkins, 2004).  By creating a detailed theatrical set for TUNE, I propose to extend this notion of “narrative architecture” into a physical environment that is more fully realized at an aesthetic level than most of the graphical and software based systems in current use in interactive narrative research.

Research Agenda and Methodology:

Phase 1 – Research: The first phase of this research will be to ground it in the overlapping disciplines that TUNE will be drawing on.  This phase will include experimenting with different tangible platforms and technologies, exploring several AI technologies for recommendation and user modeling, performing research into the stylistics of visual communication in theatre and film, and researching techniques for physical set and attraction construction.  Additionally, I will perform research into the Victorian Era, which will serve as TUNE’s historical setting.

Phase 2 – Building:  Phase two will involve authoring the various narrative threads that will be woven through TUNE, while also constructing the space in which TUNE will be set, and the AI systems that will drive the user model.  The demands of the story will guide the creation of the space and the constraints of the physical space and the tangible interactions will inform the direction that the story takes.

Phase 3 – Evaluation: Once TUNE has been deployed, I will perform the first round of user experience studies.  I will be using a mixed methods approach, drawing on techniques from design ethnography and grounded theory to shape my observations of how people interact with their environment.  Mateas and Stern propose a methodology for game studies that relies on an iterative design and analysis process, which they argue yields more insight than simply playing and studying existing games (Mateas & Stern, 2005).  This process is very similar to that used in interaction design research methodologies (Fallman, 2008).  TUNE will support a number of experimental conditions that will allow me to investigate how different choices, including the use of user modeling, effect the participant’s immersion in the narrative.  All user trials of TUNE will be video recorded for future review.

Phase 4 – Iterations: Phase four is a combination of phases two and three.  Informed by the initial evaluation phase and the video data gathered, I will iterate over the design of TUNE.  This revision phase will be followed by another evaluation phase.

This research will contribute to the interactive narrative community a set of techniques including user modeling and tangible and embodied interaction that can enhance a participant’s immersion in an interactive experience.  My research in TUNE will have implications for other fields including educational games, serious games, and interactive entertainment.

Charles, D., McNeill, M., McAlister, M., Black, M., Moore, A., Stringer, K., et al. (2005). Player-Centered Game Design: Player Modeling and Adaptive Digital Games. Paper presented at the DiGRA 2005 Conference: Changing Views – Worlds in Play, Vancouver, British Columbia, Canada.

Fallman, D. (2008). The Interaction Design Research Triangle of Design Practice, Design Studies, and Design Exploration. DesignIssues, 24(3), 4 – 18.

Jenkins, H. (2004). Game Design as Narrative Architecture. In N. Wardrip-Fruin & P. Harrigan (Eds.), First Person: New Media as Story, Performance, Game (pp. 118-130). Cambridge: MIT Press.

Mateas, M., & Stern, A. (2005). Build it to Understand It: Ludology Meets Narratology in Game Design Space. Paper presented at the DiGRA 2005 Conference: Changing Views – Worlds in Play, Vancouver, British Columbia.

Natkin, S., & Yan, C. (2006, 2006). User model in multiplayer mixed reality entertainment applications. Paper presented at the Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, Hollywood, California.

Seif El-Nasr, M. (2007). Interaction, Narrative, and Drama Creating an Adaptive Interactive Narrative using Performance Art Theories. Interaction Studies, 8(2).

Thue, D., Bulitko, V., Spetch, M., & Wasylishen, E. (2007). Learning Player Preferences to Inform Delayed Authoring. Paper presented at the AAAI Fall Symposium on Intelligent Narrative Technologies.

Thue, D., Bulitko, V., Spetch, M., & Wasylishen, E. (2007). Interactive Storytelling: A Player Modeling Approach. Paper presented at the 3rd Artificial Intelligence and Interactive Digital Entertainment Conference (AIIDE 2007).

TUNE – Tangible Ubiquitous Narrative Environment
Joshua and Karen Tanenbaum
TUNE is a story, a space, and a game. It investigates questions of interactive narrative, user modeling for informal learning, adaptivity, and tangible and embodied interaction. In TUNE, the participant is invited to assume a role in an interactive story through the exploration of a physical space that has been imbued with narrative possibilities through the use of ubiquitous and ambient computing. As the participant explores the environment, the environment becomes aware of the participant and adapts the experience in response to her interactions and decisions. TUNE uses a combination of tangible devices and tracking technologies to afford a diverse range of natural and transparent interaction possibilities, gathering data about the participant from explicit choices made within the space and from patterns of interaction that emerge over time.

TUNE’s primary valence of narrative adaptation is rooted in the notion of genre; a number of possible stories co-exist within the space, each in a distinct narrative style. The different stories told are tied together by the shared artifacts present in the space. While some artifacts may be strongly associated with one specific story, others have the possibility for a multitude of meanings spread throughout the different genres. As the participant explores the environment, the genre of the story she uncovers changes in response to her actions.

The stories are also tied together via a shared time frame: the Victorian Age. Each of the narratives takes place during this time and each one highlights a different aspect of that period. In this way, the environment also fosters an informal learning process as facts and information about Victorian times are absorbed via engagement with the narrative. For example, the “romantic” genre story might tell a tale of two lovers during the Victorian period. As their story unfolds via letters, photographs, and articles of clothing that the participant interacts with, details of period dress, social etiquette, cultural norms and morals, and societal structure are expressed. In contrast, the “action-adventure” genre story leads the participant through a tale of the British presence in India and the establishment during Victoria’s reign of the British Raj in response to the 1857 mutiny. In addition to informing the participant about historical events, authentic details about military practice, politics and trade are also conveyed.

TUNE communicates narrative information through a number of channels: the lighting and ambient audio in the environment change dynamically with the state of the system, digital artifacts in the space such as screens and projection surfaces display different information about the story, and tangible devices within the space reveal different information as the story progresses. Participants are placed directly into a narrative of exploration and discovery, but one which changes based on how they interact with the space.

TUNE as Research
Tune is a project at the intersection of a number of research questions.

User Modeling and Adaptive Personalization
Very few intelligent narrative systems have explored the potential of user modeling techniques from AIfor adapting and personalizing narrative content. TUNE will explore the utility of adaptive approaches to storytelling, but will be constructed so that we might run experimental controls between adaptive and non-adaptive versions of the system. We seek to learn more about how adaptivity impacts user experience in a ubiquitous space.

Informal Learning
One of the historical roles of narrative in culture has been to communicate information and experiences across generations. Narrative is often described as a sophisticated structure for the dissemination of complex cultural ideas. We are interested in how TUNE supports informal learning about a historical period by immersing the interactor in a narratively saturated environment.

Narrative Embodiment and Transformation
Interactive drama research often assumes that interactors take on a role within a game or interactive drama and perform that role from a first person perspective. We question this assumption, and propose TUNE as an environment for exploring an interactive drama where there is no bodily or cognitive separation between the interactor and her character.

TUNE as Experience

The following is a brief consideration of what a user interacting with TUNE might experience. As we are still in the conceptual stages of this project, this description should be considered to be an exploration of the problem space, and not a complete design document.

Anna approaches the door to the TUNE space. On a nearby rack hangs a selection of clothing items and accessories: a top hat, a pocket watch, a pair of long gloves, a toolbox, assorted coats and scarves, and a sword belt. Anna selects an outfit, placing the top hat on her head, and then enters the room.

Act I
The first thing she notices is a wooden desk with slender, sculpted legs sitting in the middle of the space. A chest of drawers sits off to one side against a wall and a bookcase leans up against another wall. On the wall in front of the desk is what appears to be a window, with a lace curtain pulled over the view, glowing faintly from the light. The room is not brightly lit, with the glow from the window and glass desk lamp providing most of the illumination. The desk, chest of drawers and bookshelf are all littered with items: books, photographs, jewelry, wine glasses, and candleholders. Anna explores the space, opening drawers on the desk, looking at books on the shelf, and picking through artifacts scattered on the table. Occasionally she will pick up an object and a voice will whisper stories in her ear; a dried rose tells her a story of how it was left behind by a hopeful suitor; a candelabra reveals that it last shed light on a battle plan for dealing with a local uprising. Picking her way through the space, Anna comes to a photo frame with a number of pictures before it. Placing a photo in the frame causes its story to be told, but as she looks up, she sees that the view through the window, though indistinct, has shifted. With each photo a new story is revealed and a new landscape unveiled behind the curtains. Anna selects a photo of a Victorian engineer, working on a steam engine. Outside the window, a trainyard can be perceived, and Anna hears the sounds of a crowd on the platform. Anna continues to explore the room. Sometimes objects she interacts with have an obvious impact on her environment,
while other times, objects that seem like they ought to have a story to tell do not reveal any information until she brings two or more of them together.

Act II
Suddenly Anna hears a metallic jingling, followed by the sound of something metal falling. She looks toward the source of the sound and sees an iron wrought key rocking slightly on top of the chest of drawers where it has just fallen. She picks it up and looks around for a lock. She finds that it fits in one of the desk drawers, and opens it to find a new collection of objects: a handwritten envelope sealed with wax, a kaleidoscope, and a new picture to place in the frame. These objects have a story to tell as well, but it is a different type of story. Where her previous interactions had provided her with vague pieces of stories, often unconnected, these objects are components of a narrative puzzle. The items in the drawer have a story they need to tell her, and as she begins to piece together their tale this sense of urgency pervades the space. The lighting becomes more dramatic. A musical score fills her ears. Anna returns to previous objects she had interacted with, only to find that some of them have different information for her now. Where before there were a multitude of stories in the space, now this one story has taken over. Anna learns that there is a task unfinished, a problem that she must solve, before the story can end. The tension in the room mounts, and Anna wonders how she will be able to solve this mystery. The spirits which have possessed the room reach a fevered pitch, howling around Anna, demanding release.

And then, silence. Silence punctuated by a creaking sound as the bookshelf swings open to reveal a short hidden passage. Anna follows this passage into a smaller room which is bare except for a round table with an old fashioned scale in the center of it, sitting in a pool of light from a single hanging lamp. Windows in the room’s walls are darkened, revealing nothing “outside”. In order to complete the story, Anna must interpret the clues given to her in Acts I and II, and find or assemble the item (or items) that must be placed on the scale. Returning to the first room she grabs the kaleidoscope and takes it to the measuring room. She places it on the scale, but she has chosen poorly. Outside the windows, red light
begins to lick upwards, like flames, and a hint of the cacophony that she had heard at the end of Act II returns. She carries on, selecting different objects and experimenting with different combinations. Correct choices move the ambiance of the room and the landscape towards a more pleasant state, while incorrect choices result in mounting chaos. Finally, Anna assembles the items needed to complete the story, and is rewarded by a dramatic transformation of the space as the final steps of the tale play out.

Thus ends our brief introduction to TUNE.  Dare I ask you to stay TUNEd for the next exciting installment of our story?