Thesis: Perception

Mapping and Re-focusing

This week, I drew a map of the subjects of interest to my thesis and how they relate to each other. It seems as though the idea of perspective or viewpoint shifting is not what I’m most interested in. I thought back to a starting point of my project: the idea of turning a clever visual comparison into an interactive experience, and the idea of representing the wonder I felt at watching animated credit sequences in the context of a game.

Previously, I was focusing a lot on the idea that the shifts were switching between different spatial relationships, but I wasn’t doing a lot of research on how this was being accomplished.

I want to understand what happens when one is continually experiencing transitions that make use of the fact that the game is a 3D space represented in 2D. (Transitions that may require one to understand the space as 3D, then as 2D, then as 3D again.) How does this change the interpretation of the space and the general experience of the player? What is the difference between achieving this in a 2D medium vs. an interactive 3D one?

Shigeo Fukuda

Recent research on the history of minimal poster art led me to Shigeo Fukuda. He was an artist known for creating posters emphasizing anti-war, using illusion and simplicity (Heller, 1). Fukuda’s posters have a lot of visual strength. They are simple and evocative. They remind me of my fascination of visual minimalism in posters and comics and how they can create a very strong sense of a world. (My next post will be about how story may or may not fit in with the project)

shigeo-fukuda-victoryshigeo-fukuda-1975shigeo-fukuda-amnesty

Left: Fukuda, Shigeo. Victory 1945. 1971. N.p. Winner of the Warsaw Poster Contest
Middle: Fukuda, Shigeo. Exhibition Keio Department Store. 1975.
Right: Cooper, Clay . End gun violence initiative for the AIGA. 2013. N.p.

Visual Perception

Visual perception is a large area of study. There are a few books from the OCAD library I’m reading to get an idea of the key theorists and to find the most relevant sub-areas. From my research so far,  I can guess that these areas are:

  • Perception of 3-dimensional shape
  • Perception of space
  • Visual illusion and cognitive biases
  • Ambiguity vs. specificity of visual information

One of the books is called 3D Shape : Its Unique Place in Visual Perception, written by Pizlo Zygmunt, a professor from Purdue University (Indiana). Zygmunt includes a history in the theoretical understanding of the perception of shape, including key theorists such as Hermann Von Helmholtz.

neckercube

A Necker cube is a good example of a multi-stable image (I’ve written more about multi-stable perception in a previous post), and is often used when talking about visual perception. I found a relevant paper by psychologist Richard Gregory on perceptual illusions. Around page 172, Gregory differentiates between optical/sensory illusions and perceptual illusions, which he explains come from “misinterpretation by the brain of sensory information”. He includes images of the Necker cube by Swiss crystallographer and geographer Louis Albert Necker, who described it in a letter as “a rhomboid (which) reverses in depth, sometimes one face appearing the nearer, sometimes another.” (172).

In addition to this research, I am taking a class on cognitive science, where a lot of these ideas are discussed.

Games

Have people been researching or theorizing about how ideas in visual perception apply to games or interactive 3D experiences?

Lev Manovich’s The Language of New Media has a chapter called Illusion, Narrative, and Interactivity where he compares different types of media and the types of illusion they create. Manovich explains that in 3D game worlds, elements such as the GUI appearing after cut-scenes and  “level of detail” (where objects gain lower poly models based on their proximity to the player character) constantly reveal the machine behind the illusion (206). Manovich explains that “as the user navigates through space, the objects switch back and forth between pale blueprints and fully fleshed out illusions. The immobility of a subject guarantees a complete illusion; the slightest movement destroys it” (206). Questions about suspension of disbelief could connect theories of visual perception with game design. Challenging the way a player perceives the game world, a designer may wonder how their relationship to it differs from other games, and what affect this may have on things like engagement or spatial presence.

Manovich’s book is also a link between games and film. For example, he goes on to compare the experience of illusion in interactive media with that of traditional film, which he claims “aims at all cost to maintain the continuity of the illusion for the duration of the performance” (207).

I’m still looking for more research that could link visual perception ideas with game theory or design ideas.

Works Cited

Gregory, Richard  “Perceptual illusions and brain models” Proc. Royal Society B 171 179-296. Published to web January 3, 2008. Web. January 18, 2016.
Heller, Steven. “Shigeo Fukuda, Graphic Designer, Dies at 76.” The New York Times. The New York Times, 19 Jan. 2009. Web. 18 Jan. 2017.
Manovich, Lev. The language of new media. Cambridge, MA: MIT Press, 2002. Print.
Pizlo, Zygmunt. 3D shape: its unique place in visual perception. Cambridge, MA: MIT Press, 2010. Print.
Advertisements

Thesis: Shrinking

(Updated to include videos)

In the prototype for the Dec. 1st critique, I tested two different kinds of transitions where:

  • It seems as though you are changing (in this case, shrinking) relative to a space
  • It seems as though the environment is changing (in this case, growing).

From here, I can observe how peoples reactions may differ. I’m especially interested in their:

  • Sense of control
  • Orientation/disorientation
  • Spatial understanding/confusion

 

soupstoryboard.png

The first transition was technical improvement on the lake shift. It took a frustratinghallstoryboard.png amount of time (and lessons from friends) to figure out the Unity side of this. The transition still isn’t completely smooth. For example, sometimes you will see the edge of a platform appear beneath you. However, I now have a better understanding of how to affect the camera script, which has unlocked possibilities for other approaches to scene changes.

I also added a script making the hall and sky change colour as the player walks forward, essentially giving them control over the time of day. The next step to strengthening the shift between the school and the mystery book world is sound and shadows.

 

 

hexroomstoryboard
The white sections are invisible meshes Unity uses to know where the player isn’t looking. (Something tricky will always happen behind them)

Creating the second scene also required learning a new tool. Instead of using triggers to see where the player is looking and standing, I’m using meshes to see if the player is not looking at a spot (a spot where something needs to happen). This method is more straightforward and does not rely on a specific screen size. In this scene, the space changes shape and grows larger as the player walks and looks around inside. It uses 7 different room models, positioned in the same spot.

Later, I may add in a small time delay to ensure a gradual transition that prevents changes from happening all at once. Delineating the walls is another issue. I used a light for this demo, which does not achieve the flat shaded look in the other scenes.With the shape of these 7 rooms, it is impossible to follow a dark-light-dark-light pattern of “artificially” lighting the walls. A solution may be to add features such as windows or decorations to each wall.

 

 

 

 

 

 

Thesis: Making Spaces

Screen Shot 2016-11-24 at 1.02.39 AM.png

Here is some furniture I created for the inside of the cracker/raft house. The objects are less informed by research and more informed by playing around and seeing what looks good. Sometimes weird proportions can make the item seem oddly cute and handmade to suit the needs of the person that uses it. I’m finding it really enjoyable to try to make objects that bring up questions and generate story.

Screen Shot 2016-11-24 at 5.37.05 PM.png

It’s easy to get attached to certain pieces that don’t fit in with the other objects. I was conflicted between some more traditional looking pieces and some more modern ones.

Screen Shot 2016-11-24 at 7.57.13 PM.png

I did a lot of research on specific historical interior design styles. Using this research, I’ve been modeling and getting feedback from others to try to create settings that evoke the feeling of being in a distinct era. It’s harder to find references for, say, Europe in the 20s, that show the homes of the less wealthy, but the OCAD library is a good resource for books that at least write about the home-lives of general populations.

Painting, data and music

I felt like I was in over my head taking this traditional painting course at OCAD. Fortunately, our instructor has a more open definition of painting.

I’d spent a lot of time learning a Mendelssohn song on the piano, practicing it again and again to focus on each layer of information in the sheet music (notes, volume, pedal, etc.) I thought it would be simple to translate the sheet music into a data set on excel. It ended up being more complicated. How do you put right hand and left hand notes in the same table when there are different amounts of each? How do you show markings that refer to groups of notes or that occur in the spaces between notes?

In Processing, I took data from:

  • pedal markings
  • note pitch
  • note length
  • volume (eg.: mp, p, f)

and turned them into:

  • line continuity
  • line curve
  • line length
  • line thickness and colour

Screen Shot 2016-11-15 at 10.25.29 AM.png

(An early test)

Instead of creating a line graph to represent music, the program “paints” a line one segment at a time, and the line curves and twists according to the relationship between each note and its predecessors. To me, this method of showing the data is more representative of the way we listen to music.  We usually don’t have perfect pitch, so we use the relationship between one note and the next to understand what’s being played. We pay attention to this relative pitch as much as we pay attention to the global pitch.

Screen Shot 2016-11-15 at 9.19.44 AM.png

This image is generated from one line of music, divided into the melody and the harmony. The shapes remind me a little of photographs of microscopic organisms.

Screen Shot 2016-11-24 at 11.16.15 AM.png

I adjusted the code to be able to include more accurate information. In both, you can see that when the line curves very smoothly, notes are progressively going up or down. Repeating patterns in the music can be see in through patterns in the line.

Screen Shot 2016-11-17 at 1.15.24 PM.png

The chunky line above on the left is something I generated after having figured out how to choose notes that would create interesting shapes in the line. It does not use data from a real piece of music.

It may be interesting to use a keyboard and generate these graphics in real time to see how someone might improvise when they have this added visual feedback.

 

Thesis: Related Games

 

This post is formatted somewhat like an annotated bibliography, listing games that relate to relevant concepts such as:

  • the impossible object
  • paradox
  • four dimensional space
  • optical illusion
  • forced-perspective
  • and non-euclidean geometry.

Antichamber

Antichamber (Demruth, 2013) is puzzle based and plays with non-euclidean spaces (space is not mapped flatly but can connect to itself in weird ways, making things like portals possible). Spaces change behind the player when they go around a corner. It is hard to watch play throughs without getting motion sickness. This is likely because the game is about looking around, and someone else is moving the camera to try to make us understand the puzzles.

The game has a minimal shading style, the ambiguity of which relates to multistable perception, which is what happens when “a single physical stimulus produces alternations between different subjective percepts. Multistability was first described for vision, where it occurs, for example, when different stimuli are presented to the two eyes or for certain ambiguous figures.” (Schwartz, 1). An example of multistability is Rubin’s face/vase image (where the multiple interpretations consist of a picture of  either two faces or of one vase).


Antichamber. [Alexander Bruce]. Demruth. January 31, 2013. Video Game.

Website

Schwartz, J.-L., N. Grimault, J.-M. Hupe, B. C. J. Moore, and D. Pressnitzer. “Multistability in Perception: Binding Sensory Modalities, an Overview.”Philosophical Transactions of the Royal Society B: Biological Sciences 367.1591 (2012): 896-905. Web.  http://rstb.royalsocietypublishing.org/content/367/1591/896

Perspective

There is literally a game called Perspective that makes use of (and constantly communicates) the fact that 3D games are 3D digital spaces mapped to 2D for screens. Perspective (DigiPen) is a platform jumping puzzle game where you can do things that are 3-dimensionally impossible, but that make sense if you make your eyes go blurry and try not to see depth. (The website has a helpful video).


Perspective. [Jason Meisel, etal] DigiPen Institute of Technology. Video Game.

Monument Valley

Similarly, Monument Valley (Ustwo, 2014), makes use of the fact that it is 2.5D (fixed isometric views) to show impossibilities (such as the Penrose triangle) that are usually part of the challenge of the game. Monument valley is similar to M.C. Escher’s implying 3D paradoxes by mapping them in 2D, and is aesthetically similar.

Escher Waterfall.jpg


Escher, M.C. Waterfall. 1961. Lithograph. N.p.

Monument Valley. Ustwo. April 3 2014. Video Game.

Website

Museum of Simulation Technology

A few years back, I saw a demo for a game in development that is now called Museum of Simulation Technology (Pillow Castle). It is a game where you walk through a series of rooms solving puzzles that are often based around forced perspective. You might pick up a tower that is large that is  far away, and in your hand, have it be as small as it looked in the distance; a souvenir of a  tower. Pillow Castle has an updated demo you can watch to better understand this concept.


Museum of Simulation Technology. Pillow Castle Games. Under development. Video Game.

Website

Fez

I bought Fez a few years ago and was really inspired by it. My favourite part was exploring the little town at the beginning (I preferred this to doing the actual puzzles, which probably says something about my relationship with games). Fez is another example of that 3D to 2D mapping trick that games have used. It’s also a good source for my research because it’s fully developed, and you can see how the essential logical/illogical aspect of the game has been tied into the aesthetics and the story.

More on surrealism

Because I’m struggling with how overtly surreal I want the transitions in my game to be, and how much I should address the impossibility of what is being portrayed, I’ve been noticing These ideas have been appearing in my research:

  • Diegetic vs non-diegetic sound
  • Breaking the 4th wall
  • Meta-textual references

Emma Westecott, who is acting as a thesis instructor this semester, noted that text is a sort of a layer between the player and the game world. How many non-diegetic or surreal things do I want in my game?

Wes Anderson’s scenes are often theatre-like and break the forth wall. Similarly, the  prolog of Dogville (2003) is highly stylized, but the scenes still feel rich and interesting. Part way through the introductory part of the game, Fez re-shows the Polytron start-up logo sequence, as if the main character’s discovery of a new dimension is so universe shifting that it affects the running of the actual game. These things bring attention to the medium and to the un-realness of what is being portrayed. Surprisingly, they don’t necessarily keep the viewer/player from feeling like they’re exploring a rich, lived-in world.


Dogville. Dir. Lars Von Trier. Lions Gate Entertainment, 2003. Film Clip. Dogville Prolog. Youtube, 11 Jan. 2011. Web. 22 Nov. 2016.

Fez. [Phil Fish, Renaurd Bédard]. Polytron Corporation. April 13, 2012. Video Game.

Website

Week 10: Prototype

For week 10, I prototyped different perspective shifting techniques and tested a few small technical and aesthetic things.

Screen Shot 2016-11-14 at 6.18.05 PM.png

I played with a new type of transition that captures and records the positional relationship between the player’s camera and the object at which they are looking. It then cuts to a different scene, positions the camera in a different direction (this is the perspective shift) and positions the object using that same relationship. What happens is, the player has the same view the whole time, from the first scene to the next. The object can then be locked to the environment, and the camera control can be returned to the player, so that they realize they are not looking in one direction anymore, but another. The technical problem with this is gaining enough control over the camera with a pre-made Unity fps controller. I have achieved this effect only for the second before something in the pre-written code automatically snaps the camera somewhere else.

This effect can be used for any perspective shift based on direction, but is best for up to down or down to up shifts because of its reliance on views that don’t have anything in them (eg.: of a flat sky) or of two views that look the same (eg.: a reflection of the sky and the sky)

Screen Shot 2016-11-14 at 6.18.32 PM.png

Another thing I prototyped was a space that grew as you turned and walked around inside it. I thought it would be as simple as making a lot of location and view triggers, but I didn’t realize that these triggers would intersect with each other and need to be very carefully positioned. It does give a cool effect of be in a confined space and then suddenly be in an open space. If I were to make this prototype again, I would use triggers not to see where you were looking but where you weren’t looking.

Screen Shot 2016-11-14 at 6.18.45 PM.png

I also tried moving the player through code toward a building to achieve a “far to close” perspective shift, but the effect wasn’t that interesting, and the sudden lack of control over movement would probably be annoying to the player.

Screen Shot 2016-11-14 at 6.17.39 PM.png

I added on to my last prototype (the one I made with the classroom and hallway). Now, as the player enters their room, they will have the opportunity to pour some soup from a flask and make some tea. The food will calm them enough to be distracted by their imagination and their book again.

Screen Shot 2016-11-14 at 6.16.08 PM.png

If the player puts their curser over the kettle, its material switches to a shader with an outline and some text will pop-up, asking the player to engage with it. If they press a key, the text will change. I put a view trigger over the soup and a location trigger in front of it. If both of these triggers are triggered, the player camera’s depth of field will decrease until they have a zoomed-in view of the bowl. The scene will then change to one where the player is now very small, looking out a blimp floating above a round lake with a raft in it. They can walk around the blimp and look out the windows to see the rest of the lake. They can also jump out and see a house take form on the raft as it is viewed from the side rather than the top (where it blends into the raft, looking like a cracker).

This sequence is a perspective shift that goes from close to far and from being big to small.

Screen Shot 2016-11-14 at 6.16.33 PM.png

Finally, I played with limited colour palettes and gradients to try to make beautiful, visually interesting environments. For this purple mountain environment, I added the gradient with texture mapping in Blender. It fades to dark purple that matches the purple of the ground plane that the mountain plane grows out of. That dark purple is also the sky colour I set in Unity, so the environment seems to move out infinitely. I wanted to create the effect of feeling isolated. The main effect I got out of it was motion sickness, because of how slowly it felt to climb the mountains (because their texture changes so gradually, it doesn’t feel like you’re going anywhere).

The mountains look better from far away. I may address this motion sickness issue by making the mountains a non-explorable aspect of the environment (e.g. a view from a tower).

 

Thesis: Week 10 Additional Reflection

I thought more about the feedback from week 10.

I think it’s a really interesting idea to shift between free-form exploration and narration. I originally wanted to merge those elements together- you reveal story through exploration and interactions with things you find, and the way that scenes flow from one to another. I hadn’t thought about having specific moments meant just for looking around, and then other moments where you’re being told things. 

I asked whether scale shifting should be considered perspective shifting, and the unanimous response was yes. I think it is too. Another perspective shift that I forgot to mention was right side up to up side down, which brings up an important issue that I’ve been thinking about: these transitions have the potential to be very surreal looking. For example, if I wanted, I could pretty easily animate a chair morphing in a building to create a smooth transition. But how overtly surreal do I want this game to be? (by surreal I mean the colloquial sense of something seeming half-real, not Surreal) Originally, I wanted the player to be able to print screen any part of the game and for it to show something that was physically possible. No half-chairs, half-buildings. Nothing floating in mid-air, nothing upside down. In a dream, crazy things happen but you don’t question them. It doesn’t feel surreal until after you think about it. It feels like it makes sense. Adding surreal elements would probably really change the tone of the game, which I’m not sure if I want.

This question may be tricky to answer, as is the question of whether or not people will gain meaning from certain transitions. The solution to both of these things will likely be play testing different versions of the same sequences and changing variables such as the initial perspective of a scene, the transition type and the method of communicating the story (eg.: voice-over narration vs. text).

Finally, shading style was mentioned as a little too flat. I’d like to populate my environments with more detail, and switch from flat shading to two-tone shading. I’ve been researching shaders to get more control over how Unity presents models. However, I still think minimalism can be powerful, if used with very thoughtful, strong composition. I don’t think a game has to have a consistent amount of detail from scene to scene- the detail can probably vary somewhat for effect.