DREAM

(2023.1)
Interactive Experience, 5-10min

Installation view at “FARAWAY” solo exhibition
Dowel, Tokyo, January 14–15, 2023



How to sneak into someone else’s dream?

It is an irrational world where unfulfilled wishes and the reflections of reality meet and interweave.
The layers of sound and space build an experience that will bring you into the dreamer's mind.
What they feel will become what you feel.

In this small black screen, whose dream there it rests?

DREAM, promotional image




DREAM, promotional image

Installation view at 3rd Grade Advanced Production Exhibition
Musashino Art University, Tokyo, October 20–22, 2022



This work uses Godot Game Engine to build the game software on the mobile device, which transfers interaction data via UDP protocol. A computer running TouchDesigner receives the data and responds by changing the output video on a projector mapped to the room. The ambient sounds are in stereo and in sync with the scenes on the mobile device, creating the whole visual and audio space.


DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, photo of audience interaction
On the device, the audience can see a 3-dimensional, VR-like space. They can rotate the device to reveal the whole virtual space and tap to interact with the scenes. As the audiences move through the scenes, they will discover that the light and sound inside the whole room are linked with the device. When they create a starry sky in the virtual space, they will be surrounded by a starry sky in the room; when they wander in a rainy forest on the device, lightning and thunder will also start to strike around them.

The interactive environment boosts the audience’s empathy with the story’s character, and they almost feel like themselves being the ones in the story. The music is also composed for the story, intensifying the audience's emotions.


DREAM, photo of audience playing
DREAM, photo of audience playing



Instead of the VR headset which separates the audience entirely from physical reality, this work uses iPad and projection to bring images and interactions into the world. This allows a wider audience to access and experience the work, and reveals the different possibilities of digital interactions.




I am always fascinated by the ancient Chinese story of Zhuangzi and His Butterfly Dream (庄周梦蝶), in which the author questioned how can we distinguish dream and reality. This old philosophy questioned our perception, consciousness, and our true self. What if the self I am experiencing now is a dream, and I will be waking up someday as a patient trapped in a hospital?



Screen recording of The Dream
Interactive Media, Fall 2021, Pratt Institute



open in browser




In September 2021, I decided to write this story and develop a web experience using p5js. The goal of the course project was to explore storytelling with an interactive interface, and I wanted to tell a story with minimum words. I chose two scenes that popped into my mind, running and getting lost in a forest, and floating in space with millions of stars lighting up.


DREAM, sketch



Given the capacity of this technological framework, I used 2D animations to create interactions with the sense of space. When the cursor is moving across the screen, trees in the forest will move in layers to simulate depth with the parallax effect. I also designed the starry sky to loop across the screen, with sections of the canvas controlled by JavaScript (shown in the sketch above), symbolizing the endless space around.


DREAM, sketch



I use short sentences in a minimum background to create a voiceover before looping the scenes, deliberately breaking away from the immersion, as a continuous dream interrupted by daylight.

This moment of break emphasizes the ancient philosophical question, reminding the audience to think from an outside perspective: is it possible that those dreams are true? How can we make sure that we are experiencing the one true reality?

DREAM, sketch


As I was writing the story, a melody suddenly appeared in my mind, so I arranged it into a musical composition. My emotions are in embedded in the notes, and I wish the music could bring my feelings directly to the audience’s mind. I continued to complete the narrative with sound.

Natural sounds were added to the forest scenes, and the use of different samples rendered contrasting atmospheres in different parts of the story. The identifiable sounds from the medical devices can be recognized without any visual hints, so I decided to only use sound, drawing the ending in a dramatic way.


DREAM, process of building the scene
DREAM, process of building the scene
DREAM, process of building the scene
DREAM, process of building the scene

DREAM, flowchart

Flowchart of the mobile game app of DREAM


In October 2022, I recreated this project in Godot, an open-source game engine. The features of mobile development allow me to create 3D environments while utilizing gyro sensors to navigate through them. I also designed an interaction tutorial and an auto-restart system to fit in with the exhibition setting.

In January 2023, I had the opportunity to show this work in a self-curated solo exhibition. The white cube space inspired me to introduce projection into this work, extending the work from the screen into the physical space.




FEATURED IN


3rd Grade Advanced Production Exhibition

3rd Grade Advanced Production Exhibition

Musashino Art University, Tokyo
October 20–22, 2022

“FARAWAY” solo exhibition

“FARAWAY” solo exhibition

Dowel, Tokyo
January 14–15, 2023


DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, photo of audience interaction
DREAM, screenshot of the first scene
DREAM, screenshot of the second scene
DREAM, screenshot of the fifth scene
DREAM, screenshot of the last scene