DREAM - enddle

This work uses Godot Game Engine to build the game software on the mobile device, which transfers interaction data via UDP protocol. A computer running TouchDesigner receives the data and responds by changing the output video on a projector mapped to the room. The ambient sounds are in stereo and in sync with the scenes on the mobile device, creating the whole visual and audio space. この作品では、Godot Game Engine を使用して、モバイルデバイスのゲームソフトを作り、UDP を通してデータを送信する。TouchDesigner を実行しているパソコンがデータを受信し、部屋にマッピングされたプロジェクターの出力ビデオを変更して応答する。環境音は立体的で、ゲームのシーンと同期しており、視覚的および聴覚的空間全体をこの作品で作り出している。

On the device, the audience can see a 3-dimensional, VR-like space. They can rotate the device to reveal the whole virtual space and tap to interact with the scenes. As the audiences move through the scenes, they will discover that the light and sound inside the whole room are linked with the device. When they create a starry sky in the virtual space, they will be surrounded by a starry sky in the room; when they wander in a rainy forest on the device, lightning and thunder will also start to strike around them.

The interactive environment boosts the audience’s empathy with the story’s character, and they almost feel like themselves being the ones in the story. The music is also composed for the story, intensifying the audience's emotions. デバイス上で、視聴者は VR 空間を見ることができる。デバイスを回転させてバーチャル空間全体を見ることができ、またタップしてシーンを操作できる。視聴者がシーンを移動すると、部屋全体の光と音がデバイスと連携していることに気付くだろう。仮想空間に星空を作ると、部屋の中も星空に囲まれるーー雨の森をさまようとき、雷も視聴者の周りを襲い始める。

インタラクティブな環境は、物語の主人公に対する視聴者の共感を高め、まるで自分自身がこの物語の中にいるように感じさせる。音楽も物語のために作曲され、視聴者につよい感情の変化を与える。

Instead of the VR headset which separates the audience entirely from physical reality, this work uses iPad and projection to bring images and interactions into the world. This allows a wider audience to access and experience the work, and reveals the different possibilities of digital interactions. VRヘッドセットの代わりに、この作品はiPadとプロジェクターを使用している。視聴者を現実から完全に切り離すわけではなく、映像とインタラクションが具現化されます。そして、より幅広い視聴者が作品にアクセスして体験できるようになり、デジタルインタラクションのさまざまな可能性が明らかになります。

I am always fascinated by the ancient Chinese story of Zhuangzi and His Butterfly Dream (庄周梦蝶), in which the author questioned how can we distinguish dream and reality. This old philosophy questioned our perception, consciousness, and our true self. What if the self I am experiencing now is a dream, and I will be waking up someday as a patient trapped in a hospital?

In September 2021, I decided to write this story and develop a web experience using p5js. The goal of the course project was to explore storytelling with an interactive interface, and I wanted to tell a story with minimum words. I chose two scenes that popped into my mind, running and getting lost in a forest, and floating in space with millions of stars lighting up.

Given the capacity of this technological framework, I used 2D animations to create interactions with the sense of space. When the cursor is moving across the screen, trees in the forest will move in layers to simulate depth with the parallax effect. I also designed the starry sky to loop across the screen, with sections of the canvas controlled by JavaScript (shown in the sketch above), symbolizing the endless space around.

I use short sentences in a minimum background to create a voiceover before looping the scenes, deliberately breaking away from the immersion, as a continuous dream interrupted by daylight.

This moment of break emphasizes the ancient philosophical question, reminding the audience to think from an outside perspective: is it possible that those dreams are true? How can we make sure that we are experiencing the one true reality?

As I was writing the story, a melody suddenly appeared in my mind, so I arranged it into a musical composition. My emotions are in embedded in the notes, and I wish the music could bring my feelings directly to the audience’s mind. I continued to complete the narrative with sound.

Natural sounds were added to the forest scenes, and the use of different samples rendered contrasting atmospheres in different parts of the story. The identifiable sounds from the medical devices can be recognized without any visual hints, so I decided to only use sound, drawing the ending in a dramatic way.

In October 2022, I recreated this project in Godot, an open-source game engine. The features of mobile development allow me to create 3D environments while utilizing gyro sensors to navigate through them. I also designed an interaction tutorial and an auto-restart system to fit in with the exhibition setting.

In January 2023, I had the opportunity to show this work in a self-curated solo exhibition. The white cube space inspired me to introduce projection into this work, extending the work from the screen into the physical space.

FEATURED IN