Into My Eye

Platform: webRTC, tensorflow.js, node.js

“Into My Eye” is a web-based project I created for the finals of both my Live Web and Machine Learning for the Web classes, which creates a poetic space for us to look into the eyes of “machines” to see the world they see.

The project is realized mainly based on webRTC, tensorflow.js and node.js running on the backend: webRTC and node.js organize and maintain the server for the program, tensorflow.js runs word2Vec, a group of related models that are used to produce word embeddings. This method allows you to perform vector operations on a given set of input vectors.

Multiple users could together visit the webpage, and trigger the program by giving a simple one-word input. When received the one word as input, the program will first compare it with the category list of Google quick-draw database, if matched, it will returns a random drawing of the category; if not matched, it will start calculate the distances between the input and the quick-draw database, and return the one word “closest” to the input based on its knowledge. All the results based on the user’s input would be framed as “To me [input] is [result]”, and results based on other users’ inputs would be framed as “Others think [input] is [result]”. Therefore, the participation of multiple users at the same time could turn the experience into a collective poetry writing experience.

For long, how algorithm processes our input, or more poetically, the way machine thinks what we think, is like a “blackbox” - and this project tries to visualize the invisible “thinking” process of machines. By framing the program results not as results, but some kind of answers to our questions, the project purposely blurs the boundary between human minds and algorithms and forms a poetic conversation between the two as part of the fantasy about cyberpunk virtual world.

Into My Eye.png

Welcome to the Assessment System

"Welcome to the Assessment System 2018, are you qualified to be Level 3 Chinese? "

“Welcome to the Assessment System” is a Twine game that sarcastically expresses my identity exploration as a Chinese in nowadays political environment . It was adapted from the Tracery project “Cautionary Tales for Chinese”, a poetry project I created early this semester.

The Twine game uses the content generated by Tracery from “Cautionary Tales for Chinese” as its base script, and develops the script into an experience of assessing the participant’s qualification to become “Certificate Level-3 Chinese Citizen” by giving the participant three questions based on the script.

Developed from a simple "cautionary tales”, this reading test-format game reinforces the feeling of political ideology propaganda by forcing the participants to pay close attention to, repeatedly read through all the content given and understanding those inner, unspoken logics and declarations.

This text-heavy project has a pretty straightforward question-answer-question-answer structure: The page with the base script is followed by three questions, each has options standing for different scores, which will add on and give you a final score at the end of the game.

The three questions are created following the classic SAT/GRE reading test format. It also gets inspiration from my memory about answering Chinese reading questions back in high school.

And the final score you get at the end of the game assesses how “ideal” you are now as a Chinese citizen, whether you move on to being an advanced “Level 4-pending Chinese”, stay on getting your Level 3 certificate or go back to being a Level 2 pending status Chinese.

The Twine game structure:

Twine-structure-wtas.png

Some screenshots of the game:

Panopticon

A surveilling installation that confronts the audience with the panopticon of digital age

Highlight: Communication Arts 2019, Interactive shortlist

Collaboration - Tong Wu & Barak Chamo

 

In the age of mass surveillance and commoditized information, we are all, willingly or not, constantly watched. Our devices, online presence, transaction history and even physical presence have become an asset for social network, world governments and corporations to own - we live in a digital panopticon devoid of consent and, in many cases, even awareness. As in Bentham’s architecture, the watchman’s all-seeing eye keeps us from misstepping and policing ourselves.

Through Panopticon, we've created an experience of being both the "observee" and the "observer". It is a critique and a wakeup call to the alarming prevalence of mass surveillance all around us, to how tracking and surveillance devices have become largely invisible to us and to how insensitive we have become toward privacy invasion. 

Inspired by the system of control of the same name, we created the project “Panopticon”, an interactive media sculpture that breaks the illusion of privacy and control of our digital identity and physical presence. Multiple camera rigs monitor users throughout the exhibition space, tracking and capturing faces, and collecting the facial data to be projected on a semi-torus sculpture.

 

 

BILL OF MATERIALS

HARDWARE
  • laser-cut modeled foam core

  • Acrylic thread

  • Apple Mac Minis

  • Logitech c920 webcams

  • LG Short-throw projector

  • DMX spotlights

SOFTWARE
  • Vectorworks

  • Max/MSP

  • MadMapper

  • Adobe Premiere CC

  • Amazon AWS Rekognition

  • OpenCV

  • Python 3

Indecision Space - Ask Oracle to help you choose

Introduction

For the first meditation assignment, with Dan Oved as my collaborator,  we created the Indecision Space, a mobile-friendly ritual accessed at  http://indecision.space. This is an “electronic” ritual that visualizes the conflicting thoughts we usually have when making a decision and explores how human's minds are influenced by unrelated implication when making decisions. In the system to perform this ritual, the participant:
  1. Thinks of something he/she feels undecided or worries about
  2. Holds his/her phone and waits patiently for 5 seconds for the results to come out
  3. Gets a sentence of "oracle" that suggests about decision making

Documentation of ITP students performing Indecider

Inspiration

The idea of Indecider came from our discussion about what part of our daily routine is worth transforming into an electronic ritual. We went from ideas that celebrates our "graduation" from a day of ITP, as many of us stay in ITP until pretty late, to posting on Facebook every single time someone solves a bug, then finally to Indecider. We want to write a small program that performs the role of the Oracle of Delphi, who is the High Priestess of the Temple of Apollo at Delphi that served as the oracle, that could let us know the "oracle" from program about our common life struggles.

How it works:

The oracle is actually nothing but a sentence "the answer has to do with something ..." plus an adjective word picked from a database with about 300 common english adjective words.  And to avoid stereotypes about "fortune telling" or "goddess", I sketched two cats, one for the "indecision" and the "undecided" pages  with a question mark and one for the "oracle"page with an exclamation mark.

IMG_5683


We add a three-second waiting time before you can hit the button "Oracle, please help me choose" and  a five-second pseudo-loading time for the oracle to show. Based on our own experience and our observation of others performing the ritual, proper amount of waiting time in a ritual raises people's interest in and attention on the results that's about to come out, and it helps to enhance the sense of the ritual.

Thoughts:

As we experienced as well as observed, most people who performed the indecider ritual consciously try to build a loose connection between the oracle and a choice they have in mind, or try to interpret the oracle and associate it with some practical suggestions. And the interpretation actually to some extend is a process to convince the person who practice the ritual something he/she prefers or has decided.

Screen Shot 2018-02-08 at 1.42.12 AMScreen Shot 2018-02-08 at 1.42.34 AM