Tell us about your research.
Are you familiar with Twitch? [Twitch is a live streaming video platform primarily focused on video game and eSports competition live streaming]. The display is generally the game, and then there's the webcam view of the streamer playing, and then there are lots of other different things, maybe statistics, sometimes they have chat data, music, and lots of different pieces on their overlay. My research is in adding biometric data to the streamers' displays on Twitch in order for the spectators to be more engaged, and maybe open a different channel of communication between the streamers and the spectators about emotion.
Often, on Twitch, you'll see the streamer’s heart rate added when they’re playing horror games so you can watch the spike at the peak moments. Inspired by that, I made this customized tool that monitors additional biometric data with an off-the-shelf wearable plus facial recognition.
Has it been used in Twitch yet?
Yes, we've done a few studies. We've been integrating it into streamers' setups to see how well it works. It's been going well. I recently went to Pennsylvania to start two collaborations with Carnegie Mellon University.
They are working on games that focus on spectators—getting the spectators more involved in the experience because it's a whole new opportunity for game design. Players connect through some games, talking to each other, but now you're adding this other layer by having spectators. Thinking about designing to incorporate those spectators into the game experience is what’s next.
There's this one horror game that they made. The player is in a house and there's a ghost chasing them, but they don't have access to the map and they can't see anything. The spectators have a screen that shows them a map of the house and the location of the ghost. They can talk to the streamer and tell the streamer how to get places so it's encouraging them to participate. I added my biometrics to the overlay so you're able to see not only all of that other data but also how the player is feeling. I'm going to start running studies soon about that.
Why emotion specifically? What do you think spectators find interesting about knowing that?
There are certain non-verbal cues that you can't get from a remote experience so I think what Twitch is doing is bringing the sociability back. Seeing somebody's emotions gives you a different outlet. Maybe it's not the same as all the cues that we have in real life, but it's getting to that point of giving people other markers for communication about emotions.
I think humans are social creatures. Twitch is basically a performance space online. It's people performing and more people watching them perform. A really big part of performance is the sense of liveness. There are a lot of different factors to how “live” and how immersed in an experience you can feel. Factors like proximity to the stage enhance your feeling of liveness. Features like adding IM chat are ways Twitch is increasing the liveness, making spectators feel more like they're participating. Making Twitch spectators feel a part of the performance is an important part of the entire experience. I think my tool is another way to do that.
How is this different from other entertainment online?
Twitch formed a collaboration with the professional gaming industry and they became like ESPN for eSports. That's what Twitch started out as, but now it's really cool that people are starting to use it for everything, like posting or streaming programming tutorials, or Twitch Creative is another sector of Twitch where you can watch streamers paint and be a part of that community.
Really what Twitch is now, I think, is a community or communities of people that may not know each other but enjoy hanging out and talking in this remote environment. It differs from other parts of the internet, like Facebook, because it’s a shared kind of space where you're all watching a performance together but there's just that one stimulus that you're watching.
How did you first get started in this area?
Two years ago I came into the program as a Computer Engineering student but I decided that was not my passion. I saw one of the talks in a CE-200 class that Katherine Isbister gave. She was talking about games and wearables and it sounded really playful and interesting to me so I switched into her lab. I had a meeting with her and she gave me some sensors, like a headband which tracks your EEG waves and a wristband which tracks GSR [galvanic skin response, which is your sweat] plus it tracks your heart rate and a bunch of other data. It's like a Fitbit. She told me to see what's interesting, see how I can apply this to something games-related. I’m interested in Twitch and I thought it would be really interesting to see the players' data.
Are there any privacy concerns with displaying data like this, or any challenges to gathering it?
Twitch is opt-in. The players are also actors. They don't have to wear it if they don't want to. But if they're interested, they can. It's also not intrusive. That was a big goal of the tool when we were designing it; most of the biometric sensors that have been used are super intrusive. Usually to gather heart rate you wear a strap around your chest and to measure galvanic skin response you have to put clips on your fingers. We wanted it to be as non-intrusive as possible because we wanted people to be able to use it while they’re playing.
We're using just a wristband and an affect recognition software. We've gotten some questions about whether or not exposing your emotions is something that is harmful in any way to the streamers but again, it's their choice whether or not they want to do it and we haven't had any people say they don't want to. If it's interesting to the streamers and it's interesting to their community, many of them want to do this kind of stuff for their community. I went to TwitchCon this year and there was this one streamer who mentioned that she would get really sweaty and nervous while streaming her programming tutorials (I kind of want to collaborate with her so I can track her sweat!), so she put the humidity level in her room as a piece of information on her stream so the viewers could see it increase as she got nervous. It's interesting to let people draw connections between the environment and the streamer.
Besides increasing the liveness of an event, why might we want to understand the relationship between Twitch's streamers and its spectators?
With a lot of sports, people might meet up with their friends and watch a sports game on tv, and that's the social experience we have with those broadcasts. And in gaming, you get on Twitch and view the game there. But when you're at the bar watching the game, you might not get an explicit sense of the overall community and what they're feeling. The cool thing about Twitch is that even if you're at an eSports event in real life, you're also looking at your phone to see what the chat participants think about it. You're a part of this whole big community also watching this event, so Twitch connects you to the whole of everybody, the thousands and thousands of people also watching the stream. It brings up interesting conversations about how much we should design for that space, too.
How do you think your schooling here at the Baskin School of Engineering has helped you in your research work?
This program has been great because it's allowing me to do research while also applying that to my classes, and my classes support my research. For example, last quarter I took a class with Noah Wardrip-Fruin and we had to do a project for the class on playable media. I thought "how can I apply my research to this?" and everybody is supportive of that. I collaborated with one of the people in his lab who was doing some studies on a game called The Wolf Among Us. I decided to do a comparative analysis study of different methods we can use to evaluate players' emotions when playing a game. I had them use my tool, biometrics, to see how they were feeling during the game versus this toolset that Katherine Isbister made called the Sensual Evaluation Instruments, which is a set of eight 3-D printed objects that range from spiky to smooth. During an intense moment, for example, people will pick up the spiky one—it's like a nonverbal communication method instead of post-game questionnaires or retrospective interviews.
I used a number of other methods and did a comparative analysis of the strengths and weaknesses of them in the context of this narrative game genre, so that was just an example of the way my classes are a really supportive environment of me and my research. I'm also organizing a panel at a Human Computer Interaction conference next year so as I'm writing the first draft of that paper I can get more professors’ feedback on it in class.
What do you like to do when you're not working on school?
Roller derby, mostly. Roller derby has been taking up a lot of time and it's very fun.
Do you play video games?
Yeah I do. It's harder to game when I'm doing a lot of research work in that field, so not as much anymore.