Build and Deploy Episode 1: Weird and Wonderful Bots with Nicole He
Time to read: 22 minutes
October 13, 2020
Written by
Reviewed by
Build and Deploy with Liz Moy is a curiosity-driven podcast that explores the lives of people who create things with code. Some might describe themselves as creatives, programmers, musicians, artists, or even fashion designers. But the truth is that no matter what their title, they’re all developers.
Our first episode is with Nicole He, a creative technologist and game developer who made sms-bot.info, a resource that empowers anyone to build a SMS bot using Twilio and Google Sheets. We talked about the first game she ever built, the motivation behind building the tutorial, and possibilities of new features, like extending it for WhatsApp.
Subscribe and Listen on: APPLE PODCASTS | RSS | OVERCAST | STITCHER | SPOTIFY | POCKET CASTS | GOOGLE
All of the music you hear in the podcast is made with code. We asked livecoder Dan Gorelick to make some tunes using TidalCycles. You can check out the music on Github.
- Why voice-controlled games and experiences have a long way to go [04:25]
- The first game Nicole ever built [05:47]
- Empowering mutual aid groups to automate processes [09:36]
- The importance of offering multiple options for problem-solving [15:01]
- How Nicole built the SMS bot code with Google Apps Script and Programmable SMS [18:35]
- Adding WhatsApp functionality [25:10]
- Games, podcasts, and organizations Nicole is paying attention to right now [30:05]
[00:00] Liz: If you go to your favorite search engine right now and type in “yelling at computers” you’ll likely get a result that’s from game developer and creative technologist Nicole He. I think this is really funny because Nicole is an expert in the voice technology field, but all of the work that she makes uses tech to make really funny and interesting art.
I saw Nicole demo a sci-fi voice controlled game that she’s working on at XOXO in Portland last year, but I’d definitely been following the work she’s created before that, like a Tinder-swiping robot and a program that generates a prompt of “the best art to make in the moment,” which then requires a human to create.
The latter project is one Nicole created while getting her master’s degree at NYU’s Interactive Telecommunications Program (ITP) at Tisch School of the Arts, which she described to Forbes as “Hogwarts with Computers.” Recently Vogue tapped Nicole to fine tune OpenAI’s GPT-2 language model to generate interview questions for pop star Billie Eilish.
The interview was weird and amazing.
So basically I wanted to trap Nicole on video chat and experience a small dose of her creative genius. And I found a way to make it happen, because a few months ago she made sms-bot.info, which is a guide to making an SMS bot in Google Sheets using a Twilio phone number that takes about 30 minutes and doesn’t require any coding experience.
I loved the accessibility and flexibility of this project and I couldn’t wait to talk to Nicole about how she built it and the uses that she envisioned may come out of people building and extending the tool.
Nicole, thanks so much for chatting with me today. Would you mind introducing yourself? I'm just saying a little bit about who you are.
[02:03] Nicole: I'm a game developer and a creative technologist based in Brooklyn. I do a lot of things with voice technology in general. I'm teaching a course at NYU ITP in the fall about voice technology, how to use it for making weird stuff specifically. A lot of my work is sort of focused on making things that are playful, and not necessarily useful, which is why it's interesting that today we're going to be talking about a project I made that is like explicitly meant to be useful, which is a little different from my usual work.
[02:38] I graduated from NYU ITP. If you're not familiar with that program, it's a creative tech master's program in New York, where basically I learned how to do everything that I know how to do. After that I worked at Google Creative Lab as a creative technologist for two years and then since about a year ago I started out on my own as an independent game developer and freelance creative technologist. So I'm working on a couple of video games right now, one sort of experimental one and one more commercial one. That's where I usually spend my time but I also do some experiments here and there still.
[03:20] Liz: Awesome. We're going to play this or that. I will read off the two things. You pick one and explain why you chose it.
Software or hardware?
[03:30] Nicole: Software. I have a soft spot for hardware as well, but I think especially these days with software you can reach a lot more people with it than physical objects, unfortunately.
[03:42] Liz: Yeah, it is. I think that hardware's really interesting and I would like to do more with it personally, but even just in terms of within Twilio, trying to get people, the kits there's always that barrier to entry.
Tabs or spaces?
[03:58] Nicole: Tabs. Like why would I type more than I have to?
Early morning or late at night?
[04:05] Nicole: Late at night. I'm not a morning person. I mean, not that I'm useful late at night either, but I'm just more likely to be awake at least
SQL or NoSQL?
[04:15] Nicole: NoSQL. I've never learned SQL. So that's why.
Coffee or tea?
[04:19] Nicole: I'm drinking coffee right now. So I got to say coffee.
Hand-control or voice-control?
[04:25] Nicole: I feel like I'm one of like the three people on earth who has to say voice-control, but actually hand-control. For creative purposes, voice, because it’s broken, which makes it more interesting.
[04:39] Liz: In what ways is it broken?
[04:42] Nicole: The technology clearly is not at the level that it needs to be in order to actually be useful in order for people to be able to speak in a normal way. It's bad at understanding accents, it’s bad at understanding unusual syntax, things like that. It works differently from how you actually want to be able to speak to another person, for example. But I think that's sort of where a lot of creative potential lies.
Like one thing that I like to talk about a lot is sort of the difference between. Our expectation of technology is informed by science fiction and how the technology works. I think voice technology is like a really good example of that.
[05:24] Liz: Yeah. I definitely agree with that. Like you were saying, there's just so many limitations and having to use it in a very specific way. When you said that it also kind of reminded me of VR because I think that VR seems really cool and like our idea of what it will be is very informed by science fiction, but it's definitely not quite there yet.
Nicole: Yep. Yep.
[05:47] Liz: I like to start the interview by asking what is the first thing that you ever built?
[05:56] Nicole: In middle school I built my first website. This isn't cool anymore, but it was a Harry Potter fan site. We're not allowed to build those anymore unfortunately.
Yeah, it's funny. I actually never thought about it as a video game until this very second, but I guess that was my first video game. I made a website for Quidditch where the idea was that the snitch was hidden on the page and you have to click around to find it. I think I built that by making the webpage black and the text black and then you had to click around.
But you could just highlight the page and find it immediately. So I never solved that problem, but that was the first thing I built.
[06:38] Liz: That's the first time that you realized that there are very simple ways to solve a problem and that as long as nobody else can figure out how you've done it, it's fine.
[06:50] Nicole: No one ever played it obviously but I made it anyway.
[06:53] Liz: That's so funny. It is really weird — not to get off on a tangent — but Harry Potter was my favorite series growing up. And it's one of, I guess, it's like, you never want to meet your heroes type situation. Just please stop talking J.K. Rowling. Let me have my thing.
[07:12] Nicole: You’ve ruined all our childhoods. Thanks.
[07:14] Liz: So you made this Harry Potter fan site. Is it still up somewhere? You still have the files?
[07:20] Nicole: No, I wish. I have no idea. It might've been a geocities? I'm not sure. Yeah, I don't know where it is anymore. It would be incredible if I could find it, but I have literally no idea how I would do that.
[07:14] Liz: For a lot of people my age, like when I was learning to code everybody's references were Xanga or MySpace and making custom layouts. I'm super fascinated by the fact that you learned coding skills in a creative context, because I think right now we're seeing such an influx of people turning to programming for very practical reasons, which certainly isn’t a bad thing, but I think it's really cool that there are so many different paths to it. There are so many different ways that you can find it and figure out what it is that you really want to make with the tools that exist.
[08:19] Nicole: I mean, there are lots of downsides to learning to code in art school. But there are a lot of good things about it too. For me, at least I've found that the best way that I learned is project based. Making something that I wanted to make, I learned so much better that way than just following a tutorial to build a database for no reason or something like that.
It's sort of like, once you have the motivation, it makes it much easier to learn. So I'm really grateful to have picked up coding in the art school context, even though again, there are downsides to that as well, but for me that worked out really well. And I feel lucky because I learned to code in grad school and one of the most valuable things about that was having the time and the space for two years to make the things that I wanted to make. Just because I wanted to, and not because of any sort of like company or product pressure or financial pressure to make something that could make money. That was really useful for me to explore creatively and on a technical level as well.
[09:36] Liz: I want to talk about your tutorial that you made. So you made a tutorial that sets up an SMS bot in Google Sheets using a Twilio phone number. Can you talk a little bit about how you came up with the idea for this or why you decided that you wanted to make this?
[09:52] Nicole: Living through the pandemic, one of the positive and inspiring things — even though ideally we wouldn't have a need for this — is seeing the mutual aid organizations pop-up everywhere.
In the community where I live, but also all around New York City, and all around the country, a lot of people are self-organizing to help their neighbors out, with groceries, helping them get their medications, in a time where people couldn't really go outside.
I live in Brooklyn and the Brooklyn-based organizations seem to actually have a lot of technical resources in terms of developers. I guess like all the developers in New York City live in Brooklyn — but not every group has that.
There's a lot of work being done collecting information from their neighbors about what they need, and figuring out how to get them what they need. I was thinking about how a lot of the friction came with intake volume. The first thing I noticed is a lot of them have phone numbers, or they have a neighbor call to get connected to a volunteer.
They have things like forms online they can fill out, but I think the thing is, not everybody can or wants to fill out an online form in order to get connected to volunteer. For a lot of people, calling and texting is the most accessible way for them to be able to communicate with an organization to get what they need. So it actually did seem quite valuable to have something that would help people do that.
I was hearing that the way a lot of these organizations worked is that they would have a phone number, probably Twilio phone number, and neighbors would call and then they would leave a message and then volunteers would call them back. Or they would text in, and then would have to like text them back, sort of like back and forth to get a lot of the information. Like, “Who are you?” “Where do you live?” “What do you need?” That kind of thing.
So I was thinking, a bot that collects the information would be really useful because they’re asking the same types of questions over and over, right? So this is a situation where it makes sense to sort of do some automation and people can build that.
There are lots of tutorials for coders to build bots. That's fine, but not everyone has a coder who can build a bot or has the time to build a bot. I figured, why not try to make something that would make it as easy as possible for people to set up a simple system? Basically that lets them design a bot in a Google spreadsheet. Then you just save the responses in the spreadsheet too. It just feels natural.
I built a tutorial to show people how to set it up without coding. You do have to copy and paste code, which is not ideal. Ideally it would be something so easy that it's set up for you. Through a lot of experimentation I realized this way is probably the easiest way to do it while respecting the privacy of people. Only those that have access to the spreadsheet or your Twilio account can see this information. Otherwise, you'd have to build some kind of intermediary service, which is both too complicated and privacy is still an issue.
I think we've all been thinking about this; what can we do to help our own communities at this time, with our own skills? Helping people automate their own processes could be something that's useful for them to help their neighbors.
[13:45] Liz: It's funny your point about how different neighborhoods within Brooklyn tend to have a lot of development resources since a lot of folks end up here. I live in South Slope, so my group is the Kensington-Windsor Terrace group. It’s been so cool and eye opening to be a part of, because it is extremely diverse. There are quite a few families who have been here just for generations. I would love to get your insight into this because one of the things that I have been trying to be really aware of is, we have a group that specifically focuses on intake and I've been so excited to be like “I can help, I can get us resources. I can set this up if you all want.”
[15:01] And there's this piece of educating everyone — and maybe that sounds a bit gateholder-like — and letting everybody know what this is and why it could be helpful to them. But in a way that’s like “I'm not trying to tell you that this is the right way and the way you're doing it is the wrong way.” I wondered if you had any experience with that or had heard any stories regarding that.
[15:12] Nicole: Yeah, it’s interesting, especially when you have a lot of people that come from a lot of different backgrounds and have a lot of different skills, organizing together to try to solve a problem that can be solved a lot of different ways. It’s one of those things where I don't think there's a right way or a wrong way to approach it. I think it's just hopefully allowing people to see some of the options available. Maybe some things that they had never considered before, or realized were possible.
I think that's probably one of the biggest challenges as far as I understand with organizing, right? It's trying to collectively solve problems with a lot of people together, making decisions together. For me, making this tool, I didn't know how many people would use it, I just wanted to put it out there as an option in case it would be useful for people.
Knowing, for example, that a lot of my local mutual aid groups use Airtable, this example doesn’t use Airtable but it could be extended to that. Thinking about that, maybe this tool wouldn’t be useful for them because they use this other system, but maybe it would be useful for another group like that who are starting from scratch. It’s just putting one thing out there. Not saying it's the best way to do it, or the only way to do it, but just as one way to do it. If it's helpful, then we can try it.
[16:44] Liz: Yeah, absolutely. Just kind of encouraging people to experiment with things and see which one will be the best fit for the problem. You brought up Airtable and it’s interesting because I recently wrote a blog post on creating a survey with Airtable, which was an idea I totally stole from another person that I work with. They did it in JavaScript and I was like, “Oooh, I'll just take this and make it for Python.” I think Airtable is really cool.
One of the issues that I ran into though, is rate-limiting issues. Then the other thing that I've noticed is that sometimes when people are familiar with something like the Google suite of products, um, it seems better to stick to what they’re more familiar with instead of saying “Hey please go learn this other thing you've never heard of before and have only worked with in a very minimal way.” What played into your factor in choosing Google Sheets?
[17:51] Nicole: I've actually never used Airtable. I see that people use it a lot. It seems like a very cool product, but yeah, it seems like there's a lot more familiarity with Google Sheets. I feel like I see that as what people use whenever they have some sort of collaborative or public spreadsheet across all types of organizations. So it seemed familiar to people and it's also familiar to me because I worked at Google. So that's part of it too: I was familiar with some of the tools that let you build these sort of things on the backend, like script stuff, that not a lot of people — even developers — know about, like the way that you can extend and make add-ons for spreadsheets or for docs.
I was familiar from my past experience and so that's why that's part of why I landed there, because it seemed like a lot of people are familiar with it and how it works and I was familiar with it on a technical level.
[18:35] Liz: I did not know about Google Scripts. So you have some JavaScript code that interfaces with Twilio and Google Sheets. Can you talk a little bit about writing that code and also about Google Scripts?
[18:55] Nicole: Sure. So the way that it works, there's actually two components to it, both built using Apps Script.
On the Twilio end we’re using the SMS messaging webhook where when your number receives a message, I hit a webhook and can do whatever I want from there. With the webhook, this is the thing that's a little bit wacky about the code here.
The SMS bot code does two different things: One is that it creates a webhook that reads a spreadsheet and then it does stuff to the spreadsheet. It finds the next message to send, and then returns it to Twilio. It also saves the message that comes in and then puts it into the spreadsheet.
The other part of the code is that it’s making a sidebar for add-ons for the spreadsheet itself. The project includes the template spreadsheet, which you have to copy. And then you basically create this add-on that's like a sidebar, sort of an `<iframe>` thing that pops up in the spreadsheet. And that allows you to not touch the sheet itself. You can edit the questions that your bot sends in the sidebar. It's designed in a way that it’s reading very specific cells in the sheets. So if we change the structure, it will mess up the code. That's why I tried to make it so people can try not to edit the spreadsheet, just edit the sidebar.
The sidebar provides an interface for you to do the things you need to do to get the bot working. Basically Google Apps Script is JavaScript but, as these things usually are, it's like a weird version of it.
[21:05] Liz: Like their flavor of it.
[21:07] Nicole: Yeah, and actually one reason why I ended up using Google Apps Script to create a webhook is because one of the biggest challenges for this is authentication issues since Twilio has to be able to read your spreadsheet.
That becomes an issue because the spreadsheet needs to have quite restrictive permissions since you're collecting personal data. The only way that I could figure out how to get the webhook to read your spreadsheet is by also making it under the same Google account, which is why like the code has access to the spreadsheet and creates the webhook, therefore has permission.
This is also why people have to build their own accounts so that essentially their Gmail account has access to both, to everything, and then they can read the spreadsheet.
It's relatively simple. I mean, it is simple in concept. It just reads your spreadsheet. Somebody texts the bot, the bot then looks for the first question to send them from that. And actually before it does that, it saves some information.
Their phone number, the timestamp, the message ID on the Twilio side, we get all this information from Twilio. And then we do some little things, like it will look in the Sheet to see if this person has texted us before.
It looks to see if they have completed the flow of the bot or not. So then when they, so it saves that and then they win and then it sends them the first message. And then when they reply, it looks in the sheet to see where they are in the flow. It saves their next message in the right spot and then it sends them the corresponding next message.
That's basically what it does. It's pretty simple, but it gets around a lot of the trickiest authentication issues when it comes to being able to access the spreadsheet.
[23:27] Liz: Very cool. I think from building some of these like SMS bots, it's really cool because, once you do it, you realize how simple it is, but when you're actually in the thick of building it, it can be very confusing.
One of the things that our PR team actually talks about a lot in terms of SMS — I never really considered it before they brought it up — is that there are a lot of people that don't have a reliable internet connection. So SMS or calling is really their main form of communication.
The other thing that I find really cool about this tool is that it hits on the whole ethos of mutual aid in that you're empowering other people to be able to do this themselves and use it, and then you can make all the tech people happy because, security.
[24:16] Nicole: Right, right,
[24:18] Liz: So have you gotten feedback from folks about how they've been using it?
[24:23] Nicole: Yeah, I've heard from a bunch of people, different people, which has been really awesome. Some mutual aid groups are using it how I imagined it, but also people using it for all sorts of other things. Like for personal projects, things with their friends, weird art projects. Then there are other things, like people trying to distribute masks to their community. It's been a range, which has been super awesome. I’m hoping whoever finds it useful for any reason will try it. It doesn’t have to be mutual aid, it can be weird stuff too — but I do hope more people use it.
[25:10] Liz: Is there anything else that you would want to add to the tool?
[25:18] Nicole: Yeah, a lot of people have asked me how to put logic into it. That feels like the natural next step. I think it's more of a design challenge because you can do this in code, right; obviously you could just put whatever logic you want in the code. But then the main challenge is, how would we be able to keep it in the spreadsheets and let you design the logic of our conversation? That would be quite interesting and I think quite complicated.
That's something to think about. Another thing that would be great if I have time or if someone else wants to do it is to build a WhatsApp version of this instead of SMS, because I saw that Twilio also has WhatsApp functionality. It seems like it would be pretty simple to extend it in a way where it does the same thing, except it uses WhatsApp instead of SMS.
I think that would be really cool. It seems like in a lot of places they use WhatsApp instead of SMS, so why not do it? If someone else wanted to do it, it’s on GitHub — It’s open source.
[26:41] Liz: To me, the most challenging thing about WhatsApp with Twilio is just getting the number. In my experience, the code you write for SMS is pretty much the same for WhatsApp and the sandbox is easy to get set up, but you have to have a business Facebook page. I just deleted my Facebook, like permanently a few months ago. So I was going to try and get a WhatsApp number, and I don’t have any friends or anything on my fake Facebook, so it thinks I’m not real.
[27:21] Nicole: That stuff is so annoying. When I was making the Tinder robot — I think Tinder has changed the rules since then — but back then you had to have a Facebook account. In order to use Tinder for my bot I had to make a fake Tinder profile with a fake Facebook and it was always getting shut down because it was like “you’re catfishing with your fake account.”
[27:48] Liz: That's really funny. Yeah. Oh yeah. Dating apps. That's another conversation. So have you, have you used this tool for anything?
[27:57] Nicole: I haven't personally used it for anything. If I used it, it would probably be for some weird thing that I haven't thought of yet. When I was setting it up I needed to test to see what it would actually be like to have a lot of people text it. So I tweeted out “Can you text this number?” Like 150 people texted, which was shocking to me. I just had some placeholder questions about sandwiches, like “What’s your favorite sandwich? What’s the worst sandwich?” It was actually super interesting to read hundreds of peoples’ answers to those questions. It makes me curious about what other kinds of things we can do with that. But now I have all this information about people's favorite sandwiches.
[28:50] Liz: The first thing I immediately thought is that I hope nobody wrote hot dog. One thing I do appreciate about building apps with Twilio is that people ask “How can I try it?” And you just give them a phone number.
[29:12] Nicole: The tool itself you can try at sms-bot.info. It brings you to a Google doc, which is the tutorial. It's also on GitHub, which is linked to there. If anyone uses it, please let me know what you're using it for and if anyone wants to extend it for WhatsApp so that I don't have to, please do that. It’s all on GitHub.
[29:37] Liz: If people want to find out more about you on the web … on the web. Is that a thing? On the web? Where can they find you online?
[29:12] Nicole: My website is nicole.pizza and I'm on Twitter @NicoleHe too much and Instagram and Facebook.
[30:05] Liz: Are there any games that you're playing right now?
[30:11] Nicole: Yeah, I just finished playing The Last of Us 2, which if you’re in the world of games, there's been a lot of capital-D-discourse about it. A lot of debates and controversy about this game, which I think is really interesting that there's all this discussion about this game at all. It feels like it’s supposed to do everything and be perfect for everybody. I guess, for me as a game developer now thinking about the conversations around games as an art form and how that's different from the way that we talk about movies and books and things like that.
I feel like The Last of Us 2, had these like high stakes, where it feels like it had to do everything perfectly in a political sense. It’s an expectation that I don’t think we have from movies. Like when we talk about movies, you're allowed to say, “Well, I liked this part, I didn't like how they did that, but in general, I liked it.”
It feels like you're not allowed to have that conversation with video games, so I've been thinking a lot about that. I think for anybody who's been following anything related to games, it’s been a big subject.
And I will say I did like that game. Another game I played recently is a very different game from the ‘90s called Night Trap. It’s unusual compared to modern games because it's a game that has full live-action video in it.
It's about a bunch of teenage girls in a house and there are vampires that walk around and you have to trigger traps that trap the vampires, but it's actual video recordings of actors — very ‘90s. So that was super interesting to play as well.
It's very frustrating, but it's on the Switch. They have a 25th anniversary edition. Those are some of the games I've been playing. Honestly, I've been spending most of my free time playing video games and Animal Crossing, of course.
[32:50] Liz: How did you find that nineties game? Was that something that just kind of came up because they released the anniversary edition of it?
[32:56] Nicole: I think a couple of people told me about it. I decided to play that on a Twitch stream. I thought it would be an interesting one to play on the stream so other people can also watch it, because otherwise you probably wouldn't ever see those games. I'm kind of surprised they released it on the Switch, but it’s, uh, I don't know if good is the word. But it's definitely a very ‘90s game.
[33:25] Liz: What about books or podcasts? Anything that you're listening to or reading?
[33:31] Nicole: I haven’t read a book in quarantine and that's really bad. But one podcast I’ve been enjoying a lot is Fun City. So this is an actual play podcast. So Fun City is a shadow run actual play podcast. So for people who are not familiar with any of this kind of stuff it’s sort of like Dungeons and Dragons. It’s a tabletop role playing game, except shadow run is sort of like a cyberpunk dystopia that has both elves and dwarves and that kind of thing, but also is a ‘90s vision of the future of cyberpunk.
So people have guns and run around the streets doing crime, kind of thing. So Fun City is a really great podcast where people play the characters and you get to follow the story. I think they have something like 20 episodes now. So I would recommend checking that out — It's called Fun City.
[34:41] Liz: I will check that out. That actually sounds like something I would play. My, my qualms with D&D are just that I can't sit still and talk and craft a story with other people for like four hours. The one campaign that I did do. The DM had music and would change his voice, it was amazing. So I deeply appreciate people that are dedicated to the game.
Any organizations that you are currently supporting or would recommend others support?
[35:20] Nicole: One organization that I've been supporting recently is called G.L.I.T.S., which is gays and lesbians living in a transgender society. So basically they have been doing a lot of work supporting transgender sex workers. Obviously we've been dealing with a lot of crisis right now, so it’s good to be helping people with housing, healthcare and things like that. It’s been a great organization to support for helping some of the most vulnerable people going through the crisis right now. You should check them out and support them if you can.
Thanks for listening and subscribing! Want to talk about something you're building? Email me at lmoy@twilio.com.
Related Resources
Twilio Docs
From APIs to SDKs to sample apps
API reference documentation, SDKs, helper libraries, quickstarts, and tutorials for your language and platform.
Resource Center
The latest ebooks, industry reports, and webinars
Learn from customer engagement experts to improve your own communication.
Ahoy
Twilio's developer community hub
Best practices, code samples, and inspiration to build communications and digital engagement experiences.