Meet Jill Watson, a first-time teaching assistant at Georgia Tech assigned to moderate an online forum for a computer science class. Jill was 1 of 9 TAs assigned to help answer questions about coursework and projects from the 300 students enrolled in the advanced course.
During the first few weeks in January, Jill really struggled. This was Knowledge-Based Artificial Intelligence, after all, a course with the goal to “build AI agents capable of human-level intelligence and gain insights into human cognition.” It was also a requirement for graduate students to earn their master’s degree. It’s no surprise then that she needed some coaching, especially since feedback is so critical to student success.
“The world is full of online classes, and they’re plagued with low retention rates,” said Professor Ashok Goel. “One of the main reasons many students drop out is because they don’t receive enough teaching support.”
Now a typical class posts approximately 10,000 messages in a given semester. Even though many of these questions are routine, the other TAs monitored Jill’s answers until they saw improvement.
“One of the secrets of online classes is that the number of questions increases if you have more students, but the number of different questions doesn’t really go up,” Goel said. “Students tend to ask the same questions over and over again.”
Eventually, Jill got her bearings. By the end of March, she was freely responding to student questions (well, at least, when she was fairly certain of the answer). She was also posting reminders of due dates and providing midweek prompts to encourage discussion.
However, near the end of April, Professor Goel had to come clean to the students: Jill was in fact software powered by IBM Watson.
Here’s the problem: Since the class first launched in the fall of 2014, the online forum had logged 40,000 posts, many of which were straightforward questions, such as inquiring about the location of assignments and readings. But even with a large team of TAs, the number of posts was simply overwhelming. Last year, Goel decided to work with his TAs to leverage this database of questions and code a virtual TA.
They gave Jill a threshold of 97 percent certainty before answering a question. They had her post answers in a private forum first so they could be screened. And of course, everyone had to keep up the ruse that Jill was actually a person.
Because of how well Jill performed her tasks, Goel plans on bringing her back next semester under a different name with the goal of accurately answering 40 percent of all questions by the end of the year. With Watson fielding the questions that have clear answers, the other TAs are freed to tackle the more technical or even philosophical questions that arise in the course.
It’s not every day you find out someone you trusted with your education is a machine. So how did students respond to the news that they had an AI among them?
According to Georgia Tech, the class was surprised though the general response was fairly positive. Student Jennifer Gavin told The Wall Street Journal, “It seemed very much like a normal conversation with a human being.” The ruse, however, wasn’t perfect—a few students suspected early on that Jill might be a computer (although other human TAs also were suspected of the same). Still some of the students were inspired to start an open source project to make their own version of Jill.
Perhaps it isn’t too shocking that the computer science students reacted this way, given that it is a course on building AI. However, this is likely not how the general public would react.
In a recent online survey on artificial intelligence conducted by the British Science Association, nearly half (46 percent) of respondents opposed imparting AI with emotions or personality. Additionally, 60 percent felt that AI will lead to fewer jobs in a decade’s time, with 27 percent indicating that decline would be significant.
Watson is demonstrating the kind of feats artificial intelligence will continue to accomplish in coming years, especially involving tasks that process information with clear connections. AI will allow human workers to focus on more complex tasks that require creativity, critical thinking, communication and collaboration (also known as the four Cs). The fear, which the survey reflects, is that this transition will ultimately lead to significant job loss.
In the case of Jill Watson, no TA lost their job…yet. But if the AI can handle 40 percent of student inquiries, will they really need 8 humans in years ahead?
Another interesting aspect of this story is how students associated personality and characteristics to Jill. Even though the priority wasn’t to make a personable AI, there was a certain threshold of believability to be maintained that required emulating personality.
On its own, personality isn’t actually a concrete thing—it’s something we infer from a collection of behaviors that map onto a mental matrix we label as personality types. Given the constraints of an online course forum where one judges personality from Q&A exchanges, Jill seemed to pass most people’s rudimentary Turing tests.
With this experiment, it’s fairly safe to say that AI is coming of age. That means, we’ll be finding ourselves in more situations where we ask someone, “Are you a computer?”
Image courtesy of Shutterstock
