The unexamined life is not worth living. – Socrates
When Deb Roy’s son invites his girlfriend over for dinner with his family, there will be much more than photo albums to embarrass the teenager. His parents will have plenty of ammunition: 90,000 hours of video and 140,000 hours of audio. It’s the most comprehensive lifelogging event the world has ever seen, dwarfing a recent frame-by-frame photolog of an aging young girl. Moreover, the lifelog wasn’t compiled solely for family reminiscence, but for scientific reasons as well. As part of the Human Speechome Project, Deb Roy, an MIT cognitive scientist, is using this deluge of video footage to investigate early language acquisition. You can see his TED presentation on the subject, which includes some of the thousands of hours of footage, in the video below. From this pedagogical case study, his team also cleverly translated their budding ideas to human behavior in digital communities. With social networking and the growing prevalence of mobile devices with cameras, is this endeavor a precursor to even grander forms of lifelogging? If so, will scientists be equipped to make sense of world-logging – recording every moment on the planet – if it emerges? Only time will tell. For now, we all can marvel at Dr. Roy’s impressive feat of lifelog analytics.
Earlier this year, Deb Roy dazzled a TED audience with his presentation, “The Birth of a Word,” featured below. Because of the unprecedented nature of the project, the captivating visualizations, and the personal touch, the video garnered widespread attention. It even earned a tweet from Ashton Kutcher. If you haven’t seen it yet, do yourself a favor and allot 20 minutes of your time to watch the TED Talk in full. You won’t be disappointed.
Roy opens with a thought-provoking scenario:
Imagine you could record your life – everything you said, everything you did – available in a perfect memory store at your fingertips, so you could go back and find memorable moments and relive them . . . or sift through traces of time and discover patterns in your own life that had previously gone undiscovered.
Here, Deb Roy is talking about using lifelogging as a tool to answer challenging questions about human behavior and life in general. The question that he and his wife, a speech scientist, were trying to address has eluded child development experts for years: How does the infant’s tabula rasa acquire human language? It has been a subject of rigorous investigation since the titan of developmental psychology, Jean Piaget, offered his own theory of child language development. While Dr. Roy and his wife did not completely answer the question, their videos certainly shed valuable light on the matter.
There were some fascinating phenomena that emerged from Dr. Roy’s daunting dataset. He starts with some raw data at 4:59, where his son gradually improves his pronunciation of the word “water.” It’s actually quite beautiful to hear the word crystallize from its linguistic predecessor, “gaga.” In a recent CNN article, Dr. Roy eloquently referred to this verbal evolution as the “acoustic equivalent of a time-lapse video of a flower blossoming.” But what does it all mean? Well, the MIT team sorted through the hundreds of words in Roy Junior’s early vocabulary and found that he learned a word when the complexity of caregiver utterances reached a minimum (5:58). In other words, the parents and their nanny were subconsciously simplifying their speech to meet the nascent talker halfway. It seems to be related to “motherese” in which parents heighten the pitch of their voice in the presence of a newborn. This adjustment is thought to assist language acquisition. Here, the Roy family has revealed a previously hidden parent-child feedback loop in speech development by applying home video analytics. Jean Piaget and perhaps Bob Saget, another home video enthusiast, would be very proud.
Okay, so lifelogging can address problems of child linguistics. What about questions of a broader scope? Luckily, the MIT Media Lab had more tricks up their sleeves. By correlating “social hotspots” in the home with specific words that Roy Junior learned, the team built “wordscapes” (10:15), mountainous, data-rich structures describing utterance frequencies at particular locations. This spurred the imagination of Dr. Roy’s PhD student, Michael Fleischman, who envisaged translating this concept from the Roy household to the digitally-based community (11:19). In “virtual living rooms,” source content in mass media leads to commentary and discussion in social media. Previously, much like pinpointing how environment influences word births, it has been tricky to monitor how source content mediates responses in the digital world. However, by utilizing high-throughput media analytics, the MIT team was able to capture these media feedback loops. This is brilliantly depicted at 14:45 as a co-viewing clique, a fundamental unit of content-commentary interaction which includes the original source and the linked responses in social media. To see the President Obama’s State of the Union make a “nation explode in conversation,” check out 15:54. From ideas born from the unassuming archives of Roy Junior’s lifelog, the MIT lab has unearthed previously unknown dynamics of the mass media world.
As you could imagine, these media feedback loops might provide invaluable audience engagement information to advertisers, TV executives, and political campaigns. Using media-social media interactions as unsolicited crowdsourcing could literally transform Internet audiences into massive focus groups. To make this a reality, Dr. Roy and his former PhD student spearheaded Bluefin Labs, a company with the mission of helping organizations “link social media comments directly to their televised source—at scale, with precision and near real-time.” One of their tools, a social heatmap of television programming, is currently available on their website. At the moment, Bluefin is demonstrating their technology to Fortune 100 companies, and they expect a fully developed product by year’s end. I wonder if Roy Junior will get a cut? If the prospects of this technology tickle your fancy, check out the promotional video below to augment your excitement.
These are just the first steps of converting the vast stockpiles of videos, photos, and commentary into meaningful data. As social media is chronicling the lives of millions and cell phone cameras are approaching omnipresence, humanity at-large is building a worldlog of epic proportions. Almost certainly, there are valuable hidden insights within this treasure trove of human artifacts. Throw in unstructured data on science, politics, public health, and history, and the possibilities grow exponentially. We might be able to track how language is born not just in the individual, but civilization as a whole. Digging further, we might discover how cultural conditions can lead to scientific breakthroughs. The question is: Who is up to the Herculean task of elucidating global behavior patterns from this ocean of world knowledge? IBM has shown interest in this effort on a small scale with social media analytics, so perhaps a more computationally powerful version of Watson will be up to the challenge.
Also, what if we expand measurable life beyond the limits of sound, images, and text? All recorded human behavior is really just the end product of neural activity, so knowledge derived from advanced neuroscience might produce more fundamental insights than lifelogging through video. For a moment, imagine recording and interpreting brain activity among multiple individuals simultaneously. Now, imagine a whole community . . . then the world. The gates of social neuroscience would be wide open, transitioning worldlog analytics from behaviorism to biological understanding.
Alright, now I’m tiptoeing into science fiction, but in light of Deb Roy’s inspirational work, I can’t help but dream. His project foreshadows new tools that could change our world for the better. Through immediate and long range feedback loops observed in lifelogs and the hypothetical worldlog, we could improve our self-awareness, understanding, and behavior on both an individual and global scale. So, the next time you post a picture of your vacation on Flickr or upload your extreme sports exploits to Youtube, don’t think of it as an exercise in vanity. You are adding an infinitesimally small piece to an ever-growing library of human events that could answer our deepest questions. This echoes the bit of Socratic wisdom at the beginning of this article, a statement that really drives at the core of Deb Roy’s work and all the other lifelogging devotees out there. Socrates, if he were cryogenically unfrozen today, might amend his famous quotation: “An unexamined life is not worth living, and an unexamined world is not worth inhabiting. So take plenty of pictures! And don’t drink the hemlock.”
<Image Credits: TED (modified), Bluefin Labs (modified)>
<Video Credits: TED, Bluefin Labs>
<Sources: TED, Bluefin Labs, CNN>
