Ross Flood, of the Cerebral Palsy Society of New Zealand, gives the Neuro-Operated Utility System a whirl.
If you’re one of the millions affected by severe disabilities, you should pay close attention to three innovative, pure-hearted Aucklanders who are whipping up a fresh batch of hope. They call themselves Thought-Wired, and they’re building a brain-computer interface (BCI)-based system that bestows the handicapped the ability to control lights, make calls, manage A/C settings, and close curtains with their mind. By integrating the $300 Emotiv EPOC neuro-headset with smart-home appliances, the Thought-Wired team is summoning BCI technology from its ivory tower and helping democratize its applications.
In the proof-of-concept video below, the team unveiled their Neuro-Operated Utility System (NOUS) to the Cerebral Palsy (CP) Society of New Zealand. While donning EEG headsets to manipulate objects on a screen isn’t a new idea, seeing BCI technology that can improve quality of life is nothing short of inspiring. Despite some competition from other human computer interfaces, such as eye-tracking and speech commands, the NOUS has real potential to build or restore independent living skills for members of the disabled community without breaking the bank. After speaking with Dmitry Selitskiy, Thought-Wired’s team leader, I came to realize that their project was imbued with a human story and continues to be driven by unyielding willpower and a deep sense of compassion. If this isn’t an example of how technology can bring out the best in us, then I don’t know what is.
A Remarkable BCI Device for the Disabled
After viewing the demo above, I hope you can see the elegant simplicity and life-changing possibilities of Thought-Wired’s prototype. It kicks off with a short introduction of the BCI concept (0:31) as a team member telekinetically directs a virtual box. Then, Thought-Wired gets right into training with Ross Flood, a representative from the CP Society of New Zealand, who calibrates the virtual box dynamics with his brain waves (0:55). Once his EEG signals are trained, Mr. Flood makes a call for assistance with only his thoughts. Check out the Mr. Flood’s priceless expression as the phone starts ringing at 1:21. Pretty wild, ain’t it?
But what good is a smart-home equipped with BCI if you can only do one task? The NOUS has got you covered. By training additional thought commands, the user can toggle among several devices (1:25). In this example, Mr. Flood extends the reach of his mental manipulation powers to two lamps (1:45). Thought-Wired has also applied this BCI to rudimentary communication, using the lift and shift left thought commands for no and yes, respectively (2:01). In all of these cases, keep in mind that the virtual cube is not necessary to execute the command, and the thoughts correlated to the function are sufficient. With this device, the smart-home can become an extension of the mind.
This is an impressive prototype, but the pragmatically-minded may try to dampen our optimism by asserting that hands-free eye-tracking technology already allows the handicapped to interface with devices. Why bring the brain into it? There are several reasons. Firstly, eye-tracking is more expensive than you would expect, with the cheapest systems pushing past the $1000 mark. At $300, the Emotiv EPOC headset is the most costly component of the NOUS, so once it comes to market, I’d expect the price to be well below most eye-tracking devices. While the NOUS is in the early stages, eye-tracking has been around for years, and there are still some kinks to work out. Spontaneous saccades can disrupt eye-driven commands, and only recently have involuntary eye-blinking issues been resolved in research circles. Plus, who wants to turn the heater up to 90 degrees if they’re visually distracted? In my opinion, BCI trumps ocular tracking because it requires neither eye nor hand to execute commands.
So, for now, the NOUS has eye-tracking beat, but Thought-Wired isn’t out of the woods just yet. Its stiffest competition will come from speech command devices for smart homes, which mirror the NOUS’s basic utility (handless interfacing) with technology you can find in your Droid. Unlike eye-tracking, speech recognition is relatively inexpensive, and if the user can speak, there is no need for additional training. Furthermore, aside from the home automation appliances, the only hardware you need is a pair of functioning vocal cords. However, these technical advantages fly out the window if the disabled individual has significant speech impediments, as is often the case in cerebral palsy. Therefore, I believe the subset of the disabled community that stands to benefit the most from the NOUS are those who have a coincident speech disorder or are entirely nonverbal.
NOUS Concept. Using a Windows-based NOUS application, the user will be able to control a variety of objects and functions.
Aside from other human computer interfaces, there are also the inherent limitations of EEG, the brain activity interpreted by the NOUS. Unlike implantable BCIs, the Emotiv EPOC relies on these inferred neural signals that are diluted by tissue and cranial bone. As a result, direct neural interfaces have higher spatial resolution, easily surpassing the binary “up/down” “on/off”, “left/right”, and “yes/no” commands of most EPOC-based devices. However, while EEG can only take BCI applications so far, direct neural interfaces are currently invasive and cost-prohibitive. Beyond the cost of the electrode arrays, there are also the operational costs of surgical implantation and maintenance. Until the financial burden is eased and long-term biocompatibility is verified on a large scale, EEG-based devices will have the greatest potential for widespread use in the near future. However, I’m sure Thought-Wired will have their eye on direct neural impants as the cost and safety concerns are sufficiently addressed by the BCI wizards. Furthermore, with programming ingenuity, long neuro-cognitive training sessions, and some true grit, the EPOC can still do some amazing things. For example, while it’s far from freeway ready, the EPOC-based BrainDriver provides a mind-guided semi-autonomous driving experience. As the NOUS moves from prototype to market-ready product, I expect Thought-Wired will accomplish equally amazing feats.
When the NOUS or a similar BCI device reaches the market, the potential benefits for the severely disabled, their families, and society-at-large cannot be overstated. Commonly, individuals with movement disorders and speech problems fully depend on family members or reside in state-funded facilities. For many, it could be the first time that these individuals can partially transcend their condition and claim a sense of freedom. Furthermore, without needing to assist the BCI user with certain household tasks, caregivers could focus on helping in other ways (i.e. academics, emotional counseling). Also, group homes and hospitals that care for the disabled could shift their treatment goals from day-to-day support to offering other services (i.e. physical therapy), leading to higher clinical efficiency. For primary, secondary, and tertiary stakeholders, the NOUS could be a win-win-win.
From Left to Right: Harvey Brunt and Ross Flood of CP Society of NZ with Dmitry and Konstantin Selitskiy of Thought-Wired.
The Brains Behind the BCI: The Story of Thought-Wired
Rome wasn’t built in a day and neither was the NOUS. Also akin to Roman history, Thought-Wired’s humble beginnings are just as remarkable as the final product. Like any ambitious project, it started with an idea. Well, maybe two ideas with their origins in a pair of TED Talks.
September 26th, 2010 started like any other for Dmitry Selitskiy, a University of Auckland post-graduate student. However, on this day, he was summoned to the TEDxAuckland conference. As Hub followers should know, TED Talk’s are generally thought-provoking and tend to jumpstart the imagination, but there was one speaker, Cindy Gallop, whose message resonated with Mr. Selitskiy. She revealed IfWeRanTheWorld, a project that aims to translate good intentions into action. During the talk, attendees were encouraged to submit their world changing ideas to the site. Dmitry’s idea came rather easily.
I wanted to (somehow) help kids with disabilities be equal with abled people. I have a personal connection to the issue of disability – my cousin has a severe case of CP which left him paralyzed from birth.
By either serendipity or mere happenstance, Dmitry stumbled upon another TED Talk online the following day. It was Tan Le’s spellbinding presentation of her Emotiv EPOC neuro-headset, and Dmitry was immediately enthralled after watching the BCI in action. At the closing of the talk, Tan Le urged developers to adapt the platform device to new applications. This represented a TED-inspired call-to-arms for Dmitry. “That same day we put together a proposal for the Auckland University Imagine Cup competition. So that’s how we started.” If you had any doubt that TED Talks have ideas worth spreading, you may now stand corrected.
With his father (a veteran entrepreneur and B-school graduate student) and another University of Auckland student, Dmitry and Thought-Wired embarked on a mission to bring BCI to the disabled. Their proposal for Microsoft’s Imagine Cup fared well, placing second in the University competition and earning them $500NZ in prizes to support the project. They also posted their project on Uncle Percy, a New Zealand based savings website, so family and friends could track their fundraising progress. To their surprise, this led to a wave of donations, and they had accrued enough cash within a week to move forward. It just goes to show that you shouldn’t have to look too far to find good angel investors.
Things started rolling, but that’s not to say there weren’t a few hiccups along the way. One major issue was that the New Zealand home automation market was limited, so many of the electrical components had to be sourced overseas. However, this presented technical issues because of the disparate voltages and frequencies of U.S. and New Zealand alternating currents (110V/60Hz vs. 220V/50Hz). Despite this setback, the team pulled through and was able to acquire all the necessary parts for the NOUS prototype.
That wasn’t Thought-Wired’s only hurdle. The team was academically diverse, ranging from information management to chemical engineering, but no one had a background in cognitive neuroscience. Consequently, Thought-Wired had to conduct a self-taught crash course in BCI. In six months, the team “had to learn very many things, from basics of EEG to automation protocols that communicate over power lines through many things around programming.” Now that’s a bucketful of sweat equity.
Prior to developing the NOUS prototype, Dmitry had the opportunity to provide a BCI preview for his cousin living with cerebral palsy on a holiday trip. After a few training sessions with the virtual cube, the boy was able to reproduce the right, left, and lift mental commands. According to Dmitry, it was a pretty mind-blowing experience for the whole family.
For the first time the boy could control and move an object in his life. We could all see how excited he was (not to mention our excitement). Now when we speak or bring out the headset, his eyes [light] up . . .
Dmitry explaining the NOUS to an audience at the Imagine Cup.
After unveiling the prototype, the Thought-Wired team ultimately came up short at New Zealand’s Imagine Cup final, but their resolve remains stronger than ever. It’s also evident that Thought-Wired’s vision extends far beyond the NOUS. Dmitry sees a world where BCI is “just as common as traditional interfaces that we routinely use . . . BCI may end up playing a big part in areas that we don’t even think about at this stage just because this is the early days of this technology.” In short, we can expect that BCI devices will reach the ubiquity of smart-phones and broaden the capabilities of both the handicapped and able-bodied. I have to agree with Dmitry on this point, for BCI is already invading our iPhones.
The impact of severe disabilities for the world, nations, families, and individuals. As you can see, saying the Thought-Wired project is ambitious is an understatement.
When I asked Dmitry how he will feel when his cousin uses the completed product, he replied, “That we made the first big step. Operating devices is only the basics, there is so much more that needs to be done: communication and mobility are the two big challenges after that.” That’s what I call intestinal fortitude. Godspeed, Thought-Wired. Godspeed.
If you’d like to follow Thought-Wired’s progress, you can Like their Facebook Page to get the latest updates. While you’re at it, go ahead and Like Singularity Hub if you haven’t already. Just saying.
You can find Singularity Hub’s full interview with Dmitry Selitskiy below:
Singularity Hub: What are the technical, educational, and/or professional backgrounds of the Thought-Wired team, and how did you guys meet?
Dmitry Selitskiy: We were (and some are) all students at the University of Auckland. Me and Mark took same classes as part of our BCom (Hons) in Information Systems. Konstantin is doing Masters of International Business here. Mark has completed his degree at the end of last year and is currently doing a grad program in New York. I previously graduated from the BBIM program at the University of Auckland in 2008. After that I worked for 2 years full time in a small IT company providing system integration and EDI solutions. Konstantin (who is my father I should have mentioned) is a qualified chemical engineer. He then went on to be an entrepreneur for more than 15 years. He relocated to New Zealand a year ago and decided to study International Business full time last June. So overall we are a very diverse team…
SH: That’s great. An academically and experientially diverse group of people. How did the idea of combining BCI and smart homes come about?
DS: Last year in September I attended a TEDx event here in Auckland. Cindy Gallop spoke at the event and presented her new project ifwerantheworld.com. As part of that attendees were encouraged to submit their world changing ideas to the site. My idea was to (somehow) help kids with disabilities to be equal with abled people. I have a personal connection to the issue of disability – my cousin has a severe case of CP which left him paralyzed from birth.
Anyway, the next day I randomly stumbled upon a TED talk by Tan Le, the CEO of Emotiv, the company that makes brain-computer interface headsets. She did a live demonstration as part of the presentation and it blew me away. If anyone could control an object on the screen with less than 10 minutes of training and without anything being implanted in the brain then that must be the solution to the problem that I raised previously.
That same day we put together a proposal for the Auckland University Imagine Cup competition (this proposal is available on our Scribd page by the way). So that’s how we started.
SH: That Tan Le talk was pretty spectacular, and the Emotiv EPOC has lots of potential. The last part of her talk was sort of a “call to arms” for developers out there like you. Besides seeing the TED talk, what were the main factors that led the team to decide on the Emotiv EPOC and not another EEG-based device, like Neurosky or a traditional EEG cap?
DS: A number of things… Neurosky while cheaper is a far less capable device. It has only one sensor and only one detection – concentration. This would be very limiting for building a comprehensive system. However, we are considering on working with Neurosky in the future. As for the traditional EEG devices, we did not go down that route because we lack neuroscience background required to interpret raw EEG data provided by medical EEG caps. Emotiv does the interpreting and gives us manageable commands. It has an SDK that allows us to easily tap into those detections. That being said, we also consider working with other EEG devices in the future when we have the necessary knowledge and skills.
SH: How involved is Emotiv with your project, and what assistance (if any) do they provide?
DS: Not very involved. Or maybe I should say not involved at all at this stage. We know they have seen our MS Imagine Cup proposal but that is as far as it goes. So far we have not asked for any assistance from them.
SH: Do you receive financial support from the Imagine Cup? How is the project supported financially?
DS: We came second in the University Imagine which gave us $500NZ cash prize and some gadgets that we also used to fund the project. We also raised the money for the headset using a New Zealand website www.unclepercy.co.nz. That was a big surprise since we did not plan on using it to raise money from people but rather just track our progress of getting the money together. Instead we ended up collecting the whole sum in about a week from our friends. That gave us a big confidence boost.
Now that we are in the NZ finals of the MS Imagine Cup we have a chance of winning bigger cash prizes. I think the winner receives $5000NZ. Otherwise, the Cup doesn’t really support us financially. Only through prizes.
SH: How did Thought-Wired translate the Emotiv hardware to this practical application? What contributions did the core TW team members make?
DS: Emotiv headset and the SDK is only a part of the system. The commands from the headset are then fed into our software which allows to perform actual tasks, for example operate an electric device. In this case, the software then talks to the devices. I can send you a diagram later to make it easier to understand where things fit in the system.
SH: Wow. So basically ready to go out of the box? Is it the same diagram from the Imagine Cup proposal?
DS: The idea is the same, yes. At the moment, it is just a proof of concept though and far from an actual product. But the intention is that yes, it will be a package that includes everything that you may need to operate many different devices through a common interface.
SH: What are the capabilities of the first prototype?
DS: Two main functions: device switch and a yes/no on-screen communication tool. It is capable of handling an unlimited number of devices which can be added or removed through application’s configuration.
Basically, the application allows you to operate (on/off) your configured devices. A video will help the explanation… We should have some kind of video in a week or two.
SH: That’s great, I look forward to it. It sounds like it has come pretty far along. Have there been any significant technical challenges or setbacks on this project? If so, could you describe them? How did the team overcome them?
DS: It’s a steep learning curve for sure. In the past 6 months we had to learn very many things, from basics of EEG to automation protocols that communicate over power lines through many things around programming.
It was a challenge to source the required electrical components since home automation market is tiny in New Zealand. Ebay helped a lot but also brought the challenges of fundamental differences between US and other countries in terms of power supply (110V/60Hz vs 220V/50Hz). The biggest challenge really is to learn a lot quickly without making too many mistakes especially those that cost money. But so far I think we’ve been doing well on this front besides an odd product return due to incompatibility…
Our main challenge at the moment is the final presentation for the MS Imagine Cup and especially the creative design aspect of things. But this is more competition related rather than the project overall.
SH: What setting is TW targeting first? It seems from the Imagine Cup proposal that you wanted to start in hospital settings and eventually move to group homes. Is that still the plan?
DS: Hospital and other managed care facilities seem like a good testing ground for us because of their existing infrastructure and skilled staff. However, now that we are comfortable with the home automation side of things we may reconsider this somewhat. This is actually something that we going to have a discussion on very shortly now that the technical side of the prototype is completed.
SH: Has the prototype been tested on any people with disabilities yet, or will that start once TW starts trying it out in hospitals?
DS: We have approached a number of organisations here in NZ to show the prototype and have it tested. Next Tuesday we are going to the Cerebral Palsy Society of NZ. So far they have been very enthusiastic about it. We all hope that tests with them will produce good results. We are definitely very confident with what we have at the moment.
SH: Beyond the Imagine Cup and the first hospital trials, what are some other short and long term goals of Thought-Wired?
DS: The shortest term goal is to do the best we can at the NZ Imagine Cup final to be able to compete in the world final in New York in July. Next we are planning in competing in Spark entrepreneur challenge (http://www.spark.auckland.ac.nz/) and thought that to put the focus back on the business side of things. The ultimate goal, of course, is to deliver a product to market.
SH: How do you think you will feel on the day that your cousin is able to use the final product of Thought-Wired?
DS: That we made the first big step. Operating devices is only the basics, there is so much more that needs to be done: communication and mobility are the two big challenges after that.
SH: That brings me to my next question. In general, where do you see BCI technology going in the next few decades?
DS: It’s the very early days. The only thing I can say for sure is that it may well become just as common as traditional interfaces that we routinely use. I definitely hope that it will make a big contribution in the disability field. However, there are a lot of other areas where it can produce great results: gaming (of course), art and performance and others. I also think that BCI may end up playing a big part in areas that we don’t even think about at this stage just because this is such early days of this technology.
<Images: NOUS video (screen capture), Thought-Wired Imagine Cup Proposal, Thought-Wired>
