Robots won't be replacing lawyers anytime soon, but computer programs are already acting like overachieving interns.

First doctors, now lawyers. I can hear them now: “I didn’t go through all those years of school just to be replaced by a computer.”

As Watson bones up on his medical knowledge in hopes of giving doctors a helping hand, lawyers are becoming increasingly antsy over computer programs that are finding their way into the law firms–and doing something lawyers do but faster, better, and cheaper.

The part the computers are hijacking is a step early in the process of two parties suing each other. Called “discovery,” each side asks the other to produce information relevant to the case. As you can imagine this is a very labor-intensive and costly endeavor. Back in the day when information was entirely in the form of documents, it wasn’t uncommon for companies to have warehouses filled with boxes containing files. Lawyers and paralegals would scour one pile after another, hour after hour, day after day, looking for something that might be important. Today, the vast majority of a company’s information is stored electronically. Gone are the days of peering over stacks of paper, but the electronic data jungle is formidable in its own right. Pertinent information can be stored in a multitude of formats. Microsoft Word documents, Excel spreadsheets and PDF files are common forms of documentation. But relevant information can also be stored as video or audio media, and of course the juicy format that always seems to nail the big guys: email. “E-discovery” became a part of the litigation vernacular to describe this new electronic discovery process. It’s vastly superior to a paper search, but e-discovery still requires digging through an enormous amount of material.

Given the fact that the vast majority of a company’s information is electronic, it was only a matter of time before e-discovery software was developed. It comes in two main flavors. A ‘linguistic’ approach looks for specific words or associated words or phrases. A filter for “baseball” will pull out documents containing “bat” or “hotdog.” More sophisticated software takes a ‘sociological’ approach by adding to the linguistic analysis an inferential element. Like a detective, the software uses seemingly unrelated information such as who talks to whom, and when, where and how they talked to deduce a conclusion.

Because “relevant information” often means illegal activity, ads for e-discovery software companies often sound more like ads for security companies. Silicon Valley’s Cataphora has developed software that takes an advanced sociological approach: it detects changes in a person’s behavior. Any “deviations from normality…is where any bad acts will always be found.”

The technology first models the “normal behavior” of an organization. Their normal behavior is derived from the aggregate of electronic information stored in the computer systems and networks of the company (I guess it’s a safe assumption that the majority of the company is not in on the bad behavior). To determine the baseline communication patterns they mine documents, emails, phone logs, calendars, and even keycard entry logs. This data is then used to characterize people’s behavior: their communications but also their document flow and meetings schedule, both inside and outside the company. Once they’ve got the baseline they can spot anomalous behavior. For example, in the image to the left normal behavior–based either on the individual’s past patterns or those of workers who have similar duties–is represented by blue lines connecting him with coworkers within the company (grey network) or his friends and family outside the company (green). But when he suddenly begins to interact significantly with competitors (red) the “anomalous behavior” flag goes up.

It can also detect “call me” moments–times when the employee suspiciously switches to another type of communication as if he doesn’t want to leave an electronic trail. Most impressively, I think, is Cataphora’s ability to detect when the tone of the employee has changed in an email. It tracks moments of “loud talking”–statements of increased emphasis that might indicate that the sender is stressed. It can also detect subtler changes in tone, as when the writer goes from informal to formal. Like the troopers’ eye that’s drawn to the car driving the speed limit–you got guilty written all over your subject-verb agreement.

A famous case of insider trading gone unnoticed is that of Jérôme Keviel. The stock broker was caught in 2008 for illegal trading that ended up costing the French bank Société Générale $7 billion. He was able to get away with the trades because his behavior did not violate the bank’s own compliance monitoring system. His behavior, however, was abnormal. Among the anomalies Cataphora claims they would have picked up on are routinely overriding chain of command (going to his second-level manager unusually often) and working excessively with just one broker. By themselves these behaviors wouldn’t attract much attention as they don’t break with company compliance. But in the context of normal behavior, they would have gotten Keviel’s butt caught. It’s a signal-to-noise extraction that a room full of humans probably aren’t going to get wise to.

The same data can be sifted for signs of sexual harassment, employee discrimination, or unfair hiring practices. And not to be limited to the identification of bad apples, the sort of artificial intelligence behind e-discovery can be used to pluck out hard to find diamonds in the rough such as those who produce valuable content that is reused by members of the company network.

E-discovery is just the latest technology to threaten the workforce by automating tasks historically performed by humans. Electronic design automation software took over the design of printed circuit boards and integrated circuits previously done by hand. Automation software is also lightening the workload for loan and mortgage officers and tax accountants. Even scientific investigation is being automated. Computers can scan the genome, come up with their own hypotheses as to what particular genes might do, and then perform experiments to test those hypotheses. This will, of course, inevitably lead to a dispute between the computer and the gene sequencer as to who should be first-author.

Even without e-discovery siphoning away their billable hours lawyers have had it rough in recent years. Like many industries, the recent recession has hit law hard, prompting law firms to lay off thousands of associates, de-equitize partners, slash budgets, and arrest hiring. What the recession didn’t do was slow down law graduates. The Wall Street Journal reported that in 2005 90.6 percent of summer law interns were offered full-time jobs upon graduating. In 2009 that number dropped to 69.3 percent. In fact, today’s law graduates are entering the worst job market in 15 years. With its ability to cut law office staff, isn’t it understandable that the rise of e-discovery–a leading analyst firm declared e-discovery market to be entering a high growth phase–is seen as a threat to new graduates and law firms alike?

I object, your honor.

Maybe I don’t know enough about the field, but it seems to me that increasing efficiency is generally a good thing. I understand that times are so tough right now that lawyers, to keep hours, are now doing tasks typically handed off to paralegals, such as discovery. And the last thing they need is for computer programs to muscle in on the already limited opportunities. But if we take a step back and acknowledge that legal fees, in general, are really freakin‘ high and that the average person, even before the Great Recession, can’t afford a lawyer, then maybe e-discovery becomes an opportunity instead of a threat. In a 1978 a high-profile antitrust case between the Justice Department and CBS the discovery phase included sifting through six million documents. It took a small army of lawyers and paralegals months to get through the documents. And it cost $2.2 million (remember, that’s 1978 dollars). Compare that to a case earlier this year in which e-discovery software was used. Blackstone Discovery’s computers went through 1.5 million documents. They completed the task in a fraction of the time and the cost was a very affordable $100,000. If law firms transferred those savings–savings of both money and time–to clients they would undoubtedly get more clients.

More clients = more revenue = more hires.

I may be missing something here, but it seems pretty straightforward to me. Don’t fight the technology, find a way to use it to your advantage. From the farms, to the clinic, and now law offices, technology is going to continue to upgrade the workplace. We just need to make sure we keep pace by upgrading the humans in the workplace too.

[image credits: 2.bp.blogspot.com, Cataphora, and Case in Point]
image 1: Lawyerbot
image 2: Cataphora
image 3:

As little as 10 years ago the petabyte seemed just as large and amazing as the yottabyte sounds today. Now it is common for companies such as Google or Facebook to hold several petabytes of information in just one of their many datacenters across the world.

In one sense the yottabyte is no big deal. I mean, its just another rung on the ladder of names we give to the ever increasing amounts of data around us, each rung growing by an order of 1,000 starting with the kilobyte. Ten years from now the yottabyte will go the way of the petabyte as the term that once represented enormous amounts of information but now is just another drop in the bucket.  It will be replaced by a new term that represents 1 million or even 1 billion yottabytes.  Nevertheless, it is fun and also illustrative to spend a moment to ponder the yottabyte, which is indeed an enormous number by today’s standards.  And let us not ignore the fact that yottabyte when pronounced with a long oh sounds like “yoda byte” – now that’s awesome!

Attack of the bits!

Lets first build up to the yottabyte in today’s standards. A one page Microsoft Word document is anywhere from 50 to 100 kilobytes (KB). A picture from your camera is typically anywhere from 0.5 to 3 megabytes (MB) and a song you might download from Itunes is usually about 3 or 4 megabytes (MB). Moving up the ladder, popular consumer devices such as iphones, ipods, and digital cameras hold anywhere from 1 to 100 gigabytes (GB) of storage capacity. The top of the line hard drives that you can buy at Best Buy are now 2 terabytes. Assuming Google has 1 million servers each with 1 terabyte of storage (a size Google has already reached or likely will reach in the next year) we can estimate that Google has leap frogged the petabyte and now boasts a total worldwide storage capacity of roughly 1 exabyte. The storage capacity of the approximately 1 billion personal computers worldwide in 2009 with an average storage capacity of perhaps 250 gigabytes (GB) of storage each is not even half a zettabyte!

A yottabyte would equal about 1 million times the current storage capacity of Google, or about 2000 times the storage capacity of every personal computer in the world in 2009.  Here are some other fun stats I have gathered from across the web, with sources:

• In 2006, the amount of digital information created, captured, and replicated [in the world] was 1,288 x 1018 bits. In computer parlance, that’s 161 exabytes or 161 billion gigabytes. This is about 3 million times the information in all the books ever written. (source)
• Between 2006 and 2010, the information added annually to the digital universe will increase more than six fold from 161 exabytes to 988 exabytes. (source)
• All hard disk capacity developed in 1995 – 20 Petabytes (source)
• All printed material in the world – 200 Petabytes (source)
• All words ever spoken by human beings – 500 Petabytes (source)

We must keep in mind of course that total storage capacity does not equal the total amount of information.  Google might have an exabyte of storage capacity, but perhaps only 70% or less of this capacity is even being used.  Of the capacity that is used, a majority of it is simply used to replicate data perhaps 3 or even 6 times over as backup in case of data loss.  This implies that as little as 10%, or 100 petabytes worth of Google’s exabyte capacity, is being used for real, unique information.  All of these numbers are rough estimates, but you get the idea.

The exponentially increasing amount of information being created and stored around the world is one of the central phenomenon upon which our technological progress and future is based. And no matter how you look at it, the total amount of information being stored and processed across the world is fascinating. Yet in spite of the impressive amount of information being digitally created and stored each day by the human race, this is still nothing compared to the total information content of the world and the universe at large that could be recorded.  Today a large portion of the digital information that is being created comes from individuals, corporations, and governments that are taking photos and videos.  As lifelogging and 24/7 surveillance become ever present in our lives the amount of photos and videos being recorded and archived is set to absolutely explode.  Imagine the amount of data that will be collected over the next decade or two when every single individual on the planet is equipped with a device that will record every minute of their lives 24/7.  Indeed, when that day arrives the yottabyte will be just a small drop in an enormous sea of life streams, experiences, and knowledge of mankind encoded into bits.

In the last few months @techcrunch (1,000,000+ followers), @tonyrobbins (1,323,000+ followers), and @timoreilly (1,087,000+ followers) have each retweeted one of our stories.  So what came of these massive tweets?  Did enormous numbers of followers read the tweet and subsequently follow the link to our story here at Singularity Hub?  The answer, at least to us, has been a surprising disappointment.  After reviewing our traffic logs we can attribute no more than 5,000 hits to our site from any one of these mega account tweets (this includes hits from retweets as well!).  From an account of roughly 1,000,000 followers no more than 5,000 of those followers, or roughly one half of a percent of them, are likely to click on the tweet.  In our experience 5,000 would actually be on the upper limit considering retweets were involved, with about 3,000 being more likely.

Thanks for the retweet Tony!

One might also expect that being picked up by a major account such as @tonyrobbins might result in large numbers of new followers following your own Twitter account.  But again the truth is disappointing.  Our data shows that getting retweeted by a mega account only results in perhaps a few dozen new followers to your own Twitter account.

How can we explain that such a low number of supposed followers actually read and click through on the information in a tweet?  There are several contributing factors to explain this, but here are what I think are the major ones:

1. Large numbers of these followers aren’t really followers at all

Many of these so called followers are robot accounts automatically following all sorts of Twitter accounts.  Other followers are real people, but they are following so many different Twitter accounts that they don’t really tune in seriously to most or all of their incoming tweets.  For whatever reason, either they are using their account for self promotion rather than to get information or they have long given up on using their account, lots of followers are simply ghosts with nobody there to actually read the tweet.

2. Even dedicated followers will click on a small number of tweets

Even when counting the followers that are true, dedicated followers that really want to read the tweets, statisically speaking we can only expect these followers to click on maybe 10% of the tweets they see.  It is just not practical that all or most of the tweets will be of sufficient interest for a real follower to click on them and read them to their fullest.  When we go to a newspaper, or a blog, or a Twitter feed, we don’t read every single story.  We skim through the list of choices and only focus on the one or two stories out of dozens in the data feed that interest us the most, then we move on to the rest of our lives.

Information Overload!

Despite all of the hype and excitement around Twitter we can see that Twitter is currently a very low penetration medium for distributing information.  Mega accounts with more than 1,000,000 followers aren’t nearly the powerhouses over people’s information lives that many of us may have expected them to be.  But before we call the game over and discount Twitter and its mega accounts we need to consider that things can – and will – change.

The Twitter service is just in its infancy and we can expect the information paradigm that it represents to evolve rapidly and significantly in the coming years.  In fact focusing only on Twitter the company is too short sighted of a way to look at all of this.  Twitter is a company, but the paradigm of sending short messages of information to legions of followers who can then re-send those messages (ie retweet them) to others is a theme that is bigger than the company itself.  Twitter the company may come and go, succeed or fail, but the paradigm itself will live on and evolve.  Even now an entire ecosystem of thousands of companies is building up around not just Twitter, but around the idea of the realtime exchange of bits of information.

In the coming years, whether it be on the Twitter platform or some other framework from another company, or even from a distributed open source collective , we can expect many of the major problems with today’s Twitter to be solved.  For one thing, we can expect that there will indeed be massive Twitter accounts whose followers are much more committed to absorbing the information beyond the 0.5% witnessed today.  Take for example official government Twitter accounts in which perhaps the entire citizenry of a nation can be expected to read a tweet that is critical to national security, pride, or interest.  Would more than 0.5% of citizens in the United States follow and then read the tweet of a government sponsored account meant to distribute breaking news of terrorist threats or viral outbreaks?  I think so.

Not only can we expect certain accounts to have more dedicated followers, but we can expect tools to manage the overwhelming stream of information coursing into our lives- tweets, blogs, feeds, images – to become more sophisticated and efficient at helping us to sort through all of the mess.  The most important and relevant information to each of us will bubble up above the rest.  Information more and more will be disseminated to the right people who actually care about that information enough to click on it and read it.  This contrasts strongly with today’s situation in which we are overwhelmed with huge streams of information that are too massive and diverse to manage effectively.

Information is coming at us from every direction at levels far beyond normal human capacity to absorb.  And yet even as tools get better at helping us to deal with today’s information overload, the influx of new information into our lives will only multiply even further, making the problem ever more difficult to manage.  Paradigm shifts in information access and management are happening ever faster to compensate.  Are Twitter mega accounts and Twitter itself already a dying paradigm in this ever evolving race, or are Twitter and its mega accounts going to maintain or even increase their relevance by evolving along with everything else?