By William WebberMy previous post described in some detail the conditions of finite population annotation that apply to e-discovery. To summarize, what we care about (or at least should care about) is not maximizing classifier accuracy in itself, but minimizing the total cost of achieving a target level of recall. The predominant cost in the review stage is that of having human experts train the classifier, and of having human reviewers review the documents that the classifier predicts as responsive. Each relevant document found in training is one fewer that must be looked at in review. Therefore, training example selection methods [...]
Tuesday, September 30, 2014
Total Review Cost of Training Selection Methods
Monday, September 29, 2014
Is the IRS’s Inability to Find Emails the Result of Unethical Behavior?
By Ralph LoseyThe IRS seems inherently incapable of finding emails. The most famous incident that everyone has heard about, and many have complained about, is the loss of emails of key witnesses in a Congressional investigation of the IRS tea party targeting scandal . In June 2014 the IRS admitted that it could not find many of the emails of the key witness, Lois Lerner. Lerner is the first IRS official to admit that agents had improperly scrutinized tax exempt applications. When she was subpoenaed to testify before Congress, she plead the Fifth Amendment and refused to answer any questions. She is not the only IRS official [...]
Friday, September 26, 2014
Trademark Infringement in Blogging? (Cartoon and Clip)
Keeping up with the many comments and commentators in the data discovery and governance blogosphere can be quite challenging given the multitude of information, opinion and news blogs. This week’s cartoon and clip features a unique challenge to today’s bloggers (cartoon) and a non-all inclusive running list of approximately 30 recent and relevant eDiscovery and information governance related blog posts (clip).
Click here to find running listing of the approximately 30 recent and relevant eDiscovery and information governance blog posts as shared on the ComplexDiscovery blog.
Thursday, September 25, 2014
At That Time: The Dynamo Case on Predictive Coding
By Hal MarcusOn September 16, I posted about the impact of timing on predictive coding in light of recent caselaw. Timing indeed being everything, an intriguing new opinion on this point was handed down the very next day. In Dynamo Holdings v. Comm’r , the IRS Commissioner sought to compel production of the contents of backup tapes containing at least several million documents. It objected to the producing parties’ request to use predictive coding to review them, calling it an “unproven technology.” In his opinion, Judge Ronald Buch rejects the IRS’ motion, detailing the substantial judicial and industry acceptance of predictive coding. He emphasizes the discretion that producing parties have in conducting document review, bluntly questioning why the use of predictive coding should even be before the court [...]
Tuesday, September 23, 2014
The Two Deadly Sins Hobbling Technology-enabled Solutions in eDiscovery
By Gerard BrittonThere are some interesting observations about predictive coding’s “failure to launch”.Here are two recommendations that mitigate most if not all of the impediments to technology-enabled solutions: inordinate cost and unsatisfactory performance. Cost: Reduce Prices Most knowledgeable people in the industry are aware that predictive coding vendors adopted a discount-off-of-the-alternative-manual-review-cost pricing model that assured hefty profits but bore no rational relationship to their own actual costs.Not only did this in some cases result in no actual savings; most importantly, it has occasioned the re-introduction of a process that advocates promised it would replace: key word culling. As the recall metrics [...]
Why NOT Use PC-TAR?
By Greg BucklesExtract: So here are my Top Ten Reasons Why NOT [To Use] PC-TAR:
- Perception that PC-TAR costs front load the discovery cost for matters that WILL settle before trial.
- High resistance to analytic upcharges. Have to justify them on every matter, so go with path of least resistance.
- Complexity of systems and fear that counsel will not be able to defend what they do not understand.
- Customers on information overload. Marketing fatigue and growing customer indifference.
- Perception that PC-TAR reinforces known relevant selection and misses unknown/new documents.
- Rumors of SEC/DOJ in some areas fighting PC-TAR proposals.
- Realization that 95-99% recall in PC-TAR training will result in 300-500% production size. Exposure of large volumes of non-relevant ESI a serious concern for companies facing serial plaintiffs that are on fishing expeditions.
- Mature corporate customers already cull and optimize during collection or processing. If they can achieve substantial savings prioritizing/clustering review sets, why pay for actual PC-TAR analytics?
- Counsel does not want to operate PC-TAR systems. Wants Litsupport or provider to run it.
- PC-TAR takes money from the firm. Takes away associate jobs.
Do I believe that most of these market perceptions are true. No, I do not. However, they all contribute to the slow adoption rates for actual use of machine learning technology in traditional discovery review for production.
Monday, September 22, 2014
Over Delegating e-Discovery Preservation and Search to a Client?
By Ralph Losey
Bad things tend to happen when lawyers delegate e-discovery responsibility to their clients. As all informed lawyers know, lawyers have a duty to actively supervise their client’s preservation. They cannot just turn a blind eye; just send out written notices and forget it. Lawyers have an even higher duty to manage discovery, including search and production of electronic evidence. They cannot just turn e-discovery over to a client and then sign the response to the request for production. The only possible exception proves the rule. If a client has in-house legal counsel, and if they appear of record in the case, and if the in-house counsel signs the discovery response, then, and only then, is outside counsel (somewhat) off the hook. Then they can lay back, a little bit, but, trust me, this almost never happens.
Saturday, September 20, 2014
Unitization: E-Discovery’s Achilles’ Heel
By Matthew Adams
By now, it is common knowledge that an e-mail thread is generally organized with the oldest string of the communication at the end of the document, and the most recent string of the communication at the beginning of the document. It all makes good logical sense, right? The hidden Achilles Heel of e-discovery is a set of ESI that is not so orderly and coherent. Ensuring that you do not fall for this common e-discovery trap is simple, and hopefully with some awareness of the issue from the outset, you will not fall prey to an un-unitized, and therefore functionally useless data set, whether [...]
Friday, September 19, 2014
Even A Judge Questioned Why Ask for Permission to Use Predictive Coding
By Josh GillilandI do not normally want to high five Federal judges, but Judge Ronald Buch, a Tax Judge in Texas, sure deserved one after his Dynamo Holdings opinion.The discovery dispute can be summed up as a battle over backup tapes that had confidential information. The Requesting Party wanted the tapes; the Producing Party wanted to use predictive coding to produce what was relevant, because the cost for reviewing the material for privilege and relevancy would cost $450,000 with manual review. Dynamo Holdings v. Comm’r, 2014 U.S. Tax Ct. LEXIS 40 (Docket Nos. 2685-11, 8393-12. Filed September 17, 2014.) The Requesting [...]
Thursday, September 18, 2014
Innovation in eDiscovery? (Cartoon and Clip)
The Cartoon and Clip of the Week for September 18, 2014
Keeping up with innovation in eDiscovery can be quite challenging given the various approaches, commentators and providers weighing in on each real or perceived innovation. This week’s cartoon and clip features a strategic approach to driving innovation (cartoon) and a non-all inclusive running listing of mergers, acquisitions and investments in the eDiscovery arena (clip).
Click here to find running listing of the latest publicly available eDiscovery related merger, acquisition and investment activity as shared on the ComplexDiscovery blog.
Black Swans and Information Governance
By Jason BaronIn the absence of a black swan recently happening to you and your organization, how can you convince the powers that be that they should take some preventive and/or precautionary course of action to stave off a subsequent disaster? These questions have direct relevance to the matter of “selling” information governance to the C-suite in our increasingly Big Data world.
Wednesday, September 17, 2014
Federal Judiciary Approves Civil Discovery Rules Changes
By Zoe TillmanFederal judiciary officials on Tuesday approved proposed changes to court rules that could reshape how discovery is handled in civil litigation—for better or for worse, depending who you ask.The Judicial Conference of the United States, the judiciary’s policymaking body, adopted the amendments during its biannual, closed-door meeting on Tuesday. The rules will go to the U.S. Supreme Court for consideration. If the high court approves the changes, they’ll take effect Dec. 1, 2015, unless Congress steps in to oppose the amendments or to make adjustments.
Tuesday, September 16, 2014
Gartner Predicts Rise of the Digital Risk Officer
By Michael Piramoon, Director of Analyst Relations, AccellionThe number of devices connected to enterprise networks is skyrocketing. One reason is mobile computing. Mobile workers in the US now carry on average 3 mobile devices, according to a recent survey by Sophos. Fifteen years ago, each of those workers would have connected to the network through a single desktop computer. The number of devices storing business data and connected to the network per employee has tripled (or quadrupled for those employees who still have desktop computers in addition to their mobile devices). And unlike the devices of a decade or more ago, many of these devices have been selected and configured by employees themselves, regardless of whether or not the organization has officially adopted a Bring Your Own Device (BYOD) policy.
Monday, September 15, 2014
New Survey Shows eDiscovery Workload, Predictive Coding Use Increasing
By Doug AustineDiscovery workload, the use of predictive coding and projected rate of adoption of technically assisted review are all up significantly, according to a new report by recruiting and staffing firm The Cowen Group.In its Executive Summary, the Q2 2014 Quarterly Critical Trends Report highlighted “three compelling and critical trends”, as follows:
- Workload and the rate of increase is up, significantly
- The demand for talent is still increasing, however at a much slower rate than last year.
- The use of predictive coding (PC) is up by 40 percent but the projected rate of adoption of technically assisted review (TAR) and PC is dramatically up by 75 percent.
The survey represents responses from one hundred eDiscovery partners and litigation support managers/directors from 85 Am Law 200 law firms.
Friday, September 12, 2014
Making Technology-Assisted Review Problems Smaller? (Cartoon and Clip)
Keeping up with the promises and problems of Technology-Assisted Review (TAR) can be quite a challenge given the amount of writers and writing on the subject. This week’s cartoon and clip features a way to make the challenges of TAR look smaller (cartoon) and a non-all inclusive listing of recent articles on the topic of TAR (clip).
Click here to find a short list of the latest articles on TAR as shared on the ComplexDiscovery blog.
New Survey Shows eDiscovery Workload, Predictive Coding Use Increasing
By Doug AustineDiscovery workload, the use of predictive coding and projected rate of adoption of technically assisted review are all up significantly, according to a new report by recruiting and staffing firm The Cowen Group.In its Executive Summary, the Q2 2014 Quarterly Critical Trends Report highlighted “three compelling and critical trends”, as follows:
- Workload and the rate of increase is up, significantly
- The demand for talent is still increasing, however at a much slower rate than last year.
- The use of predictive coding (PC) is up by 40 percent but the projected rate of adoption of technically assisted review (TAR) and PC is dramatically up by 75 percent.
The survey represents responses from one hundred eDiscovery partners and litigation support managers/directors from 85 Am Law 200 law firms.
Thursday, September 11, 2014
Convincing Employees to Use New Technology
By Didier BonnetAll of our companies are digital now – or quickly becoming that way. Almost any enterprise you can think of, no matter the industry or sector, is trying (or being pressured by competitors) to use new technology to harness the vast new oceans of data being generated by smartphones, sensors, digital cameras, GPS devices, and myriad other sources of information originating from customers and markets.Yet how many millions of dollars have been spent on analytics technology, but with no parallel improvements – or even any changes – to the way decisions are made within a business? How many companies have deployed internal wikis and social networks with great fanfare only to see slow take-up or a huge slow down after a few months? Even among digital natives, adoption of things like enterprise digital tools often doesn’t live up to lofty expectations.
Wednesday, September 10, 2014
The Difficult but Necessary Challenge Ahead of Building a Legaltech Startup Ecosystem
By Aron SolomonThere is little doubt today that we need an emerging legaltech startup vertical. But where will it come from? Any startup ecosystem needs inputs, energy, and direction in order to grow and thrive. In 2014 and beyond, these won’t simply self-create, but are rather a product of identifying a need and building, with purpose and intentionality, from there. There are several structural and practical facts, perhaps obstacles, that come into play in the legaltech space. The first is that legaltech startups often need to build in stealth because they are founded by practicing lawyers. [...]
Tuesday, September 9, 2014
After the eDiscovery Turkey
By Herbert L. RoitblatSome of the most vicious fights occur when families get together for the Holidays. Maybe there’s something in the turkey that brings it out. Grossman and Cormack have responded to my blog posts (here, here, here, and here) about their articles (and here) with a good deal of vitriol, but without addressing the fundamental questions I raised.They do seem to have gotten at least one thing right, though. A quote on our home page attributed to them the statement that predictive coding is less expensive than human review, and incorrectly cited their JOLT article. They did not write these words there and we have taken down the quotation.By the way, I don’t think that they disagree with the opinionattributed to them, only with the attribution. The title of their article was, after all, “Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review.” Still, our quote was incorrect, and it is now gone.
eRecall: No Free Lunch
By Bill DimmThere has been some debate recently about the value of the “eRecall” method compared to the “Direct Recall” method for estimating the recall achieved with technology-assisted review. This article shows why eRecall requires sampling and reviewing just as many documents as the direct method if you want to achieve the same level of certainty in the result.Here is the equation:eRecall = (TotalRelevant – RelevantDocsMissed) / TotalRelevantRearranging a little:eRecall = 1 – RelevantDocsMissed / TotalRelevant = 1 – FractionMissed * TotalDocumentsCulled / TotalRelevantIt requires estimation (via sampling) of two quantities: the total number of relevant documents, and the number of relevant documents that were culled by the TAR tool.
Monday, September 8, 2014
Cormack and Grossman Talk Turkey, Respond to Roitblat
By Gordon V. Cormack and Maura GrossmanThis guest blog constitutes the first public response by Professor Cormack and Maura Grossman, J.D., Ph.D., to articles published by one vendor, and others, that criticize their work. In the Editor’s opinion the criticisms are replete with misinformation and thus unfair. For background on the Cormack Grossman study in question, Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, SIGIR’14, July 6–11, 2014, and the Editor’s views on this important research see: Latest Grossman and Cormack Study Proves Folly of Using Random Search For Machine Training – Part One and Part Two and Part Three. After remaining silent for some time in the face of constant vendor potshots, Professor Cormack and Dr. Grossman feel that a response is now necessary. They choose to speak at this time in this blog because, in their words:We would have preferred to address criticism of our work in scientifically recognized venues, such as academic conferences and peer-reviewed journals. Others, however, have chosen to spread disinformation and to engage in disparagement through social media, direct mailings, and professional meetings. We have been asked by a number of people for comment and felt it necessary to respond in this medium.
Friday, September 5, 2014
Three Mission-Critical Applications of Big Data in Oil, Gas
By John Ebeling, Roy Lowrance and Walter McGuiganMajor players in the oil and gas industry, particularly oilfield services companies, understand that Big Data analytics can provide valuable insights that will help make exploration, production, manufacturing, and global operations more streamline, safe, and efficient. Leaders in the industry are already implementing Big Data solutions into their everyday operations and reaping the rewards of this long-term investment. As more and more oilfield services companies are committing to investing [...]
Thursday, September 4, 2014
Baseball, Birthing, and Bias: The Human Factor in Predictive Coding
By Herbert L. RoitblatIt is now widely recognized that predictive coding can, in fact, considerably reduce the cost and effort of eDiscovery and increase its accuracy. The pundits have turned lately to discussions of how predictive coding can “best” be implemented. But the technology used is only one part of the equation that determines the ultimate cost and accuracy of predictive coding. The human factor remains important as well. According to Patrick Oot, “Technology providers using similar underlying technology, but different human resources, performed in both the top and bottom tiers of all categories.” [...]
Wednesday, September 3, 2014
Information Management: 5 Big Questions Answered
By Desire Althow
There are many reasons for the dramatic proliferation of data , and this, alongside changing consumer behaviour, is having a a profound effect on the role of the Chief Information Officer. Canon recently held an 'Information at Work' event that looked at how data was impacting the workplace, so we caught up with the company's Director of Information Security, Quentyn Taylor, to find out what messages are coming out of the information segment at present. Here are his responses to our five key questions. TechRadar Pro: What is causing the massive influx of [...]
By Desire Althow
There are many reasons for the dramatic proliferation of data , and this, alongside changing consumer behaviour, is having a a profound effect on the role of the Chief Information Officer. Canon recently held an 'Information at Work' event that looked at how data was impacting the workplace, so we caught up with the company's Director of Information Security, Quentyn Taylor, to find out what messages are coming out of the information segment at present. Here are his responses to our five key questions. TechRadar Pro: What is causing the massive influx of [...]
Tech is Litigants’ Boon, Not Profession’s Doom
By David Horrigan for Predictive Coding: A Special ReportAssisted review is merely one way for attorneys to find their way through mountains of evidence.
In one of the more debated lines in English literature, Dick the Butcher, a character in Shakespeare’s “Henry VI, Part II,” said, “The first thing we do, let’s kill all the lawyers.” Some argue the line shows Shakespeare’s disdain for lawyers, others that, because Dick was engaged in a nefarious plot, the line is actually commentary about how necessary lawyers are. Fast-forward four centuries, and technology brings a new twist to the necessity for lawyers: The development of assisted review technologies, including predictive coding.
Tuesday, September 2, 2014
Daubert, Rule 26(g) and the eDiscovery Turkey: Part 2, Tasting the eDiscovery Turkey
By Herbert L. Roitblat, Ph.D.Part 2 – In which I continue to argue that how you cooked your eDiscovery turkey in the laboratory may not be a good indicator of its taste or wholesomeness when served from your kitchen. This is the second post ( see the first one here ) concerning Maura Grossman and Gordon Cormack’s article in Federal Court Law Review about how to meet one’s Federal Rules of Civil Procedure, Rule 26(g) obligations. Their article is a critique of an earlier one by Schieneman and Gricks, who argued that measuring the outcome of an [...]
Monday, September 1, 2014
Yes, Predictive Coding Works in Non-Western Languages
By John TredennickA recent U.S. Department of Justice memorandum questioned the effectiveness of using technology-assisted review with non-English documents. The fact is that, done properly, such reviews can be just as effective for non-English as it is for English documents. This is true even for the so-called “CJK languages” — Asian languages including Chinese, Japanese and Korean. Although these languages do not use standard English-language delimiters such as spaces and punctuation, they are nonetheless candidates for the successful use of technology-assisted review. John Tredennick is a former trial attorney and founder and chief executive officer of Catalyst Repository Systems. He was [...]
Subscribe to:
Posts (Atom)