How to Keep Your eDiscovery Review from Going Off the Rails

17
Dec
2019

Rachel Teisch

In any eDiscovery or investigation review, the challenge is to find the responsive (and often only the responsive) documents to meet production deadlines and your legal obligations. Review teams must also quickly adapt to unforeseen changes in review criteria, be vigilant in protecting privileged and other sensitive information from disclosure, and be as efficient as possible for your client.

If any of these areas break down, costs can be high. Consequences might include extra documents reviewed, the need for quality control of thousands of miscoded documents, time delays, weeks of Subject Matter Expert (SME) time wasted, and frustration all around.  

Review mismanagement can be avoided. Quality control, or QC, is at the heart of an efficient and accurate document review and essential to keeping the process on track. Below we discuss some best practices for QC. 

Why is QC critical for large reviews? 

QC ensures a review is consistent, accurate and defensible process. According to Federal Rules of Civil Procedure 26(g), the “golden rule” of production, states that “every… discovery request, response or objection must be signed by at least one attorney of record in the attorney's own name…. By signing, an attorney… certifies that to the best of the person's knowledge, information, and belief formed after a reasonable inquiry… it is consistent with these rules and warranted by existing law… not interposed for any improper purpose… and neither unreasonable nor unduly burdensome or expensive.” 

If an issue is identified, the attorney of record can face serious issues. QC helps protect against these risks. Additionally, QC measures can help ensure consistency to avoid unnecessary duplication of the review and aid in meeting deadlines. 

Inconsistency in coding among reviewers can lead to errors that, at best, may require the re-review of documents. At worst, mistakes can become irreparable if not caught early on. Inconsistent coding may be caused by SME error – a simple mistake or relevance “drift.” But more likely, however, mistakes happen by reviewers who miscoded responsive documents as not responsive. This represents a poor QC process, poor results and a potentially compromised outcome. 

But luckily, with the right QC processes in place, inconsistent coding errors can be caught quickly and early. 

The role of outside counsel and managed document review teams

Outside counsel, often working in conjunction with a managed review team, plays a critical role before and during the review. 

Before the review begins, outside counsel should prepare a playbook of clear review guidelines and procedures for determining the type of the review, establishing privilege management procedures, translating review criteria into an efficient coding panel, and testing the coding panel with a small sample of documents. 

During the review process, there is a degree of trial and error required to ensure an optimal QC process. Outside counsel. While the end goal is maximum efficiency, the trial and error process takes time to best acknowledge the potential misconceptions from reviewers, and the need to review criteria or make changes in the coding panel. At the end of the day, outside counsel should be responsible for providing a tested and complete QC process. 

Best-practices for conducting QC (and validation) 

QC is not a “one and done” process. Thoughtful planning, trial and error, and timely feedback course correction are all critical, but an effective QC process will also apply critical best practices below:

1. First, implement random QC questions on documents.
2. Communicate early and often! Ask questions daily or twice daily and capture all Q&A on a Decisions Log that is shared with review team and all stakeholders. Timely feedback and communication with reviewers is critical. 
3. Employ individual sample QC at the outset – This is extremely important and ensures that no reviewer can “upset” the Technology-assisted Review (TAR) 2.0 tool because of a misunderstanding of the coding protocol.
4. Track errors and error types to track reviewer performance -- Ensures that you can identify specific error types from a reviewer to correct the “root cause” of any protocol misunderstandings.
5. Use QC in conjunction with targeted QC for substantive and privilege issues
6. Use Conflicts QC to confirm QC team aligned with SME.

Review QC also affects validation QC – the tail end of the process. To defend review decisions, results need to be validated, or confirmed, that a process has achieved its intended purpose. 

Automate QC with technology-assisted review 

Lawyers should not overlook the role of machine learning tools like TAR based on continuous active learning (CAL) in an effective review and QC process. TAR operates much like Pandora, only you’re with documents, not songs. As humans train the system by reviewing and coding documents, it gets smarter about which documents are relevant and prioritizes those for review. 

Beyond the obvious cost and time savings for law firms and their clients, there are measurable benefits as it relates to QC. With TAR, the algorithm can present documents that are most likely tagged incorrectly. This gives senior reviewers or SMEs the opportunity to make corrections quickly and on-the-fly.  

Privilege and privilege QC is often where mistakes are made – even the most experienced reviewers can become fatigued or make errors in judgement. Privilege assessment takes two forms: an initial privilege review to locate privileged documents among a group of unreviewed documents for production, and a quality control measure to ensure that documents coded as privileged are indeed not privileged. In an initial privilege review, particularly in situations when documents are being produced without an eyes-on review (like second requests and subpoenas), CAL is effective in effective in locating and withholding all of the privileged documents. Beyond this, the QC algorithms incorporated into a CAL review provide one final defense to privilege disclosure, particularly in a traditional production review.  

Sleep better at night 

Ultimately, lawyers need to understand their obligations surrounding a review and responsibilities associated with the FRCP 26(g) certification. Understanding where and how to avoid common pitfalls in a review – and working with an experienced managed review team that is familiar with the best practices above – can go a long way in giving you and your client comfort that you’ve complied with your obligations.