Generative AI E-Discovery Tools and the Importance of Prompt Engineering

What is Prompt Engineering?

The explosion of LLMs (large language models) and other Generative AI tools designed to increase workplace efficiency and productivity has created a new lexicon of jargon and definitions. The term “Prompt Engineering” may be the most widely used new term to describe the method of using LLMs and Generative AI tools.

Prompt engineering is the creation of text-based instructions or cues that a person uses to direct LLMs or other generative AI tools.

These instructions are then interpreted by the AI tool. The AI tools use these instructions as a guide to search for and find responsive information and formulate a response or create an end product. The second portion (formulating a response or creating an end product) is currently the most underutilized aspect of using these tools. For instance, if you asked a first-year associate or paralegal to research a specific legal issue, you would likely include how you wanted them to present their findings, whether in a memo, a brief, a comparison chart, etc. These same instructions can be given to the AI tool.

The below tips and tricks for AI prompt creation and iteration are tailored to using AI tools, specifically Relativity’s aiR for Review, to assist with litigation, document review, and deposition or trial preparation. While Relativity performs the prompt engineering for this tool on the back end, the prompts you create and iterate to input to aiR for Review significantly influence the results you can generate with this tool. 

Before Creating and Iterating Prompts

As we all know, the number of documents reviewed and produced during the discovery stage of litigation has steadily increased over the last decade. As a result, many tools have been developed to assist and streamline this process. Generative AI technology is the most recent tool implemented by law firms and vendors. I consider these Generative AI tools the next step in technology-assisted review. However, just like Active Learning before it, these tools do not entirely replace prior analytical tools or procedures. These earlier tools and methods can be used with Generative AI to further enhance results.

Before creating prompts for AI tools, you still need to employ some of these other analytical tools and procedures. There will always be some portion of collected documents that there will be no reason to review (e.g., the document is a duplicate, not the most inclusive email thread, or outside of a relevant date range). Therefore, you should still use email threading, de-duplication tools, search terms, and date range restrictors to limit the documents requiring review.

The reasoning for limiting the universe of documents for the review remains the same: The review will be more efficient and cost-effective. You always have the option to run any excluded documents through the AI tool after the prompts have been created if you so choose.

Types of Prompts

I have evaluated and used Relativity’s aiR for Review for several months. The prompts described below are specific to this tool.

1. Case Overview Prompts

• These prompts contain background information about the case, names of important people or companies, and lists of important words, phrases, jargon, or acronyms.
• These prompts should be as clear and concise as possible. Limit the amount of “fluff” and try, as much as possible, to stick to the basic facts of the case.
• You will need to decide whether providing a list of people or companies is more important. Because aiR for Review has a character limitation on the prompts, including a list of people and companies may not be necessary.
  • For example, in one of the cases I created prompts for, the various companies were much more important in the dispute than the individual employees of the companies.
  • Therefore, I only provided a list of the companies in the prompts. I also included the email domains for the companies whose names were not easily identifiable from the email domain.
• Providing a list of relevant words, terms, and phrases is critical. Like a human reviewer, Generative AI will not be familiar with company-specific jargon or terms.
  • For instance, an AI tool cannot know that the client referred to the project in dispute as Project Hulk.
  • Therefore, you must include company-specific jargon, terms, acronyms, abbreviations, etc. in the prompts.
  • You need to ensure that your custodial interview form or questionnaire includes requests for this type of information.

2. Relevance or Key Document Prompts

• These prompts are used to determine whether a document is considered relevant.
• The person creating these prompts must be familiar with the case, and facts about the case, and be able to identify and articulate relevant topics or issues.
• These prompts should outline what information makes a document relevant and what information may make a document not relevant.
  • For instance, consider a pharmaceutical drug litigation. The relevance criteria prompts should instruct that a document is relevant if it concerns Drug A and that documents about Drugs B, C, and E are irrelevant.
  • The prompts could further identify that a document regarding ingredients I, J, and K are relevant because these ingredients are used in Drug A and that documents relating to ingredients L, M, and N are irrelevant.

Tips for Creating and Iterating Prompts

1. Remember the Character Limitation

• Be Concise – Your ultimate goal is to have the computer identify as much relevant information as possible. Anything in your prompts that does not further that goal should be eliminated. Do you need to identify 20 people? Maybe you only need to identify 5 of the most critical people involved in the dispute.
• Consider Punctuation – Ask yourself questions like these:
  • Are hard returns necessary?
  • Does a bulleted statement need a period at the end?
  • Can these sentences be shortened to eliminate commas, colons, or semi-colons?
• Recommendation – If possible, include all the information you believe is necessary in each section of the prompt. Most likely, you will be over the character limitation. Review each section and decide whether the information is redundant, and can be combined, shortened, or removed.

2. Use of ALL CAPS, Bold, or Underlines

• These methods can be straightforward and delineate critical information from information of lesser importance.
• Using these methods may help save on the number of characters used.
• Be sure to assess whether the AI tool interprets the prompt correctly.

3. Put Most Important Information at the Beginning of the Prompt

• If you write relevance criteria prompts in paragraph form, include the most critical information at the beginning of the prompt section rather than in the middle or at the end.
• My recommendation is that relevance criteria prompts be written in a bulleted format. Bulleted formats are more concise and will use fewer punctuation characters.
• However, even when using a bulleted format, the most critical bullets should be at the beginning of the prompt.

4. Include non-responsive or non-relevant information in the prompt.

• Similar to Active Learning and other technology-assisted review tools, these Generative AI tools provide better results when non-relevant information is also identified.
• Prompts should include information that would make a document irrelevant. This will help the tool provide more accurate information.
• The only drawback to including non-relevant information is the character limitation. Because of a character limitation, you must decide how many of these non-relevant prompts to include.
  • In my experience, I have found that while prompts on non-relevant topics are important, the majority of the prompts should instruct the tool on what is relevant rather than on what is not.

5. Provide instructions on how a generated response should look or what should be included.

• You can instruct the tool to include specific information in its response output.
  • For instance, if the opposing party’s 5th production request concerns financial documents, the prompts could instruct the AI tool to include “RFP 5 – Financial” in its response output if the document is responsive and includes financial information.
  • Or, if a particular topic is critical to the case, the prompts can instruct the response to include “Highly Critical,” or the prompt could direct the tool to identify documents on the topic as very relevant.
• The only real drawback to including these instructions is the character limitations, so be thoughtful about which prompts should include these additional instructions.

Conclusion

Learning to harness the power of e-discovery AI Tools like Relativity’s aiR for Review through prompts specifically constructed to produce the output you desire in your case can increase your efficiency. Implementing the above prompt creation and iteration tips and tricks will help you use Relativity’s aiR for Review tool to more effectively to assist with litigation, document review, and deposition or trial preparation.

See Starling's guest post on this topic on the Relativity Blog.

DISCLAIMER: The information contained in this blog is not intended as legal advice or as an opinion on specific facts. For more information about these issues, please contact the author(s) of this blog or your existing LitSmart contact. The invitation to contact the author is not to be construed as a solicitation for legal work. Any new attorney/client relationship will be confirmed in writing.