AI in Estate Planning: Proofreading

Not great, not terrible

Table of Contents

1. Where We Left Off

2. Proofreading and Prompts Examples

   1. ChatGPT Example 1

   2. Claude AI Example 1

   3. ChatGPT Example 2

   4. Claude AI Example 2

   5. Example - Paid Version of ChatGPT

3. Key Takeaways

Where We Left Off

To preface, I considered not publishing this article because it is somewhat hypocritical on my part.  For better or worse, I run this budding newsletter and education enterprise in a highly bootstrapped manner with a staff of one – me.  (Speaking of, if you would like to contribute articles or video/audio content, please reach out.)  Going it alone leads to errors in articles that I do not often catch.  What I should be doing is seeking editorial review – either from others, or from AI.  And, the article I used as an example for this article is one that I did not run through AI review before publishing.  Hence, the examples below exposed a lot of errors that have been sitting out in the open for the 2+ weeks since the original article was published.  But, I decided to use myself as an negative example and plow ahead for your benefit. 

The prior AI article discussed how AI cannot be used as a substitute for actually reading the law.  This may have come across as a bit self-righteous, but this was not my intent.  This series is partially about helping you stay out of trouble when using AI, and we now have several data points where attorneys have been sanctioned in situations that could have been avoided had they simply read the law (instead of citing hallucinated authority sight unseen). 

The conclusion was that AI is not as good at coming up with source material.  But, it is (sometimes) good at reading source material that you spoon-feed to it.  That being said, it is still not perfect.  I have run into multiple situations in the past two weeks in which ChatGPT in particular misread a document and did not give me a correct answer.  In one situation, it gave an incorrect section reference while claiming there was a missing period when there was not actually one.  In another situation, after prompting and re-prompting at least 4 times, it just simply would not list and summarize the percentages expressly stated in a certain provision of a document. 

I bring this up to mention that you still have to check the work of AI review services, and as we will find out below, the quality of the actual source files matters as well.  While AI services can do a decent job, we cannot always use AI output without checking for accuracy.  The state of many services can be highlighted using a meme from the HBO series, Chernobyl… “not great, not terrible.”  (And, I’m sure the prompt engineering aficionados will come out in force to scream that I was doing something wrong, but their protestations will fall on deaf ears.)

In the last article, we used examples from the paid version of Claude AI, and I bring this up to mention (as will be illustrated below) that sometimes the paid version of services like ChatGPT does a better job than the free version.  But, I am not here to endorse one off-the-shelf AI service from another.  I mentioned that Claude has a limitation on the number of different source documents you can upload for comparison purposes (5 documents).  Recently, however, Google launched NotebookLM which is designed specifically to help you create a “notebook” from multiple sources. 

Next time, we will discuss NotebookLM.  In the meantime, you can test it for free.  I would especially encourage you to look at the “Audio Overview” feature, which creates somewhat of a podcast-style interview discussion of source material based on the parameters you provide.  While you cannot change the voices, and while the questions and responses can be a bit cheesy, it is still shocking to listen to how good a job this feature does.  I have included an example below based on a recent article of mine, Does Your Trustee Know They Are A Trustee?

(If you like this format, please let me know and I will start to include more of them.)

Proofreading and Prompts Examples

Today, we are going to use a source document which I have included below – a raw copy of a prior article I wrote – but with some errors conveniently added to the document to see if our AI review service will pick them up.