Working with Long Documents Using AI: Summarize, Ask and Compare

Last updated: 29 شهریور 1405

A forty-page contract, a plugin's documentation, an annual report, tender terms. AI can cut the time you spend reading these a great deal, but not by uploading and typing "summarize."

That approach usually gives a generic summary, half of which you could have guessed from the document's title. Worse, sometimes a sentence in the summary isn't in the document. Below is a method whose output you can check.

How do we prepare the document before uploading?#

Three things: make sure the document's text is readable, separate important tables, and remove sensitive data.

• Text or scan? In a PDF, try selecting and copying a sentence. If you can't, the document is an image. Many assistants can read images too, but they make mistakes on low-quality scans. For an important document, first convert it to text with an OCR tool.
• Tables: a table inside a PDF often gets scrambled when text is extracted. If the numbers in a table matter to you, give that table separately as a spreadsheet or CSV.
• Sensitive data: replace people's names, ID numbers, account numbers and confidential amounts with made-up values before uploading. Details are in What we should never give to AI.
• Size: every tool has a limit on the amount of text, called the context window: the most text the model can consider at one time. If the document is larger than this limit, part of it is never seen. A full explanation is in Tokens, context window and hallucination.

How do we get a useful summary?#

Say who the summary is for and what decision it supports, and ask for the section reference next to each point.

"Summarize" is like telling a colleague "read this." They don't know what you're looking for. Compare it with this:

This is a website support contract. I am the client and I have to decide
whether to sign.

Give the summary in this format:
1. The contractor's obligations (one line each)
2. Our obligations
3. Termination conditions and penalties
4. Clauses that are against our interest or unclear

Write the clause number next to each item. If a topic doesn't appear in the
contract, write "Not in the contract" and don't guess.

The last sentence matters. Without it, the model fills a gap with what is "usually in contracts."

How do we ask questions and get answers we can verify?#

Ask the model to quote the document's exact sentence and give its location; then search for that sentence in the document yourself.

Answer only based on this document. For every claim, quote the exact
sentence from the document in quotation marks and give the page or clause number.
If the answer isn't in the document, write only "Not found".

Question: How many days after delivery is each installment due?

Quoting makes checking a matter of seconds: copy the sentence and search the PDF. If it isn't found, set the answer aside.

Not all questions are equally trustworthy:

The claim "it isn't in the document" is the weakest answer, because the model may not have seen that section. Studies have also shown that in very long text, models pay better attention to the beginning and end than to the middle.

How do we split a very long document?#

Break the document into natural sections, summarize each separately, and take the final conclusion from the summaries.

1. Divide the document by its own chapters or headings, not by page count. A clause cut in the middle is poorly understood.
2. Write one fixed request for all sections so the summaries have the same shape.
3. Ask each summary to keep the clause numbers.
4. Put the summaries together and ask for a conclusion.
5. Take every important claim in the conclusion back to the original text by its clause number and check it.

This is the same big-task-splitting method described in more detail in multi-step prompting. If you ask questions of the same fixed set of documents every day, for example company policies, move to RAG instead of uploading again.

How do we compare two documents or two versions?#

Use a text comparison tool to find the exact change, and AI to understand the effect of the change.

A model can miss the change of a single number or a small "not." Text comparison tools, like Word's document compare or a diff tool, don't skip any change. So work in two steps: first get the exact list of changes with the comparison tool, then give that list to the model:

These are the changes in the new version of the contract compared with the
previous version. For each change, write in a table: clause number,
old text, new text, and its effect on us in one sentence. Mark purely
editorial changes separately.

To compare two different documents, such as two price quotes, set the criteria yourself first and ask it to fill a table with those columns. When the model picks the criteria, it compares things that don't matter to you.

Where are these methods heading?#

The direction is clear: tools' context windows are getting bigger, reading tables and images is getting better, and tools cite exact sources more often.

But two things don't change: the model may misread, and the responsibility for the decision is yours. To prepare, keep your own documents tidy from now on: selectable text instead of scans, clear headings and clause numbers, and tables in separate files. A document that is readable for people is more readable for AI too. Keep the habit of checking quotes as well; however good the tool gets, that habit is the cheapest insurance.

Summary#

• Before uploading: selectable text, separate tables, sensitive data removed.
• Say who the summary is for and what decision it supports.
• Ask for a quote and a clause number for every claim, and search for it yourself.
• Don't trust "it isn't in the document" or totals of numbers.
• Summarize a large document section by section, and use a text comparison tool for exact changes.