Litigation Prep Assistant — Pipeline Architecture

Ingestion

Offline corpus pipeline — building the Pinecone index

src/rag/ingestion.py

Source

15 Kenyan Statute Files

data/raw/*.txt
Employment Act · WIBA · Constitution · Civil Procedure · Land Act · Arbitration · …

▶

Step 1

Sentence-Aware Chunker

Split on .?! boundaries
Accumulate → 800-char chunks
100-char overlap between adjacent chunks

▶

Step 2 — Quality Gate

_is_substantive_chunk

5 filters (detail below)
PASS → keep | FAIL → silently drop
~30% of raw chunks are dropped

▶

Step 3

Batch Embed

OpenAI text-embedding-3-small
1,536 dimensions · cosine space
Batched API calls for efficiency

▶

Step 4

Delete → Upsert

Delete all vectors WHERE source = filename
Then upsert fresh vectors
Metadata: {text, source, chunk_index}

▶

Result

Pinecone Index

~1,800 statute chunks
Serverless · cosine similarity
Ready for retrieval queries

▼ Quality Gate Detail — the 5 filters

Why filter at all?

Every Kenyan statute file from Kenya Law Reporting Authority starts with a publisher header, then a 2-page table of contents, then the actual law. Without filtering, these non-law chunks embed as "legal" and contaminate retrieval — the strategy agent reads a table of contents instead of a statute.

Filter 1

Exact boilerplate phrases

Drops chunks containing "KENYA LAW REPORTING", "ARRANGEMENT OF SECTIONS", "Laws of Kenya" header text

Filter 2

Publisher footer substring

Drops "www.kenyalaw.org" footer that appears on every page of the official PDFs

Filter 3

Word count < 25

Section headers like "Section 16 — Liability of employer" are 6 words. No substantive legal content.

Filter 4

TOC line ratio > 45%

If 45%+ of lines match "Section N — Title" pattern, the chunk is a contents page. Drop it.

Filter 5 — Two-condition

Interpretation section

Requires both: statutory header ("In this Act…") AND 3+ "X means Y" definition lines.

Why two conditions? Single condition produces false positives — many substantive provisions use "means" as a regular word. Both together reliably isolates actual definition-only sections.

Why sentence boundaries?

Splitting on fixed character counts truncates statutory sentences mid-phrase: "No action shall lie by an empl—" produces a broken embedding. The chunker splits on .?! boundaries first, then accumulates complete sentences up to 800 chars. The 100-char overlap means a sentence at the boundary of two chunks appears in both — retrieval never misses a provision because of where the chunk boundary fell. | Why not larger chunks? Larger = more diluted embeddings — mixed topics in one vector score lower on any specific query.

Delete-before-upsert

Without deleting first, every re-run of ingestion creates duplicates in Pinecone. Run 3 times → 3 copies of each chunk → retrieval returns the same provision 3× → dedup works much harder, index bloats indefinitely. Delete all vectors for that source file first, then upsert fresh. The operation is idempotent. | First-ever run: Pinecone returns 404 for delete on a new namespace — benign, upsert creates the namespace automatically.

statute files

~800

chars per chunk

~30%

chunks filtered out

~1,800

vectors in Pinecone

1,536

dimensions

Retrieval

Live 7-stage retrieval pipeline — runs on every case

retriever.py · query_expansion.py · reranker.py

Input

Case Text

~500-word narrative or PDF extraction
raw, unstructured

▼

Stage 1

Query Expansion

gpt-4o-mini + instructor JSON mode
→ 7 queries · statutes · issues · facts

▼

Stage 2

Parallel Pinecone Searches

7 query searches
+ supplemental per-statute
+ guarantee queries
→ ~35 raw candidates

▼

Stage 3

Score Threshold

≥ 0.60 unfiltered
≥ 0.45 source-scoped

▼

Stage 4

Fuzzy Deduplication

4 passes: exact → containment
→ 90% word → Jaccard ≥ 0.85

▼

Stage 5

LLM Judge

gpt-4o-mini scores 0–10
≥ 5 pass · < 5 dropped
→ 6–10 relevant chunks

▼

Stage 6

Contextual Compression

Extract relevant sentences only
Parallel · semaphore 6
Empty → fall back to full chunk

▼

Output

6–8 Statute Excerpts

list[str] of tight, relevant statute text
→ passed to Strategy agent

Problem 500 words of case narrative → one diluted embedding. This single vector is pulled in multiple directions (termination + wages + limitation + res judicata) and ends up mediocre for every specific legal issue. Naive top-5 retrieval misses entire angles of the case.

Solution LLM extracts: legal_issues, applicable_statutes, key_facts, and generates 7 focused queries — each targeting a distinct legal question.

Detail Uses instructor for guaranteed JSON output. If LLM call fails entirely → heuristic fallback scores sentences by legal-term density, picks top 8. You always get something to search with.

Alt considered Could use asyncio.gather instead of create_task for simplicity. Chose create_task so extraction streams to browser immediately without waiting for Pinecone cold-start latency.

3 search types Focused queries (7): each query embeds and searches Pinecone, optionally filtered to a statute file when the query mentions a specific Act (e.g., "wiba" → only search work_injury_benefits_act_cap236.txt).

3 search types Supplemental: one additional search per statute identified by expansion, using the full case summary as query — catches provisions no individual query surfaces.

3 search types Guarantee queries: hard-coded queries for provisions with known semantic gaps (WIBA s.16). Run whenever that statute is applicable — see callout below.

Why statute filter? Without scoping, "employer liability" pulls Penal Code criminal provisions into a civil compensation case. _STATUTE_MAP maps 20+ keyword phrases to filenames. Only statute-specific terms are mapped — generic terms like "negligence" are not, because they span multiple Acts legitimately.

Two thresholds Unfiltered searches: ≥ 0.60. Source-filtered: ≥ 0.45.

Why lower for filtered? When you restrict to a single statute file, the vector universe shrinks dramatically. Cosine similarity is relative — the best-matching chunk in a small corpus scores lower than in the full index. Applying 0.60 to filtered searches drops genuinely relevant chunks. Threshold was set empirically by testing WIBA retrieval.

Problem 7 queries + supplemental + guarantee searches can return the same provision multiple times with minor formatting or encoding differences. Judge evaluates duplicate text, strategy agent reads the same law twice — wasted context window.

4 passes Exact (set): O(n), catches identical chunks. Containment: chunk A is substring of B → keep B. 90% word: 90%+ of shorter chunk's unique words in longer chunk → same provision, different section header. Jaccard ≥ 0.85: near-identical of similar length.

Why 0.85 / 90%? Legal text reuses boilerplate words (employer, court, person, Act) across completely different provisions. Lower thresholds collapse distinct sections that share vocabulary. 0.85 means genuinely near-identical, not just from the same area of law.

Problem High-scoring chunks from the right statute can still be legally irrelevant — interpretation/definition sections, citation clauses, wrong-branch provisions, surface keyword matches.

Solution gpt-4o-mini scores each chunk 0–10 against a detailed rubric. The key insight: explicit 0–4 negative examples in the prompt. Without them, the judge approved bad chunks.

0–4 examples added from test failures "Criminal provision in a civil case, even if both involve cheques" · "Arbitration Act in a res judicata case" · "Interpretation section listing definitions" · "Citation clause ('as held in…')" · "Keyword match only, wrong legal context"

Problem An 800-char chunk may contain one relevant provision and the start of an unrelated one. Strategy agent reads noise alongside the relevant text.

Solution LLM extracts only the sentences relevant to this specific case. All chunks compressed in parallel (semaphore of 6 to avoid rate-limit hammering).

Critical fallback If compressor returns empty string → fall back to the full original chunk. Without this fallback, a bug caused constitutional articles approved by the judge to be silently dropped (compressed to ""), reducing retrieval from 6 sources to 1. The judge is the gatekeeper for relevance; the compressor only trims.

What strategy agent receives 6–8 statute excerpts, each trimmed to sentences directly relevant to this case, from authoritative Kenyan legislation. Strategy agent reads only this text when generating legal arguments — it cannot cite statute it was not shown.

500

words in, 1 embedding

→

focused queries

→

~35

raw Pinecone candidates

→

threshold + dedup

~12–15 unique

→

judge

6–10 relevant

→

6–8

tight excerpts to strategy

⚖️

The Hardest Problem: WIBA s.16 and the Semantic Gap

WIBA s.16 says: "No action shall lie by an employee against an employer for damages for personal injury." This section bars all common-law tort claims against employers — missing it means advising a client to sue when the claim will be struck out.

The problem: A lawyer's case uses vocabulary like "employee was injured · employer was negligent · seeking compensation." WIBA s.16 uses exclusion language: "no action shall lie." These phrases embed in completely different regions of the vector space. Even with WIBA scoping and 7 expanded queries, s.16 consistently scored 0.38 — below the 0.45 threshold.

Diagnosis path: symptom (s.16 missing from output) → checked ingestion (s.16 was indexed, confirmed by direct Pinecone fetch) → checked scores (s.16 scored 0.38) → root cause: embedding space mismatch between case vocabulary and statutory exclusion vocabulary.

Fix — Guarantee queries: _STATUTE_GUARANTEE_QUERIES = {"work_injury_benefits_act_cap236.txt": ["no action employee employer tort damages personal injury Kenya"]} Whenever WIBA is applicable, this query fires using vocabulary close to the actual statutory text. The judge then decides whether s.16 is relevant to this specific case. The guarantee query ensures s.16 is always a candidate — it does not force inclusion.

Why not just lower the threshold to 0.30? You'd include every vaguely employment-related chunk. The guarantee query solves a specific semantic gap without degrading quality across all searches.

Reference

Every design decision — problem, naive failure, and fix

Decision	What the naive approach gets wrong	Why this solution is better
Sentence-aware chunking	Fixed-size splits cut statutory sentences mid-phrase → broken embeddings, incomplete reasoning	No chunk ever breaks a sentence. 100-char overlap prevents missing provisions at boundaries.
Five-filter quality gate	TOC pages and publisher headers embed as "legal" — strategy agent reads a contents page instead of statute text	Only substantive provisions enter the index. Two-condition interpretation filter avoids false positives.
Delete-before-upsert	Re-running ingestion without deleting creates duplicate vectors that grow unboundedly	Idempotent — run 50 times, always end up with exactly one copy per chunk.
Query expansion (7 queries)	One embedding of 500 words is mediocre for every specific legal issue in a multi-issue case	7 focused queries each target a distinct legal question → up to 35 sharp candidates vs. 5 diluted ones.
Statute metadata filtering	Generic searches pull wrong-branch provisions (Penal Code in a civil case) scoring well on shared vocabulary	Only statute-specific keyword phrases are mapped — generic terms like "negligence" span Acts legitimately and are not filtered.
Dual score thresholds (0.60 / 0.45)	Applying 0.60 to source-filtered searches drops genuinely relevant chunks (smaller corpus = naturally lower scores)	0.45 for filtered searches was set empirically by testing WIBA retrieval — prevents false negatives on targeted lookups.
Guarantee queries	Exclusion-language provisions (WIBA s.16 "no action shall lie") embed far from compensation queries — missed by all 7 expanded queries	Hard-coded queries using statutory vocabulary ensure these provisions are always candidates for the judge, without lowering global thresholds.
Fuzzy dedup at 0.85 / 90%	Lower thresholds collapse distinct provisions that share legal boilerplate ("employer", "court", "person")	0.85 / 90% means genuinely near-identical text, not just provisions from the same area of law.
Explicit judge negatives	Without explicit 0–4 examples, judge approved definition sections, wrong-branch provisions, and surface keyword matches	Explicit negative categories (each from a real test-case failure) teach the judge what "wrong" looks like, not just what "right" looks like.
Compression fallback = full chunk	Empty string from compressor was treated as "no relevant content → drop chunk" — lost judge-approved provisions silently	If compressor returns "", fall back to full chunk. Judge is gatekeeper for relevance; compressor only trims.

Litigation Prep Assistant — Ingestion & Retrieval Pipeline Architecture

Offline corpus pipeline — building the Pinecone index

Live 7-stage retrieval pipeline — runs on every case

The Hardest Problem: WIBA s.16 and the Semantic Gap

Every design decision — problem, naive failure, and fix