Resilient document pipeline for complex case files
- 8xcheaper on input
- 4xcheaper on output
- 1,18Mcharacters per load
- 189PDFs in the largest case file
The problem
A document pipeline accumulated in memory the raw text of case files holding between 3 and 189 PDFs each, then passed it as an oversized payload to downstream services.
Fragile, expensive and capped in scale the moment a case file grew.
The architecture
A decoupled redesign validated over 33 case files: text consolidated per case, a persistent intermediate Markdown artefact, extraction split from summarisation, structured metadata captured at the source, and migration to a more efficient model with context up to 400K tokens.
60 tests cover the flow.
Impact
- Inference cost 8x lower on input and 4x lower on output after the model migration
- Loads of up to 1.18 million characters, with context up to 400K tokens
- Validated over 33 case files holding between 3 and 189 PDFs each
- From a fragile in-memory flow to persistent artefacts: each stage fails and retries on its own
Stack
- Python
- Azure
- Blob Storage
- Document Processing
- Architecture Refactor