BTG Pactual · Software Engineer (Operations) · 2024 – 2026
An LLM email reviewer that checks before it acts
Operations analysts read and checked counterparty emails with unstandardized financial data by hand. I built an application that extracts that data with an LLM, validates every field against an internal API and only lets clean matches through on their own. Everything else goes to human review with the evidence alongside. Result: more than 220 hours a month given back to the team.
- 220+ h
- saved per month
- ≈ R$ 14 mil
- in efficiency per month
- 4
- countries served (BR, CL, CO, MX)
Context
The Operations team receives confirmations and information from counterparties by email every day. Each bank and broker writes differently: tables, free text, attachments, Portuguese, Spanish and English, different number and date formats.
Before, an analyst opened each message, found the values and checked them one by one in the internal system. It is repetitive work where a typo or a misread has a financial cost.
The challenge
Reading unstandardized data needs flexibility that fixed rules don't give. But an LLM can be confidently wrong, and in financial operations you can't approve something just because the model said it's right.
The challenge was to use the LLM where it is good (reading) without giving it the final decision (approving).
What I built
- Extraction into structured output
- The LLM receives the email and returns an object with typed fields. No free text to parse later: either the field arrives in the expected format, or it is treated as missing.
- Validation against the system of record
- Every extracted value is compared with what the internal API has. The LLM is never the source of truth; the API is.
- Human review only where it's needed
- What matches goes through on its own. Mismatches and missing fields go to an analyst, who gets the email value and the system value side by side.
Architecture
Decisions and trade-offs
- The LLM reads, the system decides
- Separating reading from deciding is what made the solution acceptable at a bank. A model error becomes a mismatch (and goes to review), not a wrong approval.
- Structure over clever prompts
- Investing in a well-defined output schema paid off more than tuning prompt wording: format errors surface immediately instead of leaking into later steps.
- Automate the common case, not every case
- The goal wasn't to eliminate human work, but to remove the repetitive part. The analyst now looks only at exceptions.
Results
- More than 220 hours saved per month, about BRL 14k per month.
- Serving operations in Brazil, Chile, Colombia and Mexico, in three languages.
- The same approach (LLM to read, system to decide) became the basis for LangChain agents that extract data from unstructured financial documents.
Figures reported by the business area. Architecture described at a high level for confidentiality.
What I'd do next
- An evaluation set with known answers to measure every model or prompt change before production. It's what I later built into ReconAgent.
- Use analysts' decisions as new test cases.
Stack
PythonLangChainOpenAI / Anthropic APIsFastAPIAWS