Allan Simões AdegasPortuguês
Menu
Back to home

BTG Pactual · Software Engineer (Operations) · 2024 – 2026

An LLM email reviewer that checks before it acts

Operations analysts read and checked counterparty emails with unstandardized financial data by hand. I built an application that extracts that data with an LLM, validates every field against an internal API and only lets clean matches through on their own. Everything else goes to human review with the evidence alongside. Result: more than 220 hours a month given back to the team.

220+ h
saved per month
≈ R$ 14 mil
in efficiency per month
4
countries served (BR, CL, CO, MX)

Context

The Operations team receives confirmations and information from counterparties by email every day. Each bank and broker writes differently: tables, free text, attachments, Portuguese, Spanish and English, different number and date formats.

Before, an analyst opened each message, found the values and checked them one by one in the internal system. It is repetitive work where a typo or a misread has a financial cost.

The challenge

Reading unstandardized data needs flexibility that fixed rules don't give. But an LLM can be confidently wrong, and in financial operations you can't approve something just because the model said it's right.

The challenge was to use the LLM where it is good (reading) without giving it the final decision (approving).

What I built

Extraction into structured output
The LLM receives the email and returns an object with typed fields. No free text to parse later: either the field arrives in the expected format, or it is treated as missing.
Validation against the system of record
Every extracted value is compared with what the internal API has. The LLM is never the source of truth; the API is.
Human review only where it's needed
What matches goes through on its own. Mismatches and missing fields go to an analyst, who gets the email value and the system value side by side.

Architecture

Architecture: An LLM email reviewer that checks before it actsMailbox → Reading. Reading → LLM extraction. LLM extraction → Validation. Validation → Internal API. Validation → Goes through. Validation → Human reviewlookupMailboxcounterparties, 3 languagesReadingbody and attachmentsLLM extractionstructured outputValidationfield by fieldInternal APIsystem of recordGoes througheverything matchesHuman reviewwith the evidence
Simplified view of the flow. BTG internals are omitted.

Decisions and trade-offs

The LLM reads, the system decides
Separating reading from deciding is what made the solution acceptable at a bank. A model error becomes a mismatch (and goes to review), not a wrong approval.
Structure over clever prompts
Investing in a well-defined output schema paid off more than tuning prompt wording: format errors surface immediately instead of leaking into later steps.
Automate the common case, not every case
The goal wasn't to eliminate human work, but to remove the repetitive part. The analyst now looks only at exceptions.

Results

  • More than 220 hours saved per month, about BRL 14k per month.
  • Serving operations in Brazil, Chile, Colombia and Mexico, in three languages.
  • The same approach (LLM to read, system to decide) became the basis for LangChain agents that extract data from unstructured financial documents.
06601,3201,9802,6401234567891011122,640 hoursmonth
Hours returned to the team over 12 months, at 220 h/month.

Figures reported by the business area. Architecture described at a high level for confidentiality.

What I'd do next

  • An evaluation set with known answers to measure every model or prompt change before production. It's what I later built into ReconAgent.
  • Use analysts' decisions as new test cases.

Stack

PythonLangChainOpenAI / Anthropic APIsFastAPIAWS