Data-Matcher
In developmentAutomatic comparison of two spreadsheets or data files to find, in seconds, what doesn't match.
- 6
- formats read
- 5
- kinds of difference reported
- API v1
- public, with access keys
Overview
Reconciling two datasets (the bank statement and the system, the order and the invoice) is still done with spreadsheet lookups at many companies. Data-Matcher takes one file per side, lets you build the match key visually and returns the differences.
What it does
- Any tabular file
- Excel, CSV, Parquet, JSON, ODS and Arrow, with format, header, encoding and delimiter detection.
- Key built on screen
- One or more column pairs, with concatenation, fixed text and transformations (prefix, leading zeros, digits only, number, date), previewed before running.
- Actionable results
- Only on side A, only on side B, divergent values, repeated keys and blank keys, exportable as CSV, XLSX or Parquet.
Architecture
Decisions and trade-offs
- Polars for the engine
- Rules compile into Polars expressions, which keeps matching fast on large files without Python loops.
- A queue in the database
- Runs use a queue in PostgreSQL: one fewer dependency to operate, enough for the product's volume.
Quality and operations
- Unit tests without Docker and integration tests with PostgreSQL.
- Every format reader, the engine, rule preview, queue, export, sign-in and public API implemented and tested.
Status and next steps
- The production image was prepared for Pier and validated locally. Publication, the team screen, rate limiting and a run-finished webhook are pending.
Stack
PythonPolarsFastAPIPostgreSQLCloud StorageReactTanStack