Model card and evaluation
Every number on this page is produced by ml/train.py, ml/berka_validate.py and the test suite, and is read from JSON at page load — nothing is typed in by hand.
Risk model — synthetic holdout
loading…
Methodology check on real bank data (Berka, PKDD'99)
loading…
Feature importance (gain)
Statement parsing and tamper detection (through the real app)
run tests/statement_eval.py to produce this file
Receipt OCR (tesseract.js, in-browser)
run tests/ocr_eval.py to produce this file
Unseen statement layouts (out-of-distribution test)
run npm test to produce this file
Transaction categoriser
run npm test to produce this file
⚠ Measured on synthetic descriptions drawn from the generator's own vocabulary. Real statements will score lower until the lexicon is grown on bank data during the pilot — this is the single most important thing to tune in week one.
Known failure cases
- Handwritten ledgers and cash-only income: nothing to extract; the memo shows the gap and the officer attests.
- Statement PDFs without a text layer (scans): not yet OCR'd page by page in the prototype — upload XLSX/CSV export instead.
- Unseen layouts where debit and credit share one column and there is no balance column: signs are assumed from keywords.
- Mixed Latin/Cyrillic merchant strings and transliteration variants outside the lexicon → default category.
- Low-light or heavily skewed receipt photos: OCR confidence drops below 60%; fields are flagged for confirmation rather than guessed.
- The risk score inherits the synthetic population's structure; its ranking is advisory until retrained inside the bank.
Speed and cost per application
Measured in the browser on the sample cases (see the timings line under the audit log): statement parsing 1.2–1.8 s for a 12-month PDF of 500–1,800 lines, OCR about 1.6 s per receipt photo (first run adds ~3 s to load ~8 MB of language data, cached afterwards), categorisation + features + rules + risk + integrity + memo under 50 ms. Running cost per application: zero marginal cost — static hosting, no API calls, no GPU. The production design adds a local LLM only for free-text summaries, at well under one US cent per application on a single on-prem GPU.