Skip to content

Add 8 more invoice eval scenarios (issue #5) - #10

Open
hnet158158 wants to merge 2 commits into
mshojaei77:mainfrom
hnet158158:gfi/5-eval-cases
Open

Add 8 more invoice eval scenarios (issue #5)#10
hnet158158 wants to merge 2 commits into
mshojaei77:mainfrom
hnet158158:gfi/5-eval-cases

Conversation

@hnet158158

@hnet158158 hnet158158 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What this PR does

Adds 8 new evaluation scenarios to samples/eval_manifest.jsonl (13 total now), using previously unused sample PDFs from the samples/ directory.

Closes #5

New scenarios

case_id Category Invoice file Pages PO
duplicate_priparcel duplicate invoice_010_priparcel_duplicate.pdf 3 PO_004
duplicate_wordpress duplicate invoice_012_wordpress_plugin_duplicate.pdf 1
swiss_qr_invoice clean / international invoice_005_swiss_qr.pdf 1 PO_003
fedex_complex_invoice parser-challenge invoice_008_fedex_wht_en_th.pdf 2
aws_vat_invoice_multipage parser-challenge invoice_007_aws_europe_vat.pdf 8
credit_note_instead_of_invoice exception credit_note_001_demo.pdf 1
return_form_non_invoice exception return_form_001_emergency_luminaires.pdf 1
utility_bill_non_invoice exception utility_bill_001_english.pdf 4

Coverage

  • duplicate — 2 cases: PriParcel & WordPress plugin duplicate pairs
  • clean — Swiss QR invoice (Swiss-specific format) matched with PO
  • parser-challenge — FedEx (2-page, withholding tax) & AWS (8-page multi-region VAT)
  • international — Swiss QR (CHF-based, Swiss address format)
  • exception — credit note, return form (garbled binary), utility bill (not invoices)

Verification

  • All 13 JSONL lines are valid JSON
  • Every invoice_path, po_path, and delivery_note_path points to an existing file
  • pytest tests/test_eval_smoke.py — 4/4 passed
  • No new files added — all scenarios reuse existing samples

…, international, exception)

Signed-off-by: hnet158158 <hnet158158@gmail.com>
…s, credit_note, return_form, utility_bill)

- duplicate_priparcel: invoice_010 (copy of 009) + PO_004
- duplicate_wordpress: invoice_012 (copy of 011), standalone
- swiss_qr_invoice: Swiss QR invoice_005 + PO_003
- fedex_complex_invoice: FedEx 2-page withholding tax invoice
- aws_vat_invoice_multipage: AWS 8-page European VAT invoice
- credit_note_instead_of_invoice: credit note (not an invoice)
- return_form_non_invoice: return delivery form (garbled PDF)
- utility_bill_non_invoice: utility bill (not an invoice)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

good first issue: add more sample invoice eval cases

1 participant