Skip to content

Add eval scenario: T2 Mock Overuse hiding real integration failures #6

@hyhmrright

Description

@hyhmrright

Scenario

Mode: Test Quality Review (Mode 4)
Risk: T2 Mock Overuse

Input

A test suite for a payment service where every external call (database, payment gateway, email sender) is mocked, including the database layer. Tests pass at 100% but the last three production incidents were caused by ORM query bugs that only appear against real data.

def test_charge_customer(mocker):
    mocker.patch('payments.db.get_customer', return_value=fake_customer)
    mocker.patch('payments.gateway.charge', return_value={'status': 'ok'})
    mocker.patch('payments.email.send_receipt')
    
    result = charge_customer(customer_id=1, amount=100)
    assert result.success
    # The actual SQL query that caused last month's outage is never executed

Expected finding

Brooks-lint should flag this as 🔴 T2 Mock Overuse:

  • Symptom: database layer mocked out, ORM queries never tested against real schema
  • Consequence: tests pass while production queries silently break
  • Remedy: use a test database (SQLite in-memory or Docker) for repository layer tests

How to contribute

  1. Add this to evals/evals.json as the next sequential entry
  2. See CONTRIBUTING.md → "Add an eval test case" for the exact JSON format
  3. Open a PR

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions