A green test suite that tested nothing

The problem

The product is a SaaS platform for residential construction in the US: an admin web app, a mobile app for field workers and a supplier portal, across 15 repositories. The CI board was mostly green, yet bugs kept reaching the client.

Jira, early SeptemberBug backlog when I took on the QA/QC Lead role

• Open more than 30 days
• Open less than 30 days
• No longer open

• 244 bugs156 open

  926488

Context and role

• I am the BA and main point of contact for the US client, supporting QA; from 09/2026 I am also QA/QC Lead.
• I could not open pull requests on the product repositories, so every finding had to carry enough evidence for Dev to fix it themselves.
• The work ran on an AI-agent-assisted process that I designed; I checked every conclusion against real run logs.
• Client and product names and vulnerability details are withheld.

Step 1: Auditing the test suites

I re-ran every test in 6 repositories and read each kind of test, instead of trusting the green on CI.

Test suite auditBefore and after the root-cause fix (07/2026)

• Before
• After

• Fake UI tests that always pass without checking anything✓ 146 → 0

  146
  0

• Backend tests passing✓ 35 passing, 195 failing → 276 passing, 0 failing

  35 passing, 195 failing
  276 passing, 0 failing

• Field worker mobile app coverage✓ 12.6% → 21%

  12.6%
  21%

• Supplier portal unit tests✓ 0 → 133

  0
  133

Cause195 failing tests, one cause behind 183

• Broken during setup by one jsonb type-mapping error
• Other error groups

• 195 failing tests195

  18312

Step 2: Auditing CI

A green CI does not mean the tests ran. I compared each repository’s pipeline configuration with its real run history (09/2026).

CI auditThree places where green meant nothing

1. 1New code
2. 2Pipelinemany test suites never called
   configured to ignore errors, still green
3. 3Test stepnightly job: 6/6 sub-steps failed
   no repository had branch protection
4. 4Merge into main branch

Step 3: Testing each ticket with Tester Kit

The ticket, its comments and its PR are split into a list of sources; every test case must point to a source, then passes a coverage gate and an independent review by a separate AI session before it runs.

Tester Kit

1. 1Ticket, comments, PRsplit into sources
2. 2Sourced test cases
3. 3Coverage gate
4. 4Independent reviewa separate AI session
5. 5API and UI run
6. 6Tester signs offone Excel file

Step 4: Testing a whole permissions epic

Company-level permissions (multi-tenant) were split across many tickets. Every ticket passed on its own.

Why test at epic level

1. 1Each ticketpassed when tested alone
2. 2Where tickets meetNew screenExcel exportShared API
3. 3Data leaks to another company

Epic resultsCompany-level permissions

• Passed
• Real defects
• Remaining

• 146 cases146

  128108

Results

• The backend suite runs again: from 35 passing and 195 failing to 276 passing and 0 failing; 146 fake tests removed; smoke tests became a merge gate.
• A specific list of idle tests and 4 CI fixes to switch them back on, backed by run history.
• 901 automated test cases in two weeks; 9 false passes blocked before anyone relied on them.
• 10 cross-company data-isolation defects found before release.

What I learned

• A green CI means something only once you have proved the test step really runs and really blocks.
• Hundreds of failing tests usually share a few causes; finding the root is faster than fixing the symptoms.
• Permission defects live where features meet, so they have to be tested at the epic level, not only ticket by ticket.