Test Strategy

The master test and verification strategy for the Sindh IT Portal — Facilitation Desk (SITP): the testing pyramid, tools, coverage targets, CI/CD gates, test-data strategy, representative test cases (TC-<EPIC>-<nnn>-<nn>), Gherkin-as-spec, localization/RTL, accessibility, security, performance/load, UAT, regression, defect management, and bidirectional traceability from Functional Requirement → User Story → Acceptance Criterion → Test Case.

1. Purpose & How to Read This Document

This document is the authoritative verification contract for SITP. It tells engineering, QA, MAAHIR operations, and S&ITD acceptance stakeholders what is tested, at which layer, with which tools, to which targets, and how every Test Case ties back to a User Story, its Acceptance Criteria, and the originating Functional Requirement. It is written to be RFP- and tender-ready, and to satisfy the PPP accountability bar that every shipped capability is demonstrably verified.

1.1 Reading order for a single verification

Functional Requirement (FR-...) → User Story (US-...) → Acceptance Criterion (Gherkin Scenario) → Test Case (TC-<EPIC>-<nnn>-<nn>) → Test Layer (unit / integration / e2e / manual) → CI Gate / NFR target.

Every TC listed in this document exists to make one or more Gherkin scenarios in /specs/en/01-prd/ executable and repeatable. IDs are stable, language-neutral, and never translated.

1.2 Quality bar (one-line statement)

The Portal is shippable when (a) every [M] Acceptance Criterion in the PRD has at least one passing automated Test Case, (b) coverage targets in §5 are met, (c) every CI/CD gate in §6 is green, and (d) the UAT exit criteria in §13 are signed off by the pilot departments.

1.3 Scope

2. Testing Principles

1. Shift left. Defects are cheapest to fix at the unit layer. Every PR carries its own tests; nothing reaches main unverified (NFR-MAINT-003).
2. Test behaviour, not implementation. Tests assert observable outcomes from the Gherkin scenarios; they do not assert private method signatures or DB column names. This keeps the modular-monolith refactorable (NFR-MAINT-001).
3. Gherkin is the contract. A Scenario: in the PRD is a single source of truth: it documents intent for stakeholders and drives automation via Cucumber/Playwright step bindings (see §8). There is no parallel, divergent "test spec".
4. Defense in depth. Security, privacy, accessibility, and i18n are not single-gate checks; each is verified by an automated layer (CI) plus a manual/expert layer (per-release audit). See §10–§12.
5. Determinism first. Flaky tests are treated as P1 defects; a test that cannot be made deterministic is quarantined and rewritten within one sprint (NFR-RELY-004 mindset applied to QA).
6. Production realism. Performance, localization, and graceful-degradation tests run against a staging environment that mirrors production topology (NFR-PORT-001, NFR-SCAL-001) at realistic cardinality (§7).

3. Testing Pyramid

SITP follows the classic testing pyramid: a broad, fast unit base; a narrower integration band; a small, slow end-to-end (e2e) peak; and a thin manual / UAT layer reserved for exploratory testing, accessibility audit, security pen-test, and stakeholder acceptance. The shape is deliberate — the vast majority of assertions live at the cheapest, fastest layer.

Written description. The pyramid reads bottom-up: the broadest layer (Unit) holds the most assertions and runs in milliseconds on every pull request. Each successive layer holds fewer assertions, costs more to run, and runs less frequently. The base verifies that each module's pure logic — SLA calculators, the proof-of-resolution gate, the escalation-ladder scheduler, PII-redaction mappers, RBAC resolution, locale-catalog completeness — behaves correctly in isolation. The integration band verifies that modules cooperate through their published interfaces and against real MariaDB/Redis/MinIO via Testcontainers. The e2e peak exercises the few journeys that are only meaningful when the whole stack (portal + API + workers + AI service + adapters) is live — file a ticket, track by ID, resolve with proof, escalate, publish a MoM, verify a QR-signed letter. The manual layer is reserved for what cannot be automated: exploratory testing, the WCAG screen-reader audit, the independent pen-test, and the pilot departments' UAT sign-off.

3.1 Layer-by-layer contract

4. Tools & Stack

Tools are chosen to match the technology stack locked in _context.md §3 and the maintainability gates in /specs/en/03-non-functional-reqs/ §3.9. No tool is introduced that cannot run in CI on every pull request.

5. Coverage Targets

Coverage targets align with NFR-MAINT-002 and add a stricter bar for the highest-risk decision gates. Coverage is a floor, not a ceiling; it is necessary but not sufficient (a high score with shallow assertions is a known anti-pattern, countered by the mutation-testing option in §4).

A PR that lowers overall coverage below the floor fails the build. The coverage gate runs on the delta (changed lines) in addition to the whole-project total, so legacy code is not penalised but new code is held to the bar.

5.1 The 100 % decision gates

These are the narrow, high-blast-radius code paths where a missed branch is a compliance or trust failure. They must reach 100 % line and branch coverage with explicitly named test cases, regardless of the module-wide floor:

6. CI/CD Gates & Pipeline

Every gate below is a hard block on the path to main and to a production deploy, enforced by branch protection (NFR-MAINT-003). A deploy that skips a gate is an incident.

Written description. A pull request enters the pipeline at the left and must pass every gate left-to-right before it can merge. Lint and typecheck are the cheapest catches and run first. Unit tests with the coverage gate follow, then integration tests that spin up real MariaDB/Redis/MinIO via Testcontainers. Contract tests (OpenAPI diff against the published spec, Pact for webhooks) catch breaking changes before they reach partners. Build produces the containers and bundles. SCA + secrets scans dependencies and source for known vulnerabilities and leaked credentials (NFR-SEC-006, NFR-SEC-011). Accessibility (axe-core, Pa11y, Lighthouse) and i18n/RTL (locale parity, visual diff) run next. An e2e smoke suite covers the top critical journeys. License + SBOM generation closes the per-PR block. After human review and merge, the nightly full suite runs the slower, deeper checks — ZAP DAST, k6 load, full e2e across all three locales, and (optionally) mutation testing. A release gate adds the human-in-the-loop checks: the independent pen-test, the manual accessibility audit, and pilot-department UAT sign-off. Only then does a zero-downtime deploy proceed (NFR-AVAIL-006).

6.1 Gate catalogue

7. Test Data Strategy

Test data is synthetic by default; real PII never leaves production, and the few corpora that mimic PII are masked and access-controlled. Data strategy is governed by NFR-PRIV-001 (minimization) and NFR-PRIV-003 (PII vaulting).

7.1 Data sources

7.2 Entity fixture coverage

Each entity type has at least one canonical fixture and one edge-case fixture, so that conditional logic (entity-type-specific registration, category-specific ticket fields, role-template overrides) is exercised:

• Company — one per entity type: SECP company, sole proprietor/partnership, freelancer, foreign branch, early startup (provisional) (per US-ORG-001).
• Representative — one per role template: Primary, Admin, Filer, Viewer, Notify; plus a granular-override case (per US-ORG-004).
• Department — a nested fixture: Department → Section → Staff, with DG + Secretary oversight assigned (per US-ORG-005).
• Ticket — happy-path, conditional-form (per category), draft, sub-task parent/child, merged primary/secondary, confidential/VIP, RTI-category (statutory deadline).
• MoM — scanned-image (forces OCR), text-PDF, Word; with and without action items; sensitive/VIP variant.
• Letter / certificate — trilingual, signed/unsigned, tampered-hash variant for QR verification.
• Notification — per channel (email, SMS, WhatsApp, in-app) and per event (created, assigned, escalated, resolved, closed, MoM-published).

7.3 Multilingual corpus (3 locales)

Every text-bearing fixture exists in EN, UR, and SD so that OCR, search, AI translation, notification rendering, and RTL layout are all exercised against realistic script content:

• Latin (EN) — baseline.
• Urdu (UR) — Nastaliq, RTL.
• Sindhi (SD) — Naskh Arabic script, RTL.
• A mixed-script fixture (EN body + UR attachment) for translation and PII-redaction edge cases.

7.4 Data hygiene rules

• No real CNIC, real phone number, or real email appears in any fixture or committed seed. Faker locale set to en_PK for plausible shapes.
• The masked PII corpus lives in an access-controlled store; CI pulls it via short-lived token only for the redaction/pen-test jobs.
• Test databases are ephemeral (Testcontainers) and torn down per spec run; no cross-spec leakage.
• Production-like cardinality for performance tests is generated by a seeder that respects NFR-SCAL-004/005 (up to 100 k tickets, 100 k reps, 10 k staff).

8. Gherkin Scenarios Reused as Automated Specs

A Scenario: block in /specs/en/01-prd/ is not documentation that is then re-translated into a separate test script. It is the executable specification. The Gherkin text is bound to automation through two mechanisms:

1. Cucumber (Behave for PyTest / @cucumber/cucumber for TS) — the literal Gherkin Given/When/Then steps are bound to step-definition functions. The scenario text and the test are one artefact; changing the scenario changes the test.
2. Playwright + data-testid binding — for e2e, the same Gherkin scenario is annotated with stable data-testid selectors so the assertion is portable across UI refactors.

Worked example — the proof-of-resolution gate scenario (US-TKT-006) becomes TC-TKT-006-01:

# Reused verbatim from /specs/en/01-prd/ (US-TKT-006, Scenario: Resolution requires evidence and a note)
Scenario: Resolution requires evidence and a note
  Given a staff member attempts to move a ticket to "Resolved"
  When no evidence attachment and resolution note are present
  Then the transition is blocked
  And the system states which required items are missing

Bound automation (illustrative):

// tests/integration/ticket/proof-gate.steps.ts
Given('a staff member attempts to move a ticket to {string}', async (status) => {
  this.ticket = await ticketFactory.create({ status: 'InProgress' });
  this.attempt = () => ticketsApi.transition(this.ticket.id, status);
});

When('no evidence attachment and resolution note are present', async () => {
  // factory created the ticket without attachments and without a note — nothing to do
});

Then('the transition is blocked', async () => {
  await expect(this.attempt()).rejects.toMatchObject({ status: 409, code: 'PROOF_GATE_MISSING' });
});

Then('the system states which required items are missing', async () => {
  const err = await this.attempt().catch(e => e);
  expect(err.details.missing).toEqual(expect.arrayContaining(['evidence_attachment', 'resolution_note']));
});

Authoring rule: if a Test Case cannot be traced to a Gherkin scenario, either the scenario is missing (add it to the PRD) or the Test Case is testing implementation detail (refactor or delete it). This rule keeps the spec and the suite from drifting apart.

9. Sample Test Cases (TC-<EPIC>-<nnn>-<nn>)

The table below lists representative Test Cases that exercise the locked behavioural decisions in _context.md §5 and the [M]/[S] user stories in the PRD. It is not the exhaustive suite — the full catalogue (one or more TCs per Acceptance Criterion across E1–E17) lives in the QA test-management tool. Each TC here maps to its originating User Story, the Functional Requirement(s) it satisfies, the primary test layer, and the priority that drives regression inclusion (see §14).

Priority key. P0 — launch-blocking; runs in every CI pipeline and in regression. P1 — V1 must-have behaviour but lower blast radius; runs in CI and in pre-release regression.

Suite completeness note. The 22 cases above are the representative spine. The full test-management catalogue derives at least one TC-<EPIC>-<nnn>-<nn> per Acceptance Criterion across E1–E17 (per the PRD's per-epic AC counts), giving an estimated 200+ automated cases at V1 launch, grown incrementally per phase. Coverage of [M] ACs is the launch-readiness exit criterion (§1.2).

10. Localization & RTL Testing

Localization is a first-class test concern, not a final-pass check, because SITP serves three statutory-provincial languages with two of them right-to-left. This section operationalizes NFR-I18N-001…006.

10.1 Automated (every PR)

10.2 Visual (per release)

• Playwright screenshot diff for the top 20 pages, captured in all three locales and in both directions (LTR for EN, RTL for UR/SD). Pixel drift beyond tolerance requires reviewer approval.
• Mirror checks for RTL: navigation, iconography, breadcrumb order, table column order, modal close-button placement, and chart axis direction all mirror correctly. Logical CSS properties (padding-inline-start, etc.) are used exclusively — inline direction hacks are linted out (NFR-I18N-002).
• Font rendering verified for Noto Nastaliq Urdu and Noto Naskh Arabic at the brand type sizes.

10.3 Manual (per release)

• Linguistic review by an approved Urdu reviewer and an approved Sindhi reviewer for the changed screens, logged against the glossary.
• Search-quality spot check in UR and SD (Meilisearch script handling) against the multilingual corpus.

11. Accessibility Testing

Accessibility is statutory-adjacent for a government service and is locked at WCAG 2.1 Level AA across all three languages (_context.md §2). This section operationalizes NFR-A11Y-001…004.

11.1 Automated (every PR)

• axe-core (via @axe-core/playwright) on the top 20 pages — zero critical violations is a hard gate.
• Pa11y additionally scans the Docusaurus docs site.
• Lighthouse CI accessibility category ≥ 90 on the top 10 pages; trend tracked per build.
• Colour-contrast unit test on the brand-token palette (Ajrak indigo/maroon) at all text sizes — 4.5:1 normal, 3:1 large (NFR-A11Y-004).

11.2 Manual (per release)

• Keyboard-only walkthrough of every critical journey: visible focus indicator, logical tab order, no keyboard traps, skip-to-content link present (NFR-A11Y-003).
• Screen-reader pass with NVDA + Firefox, JAWS + Edge, VoiceOver + Safari/iOS, TalkBack + Android on the critical journeys; ARIA semantics verified on dynamic regions (ticket threads, dashboards, chat) (NFR-A11Y-002).
• Defects triaged by severity; WCAG-failures block release.

11.3 Phase-2 (deferred, tracked)

Pakistan Sign Language sign-video guides (NFR-A11Y-005) and the dyslexia-friendly + voice-navigation mode (NFR-A11Y-006) are Phase-2 and have their own release gate; they are not V1 blockers.

12. Security Testing

Security testing verifies NFR-SEC-001…013 and the control catalogue in /specs/en/11-security-compliance/. It combines continuous automation with independent expert review.

12.1 Continuous (CI/nightly)

12.2 Authorization matrix tests (automated)

A parameterized test matrix exercises every role × every sensitive action, asserting allow/deny matches the permission matrix in /specs/en/04-roles-permissions/. Roles: company Primary/Admin/Filer/Viewer/Notify; government Section Staff/Officer/Dept Admin/DG/Secretary/Super Admin; public visitor. Actions: read ticket, write comment, transition status, close VIP, export PII, change Primary Rep, toggle feature flag, read audit log, etc. Any mismatch blocks the build.

12.3 ABAC confidential/VIP tests (automated, 100 % gate)

The confidential/VIP access check (§5.1) is exercised with a dedicated case matrix: every staff role with and without the matching attribute, in list view, search results, direct URL access, and API call. The expected outcome is deny + log for the non-matching case; visibility must be zero (no leakage via counts, search suggestions, or audit reads by the denied party).

12.4 Independent pen-test (per-release gate)

An independent external penetration test (NFR-SEC-007) is required before public go-live, annually thereafter, and after any security-sensitive change. Scope:

• The portal, the API, the WebSocket gateway, the AI service, and the docs site.
• Authentication, session, and step-up flows (Keycloak).
• File upload/download (ClamAV bypass attempts, path traversal).
• PII redaction boundary (attempt to force a cloud AI call on a Restricted class).
• Webhook signing and replay.
• IDOR / horizontal privilege escalation across companies and departments.
• Rate-limiting and brute-force resistance on login/OTP/tracking-ID lookup.

Critical findings remediated within 30 days, high within 60; closure evidence reviewed at the phase gate. The pen-test report is the primary evidence for ASVS L2 conformance (NFR-SEC-001).

13. Performance & Load Testing

Performance targets are quantified in NFR-PERF-001…011; this section states how they are verified. All performance tests run against the staging environment that mirrors production topology (NFR-PORT-001) at realistic cardinality (§7.4).

13.1 Target summary (from doc 03)

13.2 k6 scenarios

• Smoke — per release; 50 virtual users, 5 min; top read/write paths; sanity.
• Load — per release; ramp to the V1 concurrency target (PERF-008) and hold 30 min; assert p95 latency targets and ≤ 1 % 5xx.
• Soak — per release + on-demand; 110 % of target for 60 min; detect memory leaks, connection-pool exhaustion, queue backlog growth (PERF-008 measurement method).
• Spike — on-demand; 2× target in < 30 s; verify circuit breakers and graceful degradation hold (NFR-AVAIL-003).
• Multilingual search — quarterly; the 100-query EN/UR/SD suite against Meilisearch (PERF-010).

13.3 AI/OCR timeout and graceful degradation

• AI summary and OCR jobs are asserted to respect their p95 ceilings (PERF-004/005) and to time out cleanly — a stalled engine must not block the request path (the queue-leveled architecture, NFR-SCAL-002).
• Fault-injection suite (per release): each external dependency (AI provider, OCR engine, NADRA/SECP/FBR/SRB/PSEB/e-Office, Mailjet/SMS/WhatsApp) is killed in turn; the critical user journeys (file, track, comment, resolve, close) must still pass, with the dependent capability visibly marked "temporarily unavailable" (NFR-AVAIL-003, NFR-RELY-001).

14. Regression Strategy

Regression protects the behaviours that already work while new code ships. SITP regression is risk-weighted, not "run everything always".

Tagging. Every test is tagged with: module code (e.g., TKT, ORG), priority (P0/P1/P2), locale (EN/UR/SD), direction (LTR/RTL), and the originating US-/FR- IDs. This enables fast, targeted regression on hotfix branches.

Bisection. A regression that lands in main is bisectable because the nightly suite gives a per-commit green/red signal; the offending commit is reverted within the same business day for P0 regressions.

15. UAT Plan

User Acceptance Testing is where the pilot departments and a cohort of IT companies confirm the system works for real work, not just for the test suite. UAT is the human gate before each phase goes live (per /specs/en/14-roadmap-release/).

15.1 Pilot structure

15.2 UAT entry criteria

• Pre-release regression (§14) green.
• Pilot data seeded; pilot users trained (per Epic E15 training).
• Known defects at or below the threshold in §16 (no open P0; ≤ 5 open P1).
• UAT scripts (per pilot) reviewed by the pilot lead.

15.3 UAT exit criteria (sign-off)

• 100 % of in-scope [M] Acceptance Criteria demonstrated passing to the pilot.
• System Usability Scale (SUS) ≥ 70 per pilot cohort, per language (NFR-USA-006).
• CSAT on pilot tickets collected and reviewed.
• No open P0 defects; open P1 defects have an accepted remediation plan with a date.
• Signed UAT report from the pilot department Secretary and S&ITD.

15.4 Feedback loop

Pilot findings flow into the defect tracker (§16) with a uat- label; patterns (e.g., recurring confusion in a flow) feed back into /specs/en/10-ux-sitemap-flows/ and the KB/SOPs (Epic E10).

16. Defect Management

Workflow. Every defect is logged with: reproducible steps, expected vs actual, environment, locale, the failing TC- (or "no TC" → triggers a new TC), severity, and the trace_id from the staging/production log. Defects are triaged daily during UAT and weekly otherwise. A defect that required a code change must add or update a Test Case that would have caught it — closing the loop and growing the regression suite (NFR-RELY-004 mindset).

Flakiness policy. A flaky test is a defect. It is quarantined within one occurrence and either fixed (deterministic) or rewritten within one sprint; it does not linger as a known-unstable gate.

17. Traceability

Traceability is bidirectional and is the audit evidence that every shipped capability is verified and every test exists for a documented reason. The chain is:

FR (FR-<MOD>-<nnn>) → US (US-<EPIC>-<nnn>) → AC (Gherkin Scenario) → TC (TC-<EPIC>-<nnn>-<nn>) → Test layer + NFR target.

17.1 Representative traceability extract

17.2 Coverage of acceptance criteria

Launch-readiness requires that every [M] Acceptance Criterion in /specs/en/01-prd/ has at least one passing automated Test Case (§1.2). The QA test-management tool produces a per-release traceability matrix report showing, for each Epic, the ratio of ACs-with-passing-TCs to total ACs; a value below 100 % for [M] ACs blocks the release gate.

17.3 Cross-document references

• Functional requirements: /specs/en/02-functional-reqs/.
• Non-functional targets verified by performance/security/a11y suites: /specs/en/03-non-functional-reqs/.
• Ticket workflow and state transitions under test: /specs/en/06-ticket-workflow/.
• Security control catalogue verified by §12: /specs/en/11-security-compliance/.
• Adapter mocks for integration tests: /specs/en/08-integrations-spec/.
• API/Webhook contracts for contract tests: /specs/en/12-api-contract/.

18. RACI Summary

19. Phase Alignment

This strategy is applied incrementally per the roadmap in /specs/en/14-roadmap-release/:

20. Open Items [TBD/confirm]

These items are tracked to closure in /specs/en/14-roadmap-release/ and reflected back into this document as the decisions land.

End of document.