Technical Architecture

The end-to-end engineering architecture of the Sindh IT Portal — Facilitation Desk (SITP): components, layers, data, AI, security, deployment, and operations.

1. Architecture Overview & Principles

The SITP is engineered as an API-first, modular monolith with a pluggable AI/OCR layer, multilingual by construction (EN/UR/Sindhi), role-scoped at every layer, feature-flagged end-to-end, and observability-first. The reference tenets below govern every downstream design decision in this document.

Architectural principles

The architecture favors a small number of well-operated components over a sprawl of microservices: one app DB, one cache, one search, one object store, one identity provider, one queue, one BI tool. This matches the team size, the single-host starting point on Server4Sale, and the provincial government's appetite for operational simplicity. The seams (interfaces, queues, modules) are designed so that scale-out, multi-tenant, or microservice extraction is a refactor, not a rewrite.

2. High-Level Architecture Diagram

The portal is delivered as two public front-ends (a Next.js full portal and a Docusaurus docs site) that sit behind an nginx reverse proxy. The proxy terminates TLS, serves the static docs build directly, and forwards /api to the NestJS API gateway, the long-lived WebSocket gateway, the Python FastAPI AI service, the embedded Metabase, and (in the admin plane) Keycloak. The API gateway is the single entry point to all stateful backends: MariaDB 10.11 (system of record), Redis (cache + sessions + queues), Meilisearch (multilingual search), MinIO (S3-compatible object storage, fronted by ClamAV), and BullMQ workers that offload OCR/AI/email/SMS/WhatsApp. External integrations — NADRA, SECP, FBR, SRB, PSEB, NITB e-Office, Mailjet, SMS gateways, WhatsApp Business API, Zoom/Meet/Teams — are reached only through adapter modules in the integrations layer.

The diagram groups components into planes — Clients, Edge, Application, Identity, Stateful backends, Analytics, and External integrations — to make the trust and dependency boundaries explicit. The NestJS API gateway is the only component that talks to every stateful backend and to most external integrations; workers talk to storage, search, and outbound notification channels; the AI service talks only to object storage and Redis (for its own caching/job state) and is invoked by the gateway and by workers, never directly by clients. The WebSocket gateway shares Redis with the API for pub-sub fan-out so live updates produced by the API or by workers are pushed to the correct connected sessions.

3. Component Catalog

4. Application Layers (NestJS)

The NestJS deployment is organized as a clean-architecture modular monolith: each module owns its domain models, DTOs, services, controllers, and persistence, and exposes only typed interfaces to other modules. Cross-cutting concerns (validation, auth, logging, tracing, feature flags) live in shared infrastructure that no domain module bypasses.

Module map

Cross-cutting mechanics

• DTOs & validation. Every request body is a class-validated DTO (class-validator + class-transformer); Zod-equivalent runtime schemas for hot paths. No raw any payloads cross the controller boundary.
• Guards. AuthGuard (OIDC/JWT), RolesGuard (RBAC), PermissionsGuard (granular per-permission overrides), FeatureFlagGuard (kill a route if the flag is off), StepUpGuard (re-challenge for sensitive actions), ThrottlerGuard (rate limit).
• Interceptors. LoggingInterceptor, TracingInterceptor (OTel), AuditInterceptor (append-only), LocaleInterceptor (propagate user locale into the request context).
• Pipes. Global validation pipe with whitelist + forbidNonWhitelisted; a TransformPipe that normalizes dates to UTC and locale-aware strings.
• Queue offloading. Anything estimated > 250 ms or that calls an external service is enqueued to BullMQ rather than run inline: OCR, LLM/translation/transcription, email/SMS/WA fan-out, PDF/Excel export, scheduled digests, escalation timer ticks. Each job class has its own queue and worker pool with independent concurrency and backpressure.
• Transactions. Domain services that touch multiple aggregates use explicit MariaDB transactions; long-running work is moved out of the DB transaction (enqueue within the transaction, do the work after commit) to keep row-lock dwell time short.

5. Database

5.1 Choice: MariaDB 10.11 (not PostgreSQL)

The system of record is MariaDB 10.11.14, already installed and running on the production host. This is a deliberate, locked decision (see _context.md §3). PostgreSQL is not used anywhere in the stack. The rationale:

• The host is provisioned and the DBA/ops tooling is built around MariaDB.
• MariaDB 10.11 brings uuid(), sys schema improvements, JSON (as LONGTEXT with validation functions), RETURNING on DML, SEPARATOR window-function refinements, and or replace for sequences — all of which cover this product's needs.
• The product has no PostgreSQL-only dependency (no pgvector — vector search and semantic search are delegated to Meilisearch and the AI service; no heavy GIS in PostGIS — district heatmap uses bounding-box math or a lightweight spatial index; no advanced window functions beyond what MariaDB supports).

5.2 ORM recommendation: Prisma

Both Prisma and TypeORM are listed as acceptable in the locked stack. Prisma is recommended for the SITP because:

1. First-class MariaDB/MySQL driver. Prisma's connector targets the MySQL/MariaDB wire protocol natively; no ORM-internal SQL translation surprises.
2. Type safety end-to-end from a single schema.prisma source of truth, which accelerates a TypeScript-primary team.
3. Migrations are explicit, reviewable, and CI-gated — important for a government system where schema change is auditable.
4. Better developer ergonomics for the modular monolith (one generator, one client, schema divided by // <-- module --> comments and prismaSchemaFolder previews), which lowers onboarding friction across MAAHIR engineers.
5. Raw escape hatch (prisma.$queryRaw) for the few cases that need MariaDB-specific SQL (e.g., RETURNING, materialized-view refresh).

Caveat: Prisma does not yet auto-detect MariaDB server version quirks for some JSON operations; for JSON columns we model them as String (validated JSON) and use MariaDB's JSON_* functions via $queryRaw when needed. The volume of such cases is low.

5.3 Schema strategy

• One logical schema per bounded concern, all in the same MariaDB database. Module tables are prefixed (e.g., tkt_, org_, fil_, not_, ai_, int_, aud_) so module ownership is visible without splitting databases.
• Append-only audit table (aud_event) for every state-changing action; never updated or deleted, partitioned by month.
• Soft deletes via deleted_at for user-facing records; hard deletes only for GDPR/RTI-driven purges, run as a scheduled job.
• Status as enum-like lookup tables, not free-text columns, for ticket/MoM/registration states.
• Money stored as integer minor units (PKR paisa) with explicit scale, never floating point.
• Timestamps stored as UTC (DATETIME(6)); locale formatting is an application concern.
• Identifiers: meaningful ticket IDs SITP-YYYY-<DEPT>-<NNNNNN> are a display column backed by a BIGINT/UUID surrogate primary key (so renumbering, merge, and sharding stay possible).

5.4 Character set

• Database, tables, and text columns: utf8mb4 with collation utf8mb4_unicode_520_ci (UCA-based, correct ordering for Urdu and Sindhi). Do not use legacy utf8 (3-byte) — it cannot store all Arabic-script characters needed for Sindhi.
• Connection charset: utf8mb4 enforced in the Prisma datasource URL and on the MariaDB user default.
• Server setting: character_set_server=utf8mb4, collation_server=utf8mb4_unicode_520_ci.

5.5 Indexing

• Primary keys on every table (surrogate BIGINT or UUID).
• Composite indexes on the hot access paths: (org_id, status, updated_at) for ticket queues; (assignee_id, status) for "my work"; (dept_id, sla_due_at) for escalation scans.
• Index the foreign keys MariaDB does not auto-index (e.g., on the child of a 1:N).
• Covering indexes for the most frequent dashboard group-by queries; review quarterly via EXPLAIN ANALYZE.
• Full-text search is not the primary path on MariaDB — Meilisearch owns multilingual full-text. MariaDB FULLTEXT indexes are used only as a fallback for exact-phrase Latin queries.

5.6 Read replicas (future)

Today SITP runs on a single MariaDB instance. The architecture keeps the option open to add read replicas for: Metabase analytics reads, reporting/exports, and read-heavy public-site queries. The Prisma client will be configured with a read/write split at that point; all writes (and reads inside write transactions) go to the primary, analytics/dashboard reads go to replicas. Replication lag is bounded by a configured threshold; laggy replicas are skipped automatically.

5.7 Backups

• Logical: daily mariadb-dump (per-database, --single-transaction --routines --triggers --events) shipped to MinIO and to Server4Sale off-host storage.
• Physical: MariaDB binary backup (mariabackup) on a schedule to enable fast point-in-time recovery.
• Binary log: enabled and retained long enough for point-in-time recovery (PITR) to meet the RPO in §22.
• Restore drills: a quarterly restore into an isolated environment, with a checksum report attached to the runbook.

6. AI / OCR Architecture

6.1 Engine abstraction (pluggable)

The FastAPI AI service exposes three stable interfaces. Every consumer (NestJS module or BullMQ worker) programs to the interface, never to a vendor SDK.

# Illustrative interface contract (per engine family)
class AIClient(Protocol):
    async def complete(self, req: AIRequest) -> AIResponse: ...
    async def embed(self, req: EmbedRequest) -> EmbedResponse: ...

class OCRClient(Protocol):
    async def extract(self, req: OCRRequest) -> OCRResult: ...

class TranscriptionClient(Protocol):
    async def transcribe(self, req: AudioRequest) -> Transcript: ...

Each interface has multiple concrete adapters. The service selects an adapter per feature and per data-sensitivity class at call time, using (a) the feature-flag service, (b) the data classification of the input, and (c) an environment-level allow-list (so a "sovereign" environment can forbid cloud engines entirely).

6.2 AI request flow

Each AI call follows the same shape: resolve policy → resolve the engine for this feature+sensitivity → redact PII → call → on failure, fall back → log + audit → respond. The written description follows the diagram.

The flow ensures four invariants: (1) no engine is ever called that the policy disallows for the input's sensitivity class; (2) PII is redacted before the payload leaves the trusted boundary if a cloud engine is selected; (3) any single engine failure is recoverable via a fallback so the user-facing feature stays available; (4) every call — including its engine, latency, cost, and the count of redacted PII tokens — is captured in an immutable audit row for transparency and cost governance.

6.3 The 11 AI capabilities and engine guidance

6.4 PII redaction and on-prem fallback

PII redaction runs first in every flow. For inputs whose data class forbids cloud processing (e.g., raw CNIC, NADRA payloads, confidential/VIP tickets), the selector routes the call to the on-prem path (Llama/Qwen + Tesseract/Whisper). The redaction layer outputs both the redacted payload sent downstream and a reversible mapping (kept only in memory for the duration of the call, never persisted) so the final response can be re-hydrated before being returned to the caller. This satisfies the data-residency-aware principle from §1.

7. Real-time Layer

The WebSocket gateway powers four classes of live experience:

1. Ticket updates — status, assignment, SLA, watcher notifications, file attachments.
2. Presence and typing — for ticket-scoped threads and internal comms.
3. 3-tier internal comms — (1) ticket-scoped private threads, (2) org-wide inbox DMs/groups, (3) full Slack-style channels (threads, pins, read receipts).
4. Live dashboards — analytics push (e.g., open-ticket counts, SLA breach alerts) into role dashboards.

The gateway authenticates connections against Keycloak tokens (verified on connect and on periodic refresh), authorizes each subscription against RBAC and ABAC, and never trusts a client-asserted scope. Scaling: when more than one gateway instance is needed, all instances share a Redis pub-sub adapter so an event produced by any NestJS module or any BullMQ worker on any host reaches the correct connected session no matter which instance it lives on. Backpressure is handled per-connection (drop typing indicators before dropping ticket updates) and the gateway emits OTel spans for connect/disconnect/message so that the Loki/Grafana plane can correlate latency and drops.

8. Search (Meilisearch)

MariaDB full-text is weak for Urdu and Sindhi scripts; Meilisearch owns multilingual search across the portal.

• What is indexed: tickets (title, description, status, department, requester), KB articles (title, body, attachments-as-text), officials (from the Brand & Officials CMS), documents (extracted text after OCR), and chat messages (the 3-tier comms).
• Indexing pipeline: NestJS modules and BullMQ workers write to MariaDB as the system of record; a search-indexer worker listens to domain events (or polls change streams) and pushes normalized, redacted documents to Meilisearch. PII is redacted before indexing; confidential/VIP tickets are excluded or masked per policy.
• Multilingual tokenization: Meilisearch's built-in Roman/Arabic-script normalization handles Urdu and Sindhi acceptably; locale is stored per document and queries carry the user's locale so result ranking can prefer same-language matches.
• Search API surface: the NestJS API exposes /search (and scoped variants like /search/tickets, /search/kb) backed by Meilisearch, with role-scoped filtering applied so a requester cannot search into another company's tickets.
• Re-indexing: indexes are versioned; re-indexing happens in a shadow index that is atomically swapped to avoid search downtime during schema changes.

9. Storage & Files

• Object store: MinIO (S3-compatible), deployed on the same host. Buckets per content class: uploads, generated-docs, moms, avatars, exports. All buckets are encrypted at rest (MinIO KMS or SSE-S3) and versioned.
• AV scan pipeline: every upload is enqueued to a ClamAV worker; the file is quarantined (not visible to other users) until the scan returns clean. Infected files are isolated and an audit event is raised; the requester is notified in-app.
• Presigned URLs: all downloads go through time-limited presigned URLs minted by the API after an authorization check; URLs expire in minutes, are bound to the requesting user where possible, and are logged.
• Retention linkage: every file links to its source record (ticket, MoM, KB article) via a fil_attachment row; retention rules (from the data-classification policy, see §16) drive a scheduled cleanup worker that revokes links, expires presigned URLs, and (after the retention window) purges the object.
• Backup: MinIO buckets are mirrored nightly to off-host storage; versioning provides point-in-time recovery for accidental overwrites.

10. Identity & Auth

• Keycloak is the identity provider, speaking OIDC to the NestJS API and the Next.js portal. Two realm classes: one for government staff (federated to existing government IdP where possible) and one for company representatives / citizens.
• 2FA: TOTP (Authenticator apps) by default for staff; SMS OTP as a fallback or for company reps where device support is limited. Step-up auth re-challenges the user for sensitive actions (closing a VIP ticket, deleting a record, exporting sensitive analytics, changing a Primary Authorized Rep).
• Sessions/JWT: short-lived access tokens (minutes), longer-lived refresh tokens (days), both revocable via a Keycloak-side denylist mirrored into Redis for fast enforcement.
• RBAC enforcement runs at two layers: the API layer (RolesGuard / PermissionsGuard) for coarse route-level access, and the data layer (Prisma query extensions / row-level scopes) so that even a logged-in staff member can only read tickets their role and department entitle them to. For confidential/VIP tickets, an ABAC check extends RBAC: only explicitly listed watchers and the escalation chain can read.
• Audit: every login, step-up, role change, and permission override is written to the append-only audit log.

11. Feature Flag Service

Every capability in the system is toggleable by the Super Admin, per department and per environment.

• Storage: flags live in MariaDB (ffg_flag, ffg_override) with a Redis cache in front; reads are sub-millisecond.
• Resolution order: environment default → department override → user-segment override → off. The first match wins.
• Gating: code paths check flags via a thin FeatureFlags client. The FeatureFlagGuard can short-circuit an entire route; inside services, flags branch logic (e.g., "if MoM-transcription is on, also transcribe audio; else upload-only").
• Caching & invalidation: flags are cached per request and invalidated via Redis pub-sub when the admin saves a change, so toggles take effect within seconds without a redeploy.
• Audit: every flag change (who, what, when, before/after) is recorded in aud_event and surfaced in the admin UI; the current flag set is exportable as JSON for reproducibility.
• Bootstrap flags: critical defaults (e.g., "AI enabled", "two-way inbound email") are seeded in migrations so a fresh environment starts in a known state.

12. Notifications Pipeline

The pipeline fans a single domain event (e.g., "ticket updated") out to multiple channels per the recipient's preferences.

A domain event reaches the dispatcher, which resolves the per-recipient channels (email/SMS/WhatsApp/in-app), locale, and preference-center settings (digest vs immediate, quiet hours, per-channel opt-ins). Templates are rendered server-side in the recipient's locale with both Gregorian and Hijri dates. Each message is enqueued to a per-channel BullMQ queue so that a WhatsApp outage cannot block email. Outbound adapters call Mailjet, the SMS gateway, the WhatsApp Business API, and the in-app WebSocket push.

Two-way inbound is a first-class path: inbound email replies (via Mailjet's inbound parse) and inbound WhatsApp replies (via the WhatsApp webhook) are parsed, the ticket is resolved from the reference in the subject/recipient, and the message is appended to the ticket thread, triggering normal watcher notifications. Rate limiting is applied per channel, per recipient, and per tenant to avoid provider throttling and to respect user quiet hours. Digests and preference center let users batch or mute non-urgent notifications.

13. Integrations Layer

The integrations module exposes every external government and third-party system behind an adapter. The rest of the system never imports a vendor SDK directly.

Cross-cutting concerns:

• Idempotency: every outbound call carries an idempotency key; retries never double-write.
• Retries with exponential backoff + jitter; max attempts and a dead-letter queue for permanently-failed calls.
• Circuit breakers per adapter (e.g., Opossum) so an external outage degrades gracefully instead of cascading.
• Secrets for every adapter live in the secrets vault (see §16), never in env files in the repo.
• Audit: every outbound call and every inbound webhook payload is recorded (with sensitive fields redacted) in int_call.

14. Analytics & BI

• Aggregates in MariaDB: the Analytics module maintains materialized views and lightweight cubes (refreshed by scheduled workers) so dashboard reads do not hit the transactional tables. Heavy historical queries run against a read replica once replicas are introduced (see §5.6).
• Metabase: self-hosted and embedded via signed URLs for internal roles; runs read-only against a dedicated analytics schema/replica. Used for ad-hoc exploration and operational reporting.
• Custom dashboards: ECharts/Recharts in the Next.js portal for the seven role dashboards, the GIS/district heatmap, and the public transparency dashboard. Public-facing dashboards read from anonymized aggregates (no PII, no individually-identifiable data) sourced from a separate denormalized table populated by a scheduled worker.
• Live push: selected dashboard tiles subscribe to the WebSocket gateway for real-time counts (e.g., open tickets this hour).
• Exports: server-side PDF via Puppeteer (for letter-quality reports) and Excel (via a Node library) generated by BullMQ workers; the user is notified when the export is ready and downloads via a presigned URL.
• Scheduled digests: a cron-driven BullMQ job produces per-role/per-department digests on daily/weekly cadences, rendered multilingually and sent through the notifications pipeline.

15. Multilingual / i18n / RTL

• Languages: EN (master/source), UR, Sindhi — all three treated as first-class. The backend stores locale per user and propagates it via the JWT and every job payload, so templates, search ranking, and AI translation all receive the right locale without re-prompting the user.
• RTL: handled by the renderer (Docusaurus locale for docs; Next.js dir switching for the portal). No inline direction hacks; layout uses logical properties (padding-inline-start, etc.) so a single component tree serves both directions.
• Dual calendar: Gregorian and Hijri displayed together (government convention). Date formatting goes through a locale-aware formatter that emits both; storage is always UTC Gregorian, Hijri is derived at presentation.
• Translation workflow: source content (KB articles, circulars, SOPs) is authored in EN and translated into UR/Sindhi via the AI translation capability (§6.3 #6) using _glossary.md constraints; human reviewers approve before publication. Status of each translation is tracked per content item.
• Backend storage: locale on usr_user and on every content row; localized columns use a *_en / *_ur / *_sd pattern or a json map per the module's needs, decided once per table and applied consistently.

16. Security Architecture

• Encryption at rest: MariaDB (table-level/TDE where supported), MinIO (bucket SSE), and backups all encrypted. Keys managed by the secrets vault / KMS, not embedded in images.
• Encryption in transit: TLS terminated at nginx; internal service-to-service runs TLS within the host network where feasible; all external provider calls are HTTPS with certificate validation.
• Secrets management: a secrets vault (e.g., HashiCorp Vault, or the Server4Sale-managed secrets store) holds DB passwords, provider API keys, signing keys. Applications fetch at startup; nothing is committed to the repo.
• RBAC + ABAC: as in §10 — coarse route guards plus row-level scoping; confidential/VIP tickets require explicit ABAC inclusion.
• Audit trail: append-only (aud_event), partitioned by month, exported to off-host storage; tampering is detectable via hash chaining.
• Rate limiting & WAF: nginx-level rate limits and OWASP CRS (modsecurity) in front of the API; per-user throttling in NestJS via @nestjs/throttler backed by Redis.
• Pen-testing plan: scheduled external penetration tests before each major release and after any security-sensitive change; findings tracked to closure.
• CERT-PK coordination: incident-response runbook aligned with CERT-PK notification expectations; contact and escalation paths maintained in the ops runbook.
• CII registration: SITP is treated as Critical Information Infrastructure under Pakistan's framework; registration is pursued and the controls above support the CII posture.
• Data classification & retention: every record is tagged with a data class (Public / Internal / Confidential / Restricted); retention rules in §9 drive cleanup; archival aligns with Sindh Archives rules.
• Detailed security and compliance controls: see 16-security-and-compliance/en.md.

17. Deployment & Hosting

Provider: the SITP is hosted on Server4Sale ("Powered by Server4Sale"). The detailed hosting topology is TBD, but the provider is fixed. The verified server environment on the host is:

Static docs site is served by nginx directly at https://sindhitportal.maahir.io/docs/ from the Docusaurus static build output.

The reverse proxy is the single ingress. The static plane is the Docusaurus build served straight from disk. The app plane runs as Docker containers — the portal, API, WebSocket gateway, AI service, BullMQ worker pools, Metabase, and Keycloak — each with its own container and resource limits. The stateful plane — MariaDB 10.11, Redis, Meilisearch, MinIO + ClamAV — runs on the host (MariaDB is already installed at the OS level and shared with the containers over the local network).

Orchestration: Docker Compose today, with the option to graduate to Kubernetes when the deployment footprint outgrows a single host. The Compose file is structured so that each service definition maps cleanly to a future Kubernetes manifest (one service per Deploy, clear healthchecks, explicit resource requests/limits, secrets as env-from).

18. Observability

• Logs: structured JSON logs from every service, shipped to Loki; correlated by trace_id and user_id.
• Metrics: application and infrastructure metrics in Grafana (Prometheus-style exposition where useful; MariaDB exporter, Redis exporter, nginx exporter, Node exporter).
• Traces: OpenTelemetry instrumentation across NestJS, the FastAPI AI service, and BullMQ workers; traces collected and visualized in Grafana Tempo or equivalent.
• Errors: Sentry for client- and server-side error aggregation, with release health and source maps.
• Uptime & status: a public status page reporting portal, API, docs site, and notifications health; internal SLO dashboards in Grafana.

19. CI/CD

• Pipeline (GitHub Actions or GitLab CI, per the locked stack): on every push — lint → typecheck → unit tests → build. On merge to main — full integration test suite, container build, and push to the registry.
• Environments: dev → staging → prod, with manual approval gates before staging and prod.
• Database migrations: Prisma migrations run as a gated, pre-deploy step against the target environment; migrations are forward-only and reversible where safe; rollback is a deliberate, reviewed migration, not an automated revert.
• Deploy: containers are pulled and recreated via Compose (today); rolling updates minimize downtime. Healthchecks gate the cutover; nginx drains before removing the old container.
• Hotfix path: a fast-track pipeline branch with the same quality gates but expedited approvals.

20. Scalability & Performance

• Caching: Redis caches hot reads (RBAC resolution, feature flags, KB lookups, dashboard aggregates) with explicit TTLs and pub-sub invalidation. Cache misses fall through to MariaDB and re-populate.
• Queue offload: long-running work (OCR, AI, exports, fan-out) is enqueued — see §4 and §12 — so the request path stays fast.
• Read replicas: planned for analytics and read-heavy public paths — see §5.6.
• CDN: static assets (portal bundles, docs site, images) served via CDN in front of nginx; PWA assets cached client-side.
• Connection pooling: Prisma's connection pool sized to MariaDB's max_connections; long-lived workers reuse pooled connections; transient serverless-style callers are avoided.
• Target NFRs (non-functional requirements — see 03-non-functional-requirements/en.md for the authoritative list): the architecture is sized to meet the latency, throughput, availability, and concurrency targets set there; capacity reviews are run against Grafana dashboards each quarter.

21. Disaster Recovery (baseline)

Baseline DR is a locked non-functional requirement; detailed runbooks are deferred. The baseline posture:

• Backups: MariaDB logical + physical + binlog (§5.7); MinIO bucket mirror; Keycloak realm export; Metabase configuration export. All shipped off-host.
• RPO target: placeholder — to be confirmed in the DR runbook; current capability supports low single-digit minutes via binlog shipping.
• RTO target: placeholder — to be confirmed in the DR runbook; current capability supports recovery on the same host within hours via mariabackup restore, and off-host recovery within a longer window.
• Failover: single-host today; a warm standby host and DNS-level failover are the documented next step.
• Drills: quarterly restore drills, with results attached to the runbook.

Detailed runbooks (per-service recovery, contact tree, comms templates) are produced as a separate operational document.

22. Open Questions / TBD

End of document.