AI / OCR Specification

The authoritative engineering specification for all AI capabilities, the pluggable engine abstraction, prompt/pipeline design, PII redaction, governance, evaluation, and ethics on the Sindh IT Portal — Facilitation Desk (SITP).

1. Scope & Principles

This document is the single source of truth for how AI is used inside SITP. It defines the interfaces every AI call goes through, the engines that implement them, the eleven product capabilities that consume them, the prompt/pipeline governance, the PII redaction pipeline, the cost/usage model, the evaluation regime, and the ethics guardrails. It implements the "Pluggable AI/OCR" decision locked in _context.md §3 and the AI architecture described in /specs/en/15-tech-architecture/ §6.

The system of record is MariaDB 10.11 (not PostgreSQL). Semantic/vector search is delegated to Meilisearch and the AI service's own embedding store; no pgvector dependency exists anywhere in the stack. The AI service is a Python FastAPI microservice invoked by the NestJS API gateway and by BullMQ workers — never directly by clients.

1.1 AI principles

1.2 Non-goals

• This document does not specify the chatbot's knowledge-base curation workflow (see /specs/en/18-knowledge-base-sop/).
• It does not define RBAC for who may consume AI outputs (see /specs/en/04-roles-permissions/); it only requires that AI outputs inherit the recipient's existing scoping.
• It does not cover model fine-tuning of vendor cloud models (out of scope for V1; revisit in Phase 2).

2. Pluggable Engine Architecture

2.1 Interface contracts

The FastAPI AI service exposes four stable interface families. Every consumer (NestJS module or BullMQ worker) programs to the interface, never to a vendor SDK. The contracts below are TypeScript-flavored pseudocode; the Python implementations satisfy the same shape.

// ---- LLM family -----------------------------------------------------------
interface LLMClient {
  /** Single-turn completion (instructions + input → text). */
  complete(req: LLMRequest): Promise<LLMResponse>;
  /** Multi-turn chat (system + user/assistant turns → assistant text). */
  chat(req: ChatRequest): Promise<ChatResponse>;
  /** Embedding vectors for retrieval / similarity (on-prem by default). */
  embed(req: EmbedRequest): Promise<EmbedResponse>;
}

interface LLMRequest {
  promptTemplateId: string;        // e.g. "summary.v3" — versioned (§8)
  variables: Record<string, unknown>;
  locale: "en" | "ur" | "sd";
  sensitivityClass: "public" | "internal" | "confidential" | "restricted";
  maxOutputTokens: number;
  temperature: number;             // capability-default unless overridden
  responseFormat?: "text" | "json"; // json → schema-validated output
  jsonSchema?: object;             // required when responseFormat = "json"
  citationsRequired: boolean;      // force the model to cite source IDs
}

interface LLMResponse {
  text: string;
  structured?: object;             // populated when responseFormat = "json"
  citations: Citation[];           // [{type:"ticket"|"attachment"|"kb", id, span}]
  finishReason: "stop" | "length" | "content_filter" | "error";
  usage: { promptTokens: number; completionTokens: number; };
  engineUsed: string;              // e.g. "azure-openai:gpt-4o"
  redactionsApplied: number;       // count of PII tokens masked in input
  latencyMs: number;
}

// ---- OCR family -----------------------------------------------------------
interface OCRClient {
  /** Extract text + layout from a scanned/image/PDF document. */
  extract(req: OCRRequest): Promise<OCRResult>;
}

interface OCRRequest {
  objectKey: string;               // MinIO key (already AV-scanned)
  languages: ("en" | "ur" | "sd")[]; // drives tessdata / cloud language hint
  preserveLayout: boolean;         // keep reading order + table structure
  confidenceFloor: number;         // 0..1; below this → review queue
  sensitivityClass: SensitivityClass;
}

interface OCRResult {
  text: string;                    // plain text, reading-order preserved
  blocks: OCRBlock[];              // paragraphs/tables/lines with bbox + confidence
  languageDetected: "en" | "ur" | "sd" | "mixed";
  meanConfidence: number;          // 0..1
  lowConfidenceSpans: OCRSpan[];   // text spans below confidenceFloor
  needsManualReview: boolean;      // true iff meanConfidence < floor OR spans>threshold
  pageCount: number;
  engineUsed: string;
  latencyMs: number;
}

// ---- Transcription family -------------------------------------------------
interface TranscriptionClient {
  /** Speech-to-text for meeting recordings / voice notes. */
  transcribe(req: AudioRequest): Promise<Transcript>;
}

interface AudioRequest {
  objectKey: string;
  languages: ("en" | "ur" | "sd")[];
  diarize: boolean;                // separate speakers (Company / S&ITD / Dept)
  sensitivityClass: SensitivityClass;
}

// ---- Translation family ---------------------------------------------------
interface TranslationClient {
  /** Translate text between EN/UR/Sindhi with glossary constraints. */
  translate(req: TranslationRequest): Promise<TranslationResult>;
}

interface TranslationRequest {
  text: string;
  sourceLocale: "en" | "ur" | "sd" | "auto";
  targetLocale: "en" | "ur" | "sd";
  domain: "general" | "legal" | "technical" | "official-letter";
  glossaryId?: string;             // injected from _glossary.md constraints
  preserveFormatting: boolean;
  sensitivityClass: SensitivityClass;
}

type SensitivityClass = "public" | "internal" | "confidential" | "restricted";

interface Citation {
  type: "ticket" | "attachment" | "kb" | "mom" | "comms";
  id: string;
  span?: { start: number; end: number }; // char offsets in output text
}

2.2 Engine registry and selection

Engines are registered declaratively and selected per feature and per sensitivity class at call time. Selection inputs: (a) the feature-flag service, (b) the data classification of the input, and (c) an environment-level allow-list so a "sovereign" environment can forbid cloud engines entirely.

// ai_engine_configs — stored in MariaDB, cached in Redis, audited on change.
// One row per (capability, sensitivityClass) → ordered fallback chain.
type EngineConfig = {
  capability: CapabilityId;          // "ocr" | "summary" | "routing" | ...
  sensitivityClass: SensitivityClass;
  primary: EngineRef;                // first choice
  fallback: EngineRef[];             // ordered; tried on primary failure
  costBudgetPerCall?: Money;         // hard ceiling; call rejected if exceeded
  rateLimitPerMin?: number;          // per-tenant throttle
  enabled: boolean;                  // kill switch
};

type EngineRef = {
  family: "llm" | "ocr" | "transcription" | "translation";
  provider: "azure-openai" | "gemini" | "bedrock"
          | "ollama" | "vllm"              // self-hosted LLM
          | "tesseract"                     // self-hosted OCR
          | "google-doc-ai" | "azure-di" | "textract"  // cloud OCR
          | "zoom" | "meet"                 // cloud transcription
          | "whisper";                      // self-hosted transcription
  model: string;                      // e.g. "gpt-4o", "llama-3.1-70b", "qwen2.5-32b"
  residency: "cloud" | "on-prem";
};

Example registry rows (defaults — confirmed via §4 use-cases):

Invariant: for restricted inputs, the allow-list contains only residency: "on-prem" engines. The selector refuses to emit a cloud engine for a restricted class even if an admin misconfigures the registry — a hard policy check fails the call closed.

2.3 Request flow

Each AI call follows the same shape: resolve policy → select engine → redact PII → call → on failure fall back → log + audit → respond.

Written description. A request enters carrying its capability id, sensitivity class, locale, and payload. The gatekeeper first checks the feature flag (a disabled capability returns a structured "no suggestion" — never an exception to the caller). It then enforces the residency allow-list: a restricted payload can never select a cloud engine. It checks the per-call cost budget and per-tenant rate limit. The selector resolves the ordered engine chain from ai_engine_configs. PII is redacted before the payload leaves the trusted boundary if a cloud engine is selected; for on-prem engines, redaction is still applied for defence-in-depth but the mapping is retained for re-hydration. The adapter calls the engine; on failure (timeout, 5xx, content-filter, budget breach, or output-schema validation failure) the next fallback is tried, re-redacting fresh if needed. A successful output is de-redacted (PII tokens re-inserted from the in-memory vault), validated against its JSON schema and guardrails, and a citation check ensures traceable sources. Every call — success or failure — writes one immutable ai_runs row capturing engine, latency, tokens, cost, redaction count, prompt version, and output hash. If all engines fail, the caller receives a structured failure and degrades to manual mode (§11).

2.4 Data model (AI-specific tables)

The following tables (prefixed ai_) live in the same MariaDB database; full schemas are in /specs/en/05-data-model/.

3. Engine Catalog

For each engine family, the catalog lists concrete options, strengths, Urdu/Sindhi quality, indicative cost, data-residency posture, and recommended use in SITP. Costs are indicative, expressed relative to a baseline; actual budgets are set in ai_engine_configs.costBudgetPerCall.

3.1 LLM engines

Selection guidance. Use cloud LLMs for quality on public/internal data. Use Qwen 2.5 as the on-prem primary for Urdu/Sindhi because of its stronger Arabic-script training; fall back to Llama 3.1 for English-dominant tasks. Never route restricted payloads to any cloud LLM.

3.2 OCR engines

Selection guidance. Tesseract is the on-prem floor and the only permitted OCR for restricted inputs (CNIC scans, NADRA payloads, confidential evidence). Install the Urdu and Sindhi tessdata packages (tesseract-ocr-urd, tesseract-ocr-snd from the tessdata_fast repository, plus a curated snd.traineddata reviewed against Sindh-government document samples). Apply a post-OCR cleanup pass (whitespace normalization, diacritic repair, digit-form normalization Western↔Arabic-Indic) before handing text to the LLM.

3.3 Transcription engines

Selection guidance. Use the meeting provider's built-in transcription when the meeting is on that provider and the data class permits cloud processing. Use Whisper on-prem for restricted audio and as the universal fallback. Whisper benefits from a Sindhi-specific fine-tune evaluated against the golden set in §10.

3.4 Translation engines

Selection guidance. LLM-based translation is the default because it accepts _glossary.md constraints and produces tone-appropriate official text. Official letters and citizen-facing legal text require human review regardless of engine (§5.6).

4. The 11 AI Capabilities

Each capability is specified with: purpose, input, output, engine recommendation, prompt template (where applicable), human-in-the-loop checkpoint, fallback, cost estimate, and success metrics. Each capability has a stable CapabilityId used in ai_engine_configs and in feature flags (§13).

4.1 Capability 1 — OCR

Purpose. Digitize scanned notices, MoM images, ID documents, and any non-text PDF/image upload into structured, language-tagged text with layout preservation and confidence scores. Output feeds search indexing (Meilisearch), MoM extraction (§4.11), translation (§4.6), and the Files preview.

Input. A MinIO object key (already AV-scanned by ClamAV per /specs/en/15-tech-architecture/ §9), the requested languages, a layout-preservation flag, a confidence floor (default 0.75), and the sensitivity class of the owning record.

Output. Plain text (reading-order preserved), structured blocks (paragraph/table/line with bbox + per-block confidence), detected language, mean confidence, low-confidence spans, a needsManualReview flag, page count, engine used, latency.

Engine recommendation. Cloud-first for public/internal (Google Document AI → Azure Document Intelligence → Tesseract); Tesseract on-prem only for confidential/restricted. See §3.2.

Human-in-the-loop checkpoint. When needsManualReview is true, the document is routed to a per-department OCR review queue. A reviewer sees the original image side-by-side with the OCR text, edits inline, and confirms. Confirmed text replaces the AI text and is flagged human_verified=true in the search index.

Fallback. Cloud OCR unavailable → next cloud OCR → Tesseract on-prem. If all OCR fails, the file is still attached and visible; a placeholder note ("OCR pending — manual entry available") is shown and a retry job is scheduled.

Cost estimate. Cloud OCR ~$0.05–$0.15 per page (provider-dependent); Tesseract effectively free (capex). Budget per call set in ai_engine_configs.

Success metrics. Mean character-level accuracy vs. golden set (≥ 97% EN, ≥ 90% UR, ≥ 85% SD); % of documents needing manual review (target ≤ 15%); OCR latency P95 ≤ 8 s for a 10-page document.

{{templateId: ocr.cleanup.v2}}
{{locale: en}}
You are correcting OCR output for a {{domain}} document in {{detected_language}}.
The source is OCR which may introduce: broken ligatures, dropped diacritics,
Arabic-Indic digits mixed with Western digits, broken words across lines,
spurious whitespace, and table-cell bleed.

Rules:
1. Output the corrected text only — no commentary.
2. Preserve the original paragraph and table structure.
3. Normalize all digits to Western (0-9) unless the source is a date in
   Hijri format (preserve as-is and tag [HIJRI]).
4. Repair obvious ligature/diacritic errors using context; if a word is
   ambiguous, keep the OCR reading and append [?].
5. Do NOT invent content. If a span is illegible, output [ILLEGIBLE].
6. If the document is in Urdu or Sindhi, ensure right-to-left joining is
   preserved; do not reorder words.

Raw OCR:
"""
{{raw_ocr_text}}
"""

4.2 Capability 2 — Summary

Purpose. Produce a concise summary of a ticket and of each attachment, configurable in length, multilingual, with citations back to the source spans. Used in triage views, ticket lists, the RAG assistant (§6), and digests.

Input. Ticket id (with locale), the ticket body + all message history + attachment OCR text (PII-redacted), a target length (short ≤ 60 words, medium ≤ 150, long ≤ 400), and the target locale.

Output. Summary text + citations array + structured keyPoints[] (≤ 5 bullets) + entities[] (departments, companies, statutes referenced).

Engine recommendation. Cloud LLM (Azure OpenAI GPT-4o) for public/internal; on-prem Qwen 2.5 for confidential/restricted. See §3.1.

Human-in-the-loop checkpoint. Summaries are presented as suggestions; an officer can edit, dismiss, or pin. Pinned summaries appear in the ticket header. No summary is auto-published to a citizen-facing surface without officer approval.

Fallback. Cloud LLM → on-prem LLM → if both fail, show the first 400 characters of the ticket body as a fallback "preview".

Cost estimate. ~$0.002 per summary (GPT-4o, ~1.5k tokens in / 300 out).

Success metrics. ROUGE-L vs. human reference ≥ 0.45 on golden set; citation precision 100% (every cited span must exist); officer "kept as-is" rate ≥ 60%.

{{templateId: summary.ticket.v3}}
{{locale: {{target_locale}}}}
You are summarizing a Government of Sindh facilitation-desk ticket for an
officer. Be factual, neutral, and government-formal.

Inputs:
- Ticket title: {{title}}
- Filed by: {{requester_company}} ({{entity_type}})
- Against department: {{department}}
- History (newest last):
{{message_history_redacted}}
- Attachment excerpts (PII-redacted):
{{attachment_excerpts}}

Produce JSON matching this schema:
{
  "summary":   "string, {{length}} words or fewer, in {{target_locale}}",
  "keyPoints": ["string", "..."],     // up to 5, each ≤ 20 words
  "entities":  [{"type":"dept|company|statute|person", "name":"string"}],
  "citations": [{"type":"ticket|attachment", "id":"string", "span":[start,end]}]
}

Rules:
- Cite every factual claim with a citation whose span maps into the inputs above.
- Do NOT include CNIC, phone, email, or bank details even if present in inputs.
- Do NOT speculate. If status is unclear, say so.
- Translate the summary to {{target_locale}} using the approved glossary.

4.3 Capability 3 — Auto-routing / classification

Purpose. Predict the destination department, section, and category from the free-text complaint, with a confidence score and top-3 suggestions. The officer confirms; confirmed or corrected predictions feed a feedback loop that improves the model.

Input. Ticket title + body + requester entity type + locale. (No PII is needed for routing — it is redacted first.)

Output. Structured: top3: [{department, category, confidence, rationale}], overallConfidence, needsHumanReview flag (true if top-1 confidence < 0.70), plus a rulesHit[] array (the deterministic rules layer fired before the LLM).

Engine recommendation. Hybrid: deterministic rule layer first (keyword/regex/known-patterns table maintained by S&ITD admins) → LLM only for ambiguous cases. Cloud LLM for public/internal; on-prem for restricted.

Human-in-the-loop checkpoint. A triage officer sees the top-3 suggestions and confirms or overrides. Every override is logged to ai_feedback and used to retrain the rule layer and to build a fine-tuning corpus (Phase 2).

Fallback. If the LLM is unavailable, the rule layer alone produces suggestions; if the rule layer is also empty, the ticket is routed to the default S&ITD triage queue unchanged.

Cost estimate. ~$0.0005 per classification (GPT-4o-mini, short prompt).

Success metrics. Top-1 accuracy ≥ 75%; top-3 accuracy ≥ 92%; officer override rate ≤ 25%; median latency ≤ 1.5 s.

{{templateId: routing.classify.v2}}
{{locale: {{source_locale}}}}
You are a routing assistant for the Sindh IT Portal. Classify the complaint
into exactly one department and one category from the provided catalogs.

Departments (code → name):
{{department_catalog}}

Categories (code → name, scoped to department):
{{category_catalog}}

Complaint (PII-redacted):
Title: {{title}}
Body: {{body_redacted}}
Requester entity type: {{entity_type}}

Produce JSON:
{
  "top3": [
    {"department":"<code>","category":"<code>","confidence":0.0,
     "rationale":"≤ 25 words"}
  ],
  "overallConfidence": 0.0,
  "needsHumanReview": true,
  "signals": ["keyword matched: ...", "..."]
}

Rules:
- Confidence values are in [0,1] and must sum to ~1 across top3.
- Use ONLY departments and categories from the catalogs. Never invent codes.
- Rationale must reference specific words from the complaint.
- If the complaint is out of scope (not a GoS department matter), set
  department = "OUT_OF_SCOPE" and confidence high.

4.4 Capability 4 — Urgency / sentiment scoring

Purpose. Suggest a priority (Low / Medium / High / Critical) and flags (vip, urgent, legal_deadline, media_attention) with a rationale. Drives triage ordering and SLA-tick weighting.

Input. Ticket body + requester context (is the company PSEB-registered / large employer? — boolean only, no PII) + any explicit deadlines mentioned.

Output. { priority, vipFlag, urgentFlag, sentimentScore (-1..1), rationale, signals[] }.

Engine recommendation. On-prem Llama 3.1 70B or Qwen 2.5 for cost on the high volume; cloud for the rare needsHumanReview cases. Deterministic rules first (regex for "RTI", statutory deadline phrases, VIP name list).

Human-in-the-loop checkpoint. Suggestion is shown to the triage officer who sets the final priority. Changes feed ai_feedback.

Fallback. Rules-only priority when the LLM is unavailable; default Medium if rules are empty.

Cost estimate. ~$0.0002 per call (on-prem amortized to ~0).

Success metrics. Priority agreement with officer ≥ 80%; VIP recall ≥ 95% (false negatives are costly); sentiment correlation with CSAT ≥ 0.5.

{{templateId: urgency.score.v2}}
{{locale: {{source_locale}}}}
Assess the urgency of this facilitation-desk ticket.

Ticket (PII-redacted):
{{body_redacted}}

Requester context (no PII):
- PSEB-registered: {{pseb_registered}}
- Employee-band: {{employee_band}}   // small | medium | large
- Explicit deadline mentioned: {{deadline_extracted}}

Produce JSON:
{
  "priority": "Low|Medium|High|Critical",
  "vipFlag": false,
  "urgentFlag": false,
  "legalDeadlineFlag": false,
  "mediaAttentionFlag": false,
  "sentimentScore": 0.0,   // -1 (very negative) .. 1 (very positive)
  "rationale": "≤ 30 words",
  "signals": ["..."]
}

Rules:
- "Critical" implies action today and is reserved for: legal/statutory
  deadlines within 48h, VIP requesters, or safety-of-life language.
- VIP list is provided server-side; do not infer VIP from names alone.
- Never mark Critical based on sentiment alone.

4.5 Capability 5 — Draft reply suggestions

Purpose. Generate context-aware officer reply drafts in the requester's preferred locale, in the correct tone, editable before send. Reduces officer typing and enforces a consistent, respectful government register.

Input. Ticket id, locale, the latest inbound message, the ticket history (redacted), the relevant KB/SOP excerpts retrieved by the RAG layer (§6), and a tone hint (informational, apologetic, directive, closure).

Output. Up to 3 alternative drafts, each { body, tone, citations[] }.

Engine recommendation. Cloud LLM (Azure OpenAI GPT-4o) primary; on-prem Qwen fallback.

Human-in-the-loop checkpoint. Drafts are never auto-sent. The officer selects one, edits, and sends. The final sent message is stored; the diff between draft and final feeds the improvement loop.

Fallback. If the LLM is unavailable, show a curated template library (per category) the officer can pick from manually.

Cost estimate. ~$0.003 per draft set.

Success metrics. Draft acceptance (sent with ≤ 20% edit) ≥ 50%; officer time-to-first-response reduction ≥ 30% in A/B; tone compliance (reviewer-rated) ≥ 90%.

{{templateId: draft_reply.v3}}
{{locale: {{target_locale}}}}
You are drafting a reply for an officer of the Sindh IT Portal facilitation
desk to a company representative. Write in {{target_locale}}.

Context (PII-redacted):
- Ticket: {{ticket_ref}} ({{department}})
- Latest message from requester: {{latest_message_redacted}}
- History summary: {{history_summary}}
- Applicable SOP/KB (retrieved):
{{retrieved_context_with_ids}}
- Tone hint: {{tone_hint}}

Produce JSON:
{
  "drafts": [
    {"body":"string in {{target_locale}}, 80-200 words",
     "tone":"informational|apologetic|directive|closure",
     "citations":[{"type":"kb|sop|ticket","id":"string"}]}
  ]
}

Rules:
- Address the requester respectfully per {{target_locale}} convention.
- State the next concrete step and a realistic timeframe.
- Cite the SOP/KB used; never invent policy or commit to actions outside SOP.
- Do NOT include any PII not already in {{latest_message_redacted}}.
- Offer 2-3 variants differing in tone or length, not in substance.
- Close with the standard sign-off block (added by the caller, not the model).

4.6 Capability 6 — Translation (EN ↔ UR ↔ Sindhi)

Purpose. Translate messages, MoM, KB articles, circulars, and official letters between EN/UR/Sindhi with glossary constraints and tone appropriate to the target audience. Citizen-facing and official-legal outputs always go through human review.

Input. Source text (redacted), source locale (or auto), target locale, domain (general | legal | technical | official-letter), glossary id.

Output. { translated, detectedSourceLocale, glossaryHits[], confidence }.

Engine recommendation. Cloud LLM with glossary injection for public/internal; on-prem Qwen 2.5 (best SD) for confidential/restricted.

Human-in-the-loop checkpoint. Required for: official letters (Module L), citizen-facing legal text, RTI-category responses, and any text marked official-letter domain. Optional for internal messages.

Fallback. Dedicated MT API → on-prem LLM → if all fail, the original-language text is shown with a banner "Translation unavailable".

Cost estimate. ~$0.001–$0.005 per page depending on engine.

Success metrics. COMET or BLEU vs. golden set: EN↔UR ≥ 0.40 BLEU; UR↔SD ≥ 0.35 BLEU; glossary term adherence 100%; human reviewer acceptance ≥ 80%.

{{templateId: translation.glossary.v2}}
{{locale: {{target_locale}}}}
Translate the following {{source_locale}} text into {{target_locale}}.

Domain: {{domain}}
Use ONLY these approved term mappings (do not paraphrase the left column):
{{glossary_pairs}}

Text (PII-redacted):
"""
{{source_text}}
"""

Rules:
- Preserve paragraph breaks, list markers, and table structure.
- Preserve all numbers, dates, ticket IDs, and URLs verbatim.
- Preserve [BRACKETED_PLACEHOLDERS] verbatim — they are template tokens.
- If a term is ambiguous, choose the glossary mapping; if no mapping,
  keep the source term in parentheses after the translation on first use.
- For domain=official-letter, use formal government register and the
  approved salutation/valediction for {{target_locale}}.
- Output the translation only, no commentary.

4.7 Capability 7 — Duplicate / similarity detection

Purpose. Detect duplicate or highly-similar tickets (same company re-filing, or multiple companies affected by the same issue), suggest merges, and surface related tickets to the officer.

Input. A new ticket's title + body (redacted), or an explicit "find similar" query on an existing ticket. Optionally scoped by company, department, or time window.

Output. candidates: [{ ticketId, score, reason }] sorted by score, with mergeSuggested flag when score ≥ threshold and the same requester/entity is involved.

Engine recommendation. Embeddings stay on-prem always. Generate embeddings with the on-prem embedding model; similarity search runs in Meilisearch (which supports vector search alongside the multilingual index). For confidential/VIP tickets, embeddings are computed but only matched within the same confidentiality ring.

Human-in-the-loop checkpoint. Merge is always an explicit officer action with a reason. The system never auto-merges.

Fallback. If embeddings are unavailable, fall back to Meilisearch lexical similarity (BM25-like) over titles.

Cost estimate. Negligible (on-prem embeddings amortized).

Success metrics. Precision@10 ≥ 0.6 on the golden duplicate set; recall of true duplicates ≥ 0.85; false-merge-suggestion rate ≤ 5%.

{{templateId: dedup.explain.v1}}
{{locale: {{source_locale}}}}
You are explaining why two tickets may be duplicates, to help an officer
decide whether to merge. Both ticket texts below are PII-redacted.

Ticket A ({{ticket_a_ref}}):
{{ticket_a_redacted}}

Ticket B ({{ticket_b_ref}}):
{{ticket_b_redacted}}

Cosine similarity of embeddings: {{similarity_score}}

Produce JSON:
{
  "areDuplicates": true|false,
  "confidence": 0.0,
  "sharedAspects": ["...", "..."],   // concrete shared facts
  "differences":   ["...", "..."],
  "mergeSuggested": true|false,
  "rationale": "≤ 40 words"
}

Rules:
- Two tickets are duplicates only if they describe the SAME underlying issue
  from the SAME requester (or a verifiable proxy). Similar topic ≠ duplicate.
- Never suggest merging across different companies unless evidence is strong.

4.8 Capability 8 — Public chatbot

Purpose. A public-facing assistant on the portal that answers FAQ, assists with filing, and looks up ticket status. Retrieval-Augmented Generation (RAG) over the public Knowledge Base; strict scope; human handoff with full context.

Input. User message (free text, any locale), session id, optional ticketRef for status lookup (only if the user is authenticated as the ticket's owner), locale.

Output. { answer, citations[], suggestedActions[], handoff: bool }. Answers cite KB articles by id and title.

Engine recommendation. Cloud LLM (Azure OpenAI GPT-4o) for quality; the retrieval index is public KB content only (no ticket bodies, no PII).

Human-in-the-loop checkpoint. The bot never files, modifies, or closes a ticket autonomously. A "Talk to a facilitator" handoff packages the conversation transcript + cited articles and creates a ticket-scoped thread for a human.

Fallback. If the LLM is down, the bot degrades to pure keyword search over the KB with article links.

Cost estimate. ~$0.005 per conversation (avg 4 turns).

Success metrics. Deflection rate (resolved without human handoff) ≥ 40%; citation precision 100%; guardrail violation rate = 0; CSAT on bot answers ≥ 4/5.

Guardrails. The system prompt enforces: answer only from retrieved KB; never accept or store PII (if the user types a CNIC/phone, the bot refuses and points to the authenticated ticket status page); never opine on politics, religion, or pending litigation; never promise deadlines outside published SLA; always offer human handoff.

{{templateId: chatbot.public.v3}}
{{locale: {{user_locale}}}}
You are the Sindh IT Portal public assistant. You help IT companies and
citizens with the Sindh IT Portal — Facilitation Desk.

Retrieved Knowledge Base context (cite these by id):
{{retrieved_kb_chunks}}

Conversation so far:
{{conversation}}

User's latest message:
{{user_message}}

Produce JSON:
{
  "answer": "string in {{user_locale}}, ≤ 150 words",
  "citations": [{"type":"kb","id":"...","title":"..."}],
  "suggestedActions": [{"label":"...","action":"file|track|handoff|kb:read"}],
  "handoff": false
}

Hard rules:
- Answer ONLY using the retrieved KB context. If the answer is not there,
  say you don't know and suggest handoff.
- NEVER accept or repeat PII (CNIC, phone, email, bank). If the user sends
  any, warn them and do not store it.
- NEVER give legal advice or predict outcomes of specific cases.
- NEVER quote SLA numbers not present in the KB.
- Always offer handoff to a human facilitator as a suggested action.
- Keep answers concise and actionable.

4.9 Capability 9 — PII redaction

Purpose. Detect and mask personally-identifiable information (CNIC, phone, email, financial) before any cloud engine call. Reversible only internally via the short-TTL PII vault. This is the foundational capability — every other capability depends on it. Detailed pipeline in §7.

Input. Free text (or a document body) with sensitivity class and the destination residency (cloud vs on-prem).

Output. { redactedText, mappings: [{token, type, originalRef}], redactionCount }. The originalRef is an opaque handle into the encrypted ai_pii_vault; raw values never leave the AI service.

Engine recommendation. Always on-prem. Regex + curated gazetteer + a small on-prem NER model (e.g. a fine-tuned XLM-R or a spaCy pipeline with Urdu/Sindhi extensions). This capability never calls a cloud engine.

Human-in-the-loop checkpoint. None at runtime (must be fully automatic). Periodically sampled by the data-governance reviewer for precision/recall audit (§10).

Fallback. If the NER model is unavailable, regex-only redaction still runs. If even regex is down, the call is failed closed: no cloud engine is invoked for the payload; on-prem engines continue.

Cost estimate. Negligible (on-prem, CPU-only).

Success metrics. Recall ≥ 0.99 for CNIC (13-digit), ≥ 0.97 for phone/email, ≥ 0.95 for bank/IBAN; precision ≥ 0.98 (false redactions break reading, so precision matters); zero raw-PII egress incidents (monitored by a downstream canary).

{{templateId: pii_redact.rules.v1}}
{{locale: auto}}
PII redaction is rule-and-model based, not LLM-based, to guarantee determinism.
This template documents the rule set the redaction service applies.

Entity patterns (regex, applied with Unicode case-folding):
- CNIC:            ^\d{5}-\d{7}-\d$   (and un-dashed variant)
- Phone PK:        ^(\+92|0)?3\d{2}[-\s]?\d{7}$
- Email:           RFC-5322 simplified
- IBAN PK:         ^PK\d{2}[A-Z0-9]{24}$
- Bank account:    \b\d{10,18}\b within 2 tokens of {account|a/c|bank}
- Credit card:     Luhn-valid 13-19 digit
- Vehicle reg:     3-letter city code + digit patterns (Sindh plates)

Masking format: [{{TYPE}}_{{seq}}]   e.g. [CNIC_1], [PHONE_2], [EMAIL_1]

Behaviour:
- Each distinct raw value gets a unique token within the request scope.
- The mapping {token → raw} is written to ai_pii_vault (encrypted, TTL=10 min).
- For on-prem engines, the mapping is retained in memory for re-hydration.
- For cloud engines, the mapping NEVER leaves the AI service boundary.
- Aggressive mode (sensitivity=restricted): also mask named persons not on
  the public-officials allow-list.

4.10 Capability 10 — Trends / analytics

Purpose. Surface non-obvious patterns: topic clusters, anomalies (sudden spike in a category), SLA-breach predictors, and policy insights. Inputs are aggregated/anonymized; raw ticket bodies are never sent to a cloud LLM.

Input. Pre-aggregated time series and topical rollups produced by the Analytics module (Module I) — counts by department, category, district, SLA status, sentiment bucket. Optionally anonymized sample titles (PII-redacted, company name generalized to band).

Output. { insights: [{ finding, evidence, severity, suggestedAction }], topicClusters[], anomalies[] }.

Engine recommendation. Cloud LLM on aggregates only; classical ML (Prophet/STL for forecasting, Isolation Forest for anomalies) runs on-prem on the raw aggregates.

Human-in-the-loop checkpoint. Insights are presented to the S&ITD analytics reviewer and the Secretary briefing pipeline (§6). No insight auto-triggers an action.

Fallback. Classical ML alone; the LLM narrative is omitted.

Cost estimate. ~$0.01 per weekly insights run (low frequency).

Success metrics. Insight relevance rated ≥ 4/5 by reviewers; anomaly detection F1 ≥ 0.7 vs. labelled incidents; SLA-breach prediction AUC ≥ 0.8 at 48 h horizon.

{{templateId: trends.insights.v2}}
{{locale: en}}
You are analyzing aggregated, anonymized facilitation-desk metrics to produce
insights for S&ITD leadership. Input is AGGREGATES ONLY — no individual PII.

Aggregates (last {{window}}):
{{aggregates_json}}

Optional anonymized title samples (already PII-redacted):
{{anonymized_titles}}

Produce JSON:
{
  "insights": [
    {"finding":"≤ 30 words",
     "evidence":"reference to specific aggregate value",
     "severity":"info|watch|alert",
     "suggestedAction":"≤ 25 words"}
  ],
  "topicClusters": [{"label":"...","ticketCount":0,"trend":"up|down|flat"}],
  "anomalies":     [{"metric":"...","expected":0,"observed":0,"note":"..."}]
}

Rules:
- Cite the specific aggregate number that supports each finding.
- Do NOT name companies or individuals; refer to bands/sectors.
- Distinguish correlation from causation explicitly.
- suggestedAction must be within S&ITD's mandate (policy, escalation, SOP).

4.11 Capability 11 — MoM action-item extraction

Purpose. From an uploaded Minutes of Meeting (PDF/Word/images — OCR'd first if scanned), extract structured action items (owner, due date, decision, priority), produce a summary, and let the officer confirm before action items become sub-tasks. Implements the MoM-upload-first decision in _context.md §5.

Input. MoM file object key (already OCR'd via Capability 1 if needed), meeting metadata (date, participants, tripartite parties), locale(s) of the document.

Output. { summary, decisions[], actionItems: [{ owner, ownerType, dueDate, description, priority, linkedTicket }], rawTextRef }. Dates are parsed to ISO-8601; Hijri dates preserved and tagged.

Engine recommendation. Cloud LLM (GPT-4o, long context) after OCR + redaction for public/internal; on-prem Qwen 2.5 72B for restricted (some MoMs are confidential/VIP).

Human-in-the-loop checkpoint. Mandatory. The officer reviews the extracted action items in a side-by-side editor (raw MoM text ↔ extracted table), corrects owners/dates/descriptions, removes false positives, and confirms. Confirmed items become ticket sub-tasks; the confirmation is audit-logged. For sensitive/VIP tickets, the existing Chair/DG approval gate (see /specs/en/21-mom-meetings/) still applies before publication.

Fallback. If extraction fails, the raw OCR text is shown and the officer enters action items manually.

Cost estimate. ~$0.01 per MoM (GPT-4o, ~6k tokens in).

Success metrics. Action-item precision ≥ 0.85; recall ≥ 0.80; date-parse accuracy ≥ 0.90; officer median edit time ≤ 5 minutes per MoM.

{{templateId: mom.extract.v3}}
{{locale: {{source_locale}}}}
You are extracting structured action items from a Minutes of Meeting (MoM)
of a tripartite review (Company + S&ITD + Department) on the Sindh IT Portal.

MoM text (OCR'd, PII-redacted):
{{mom_text}}

Meeting metadata:
- Date: {{meeting_date}}  (Hijri: {{meeting_date_hijri}})
- Parties: Company ({{company_band}}), S&ITD facilitator, {{department}}
- Linked ticket: {{ticket_ref}}

Produce JSON:
{
  "summary": "≤ 120 words in {{source_locale}}",
  "decisions": [
    {"text":"...","decidingParty":"company|sitd|dept|joint"}
  ],
  "actionItems": [
    {
      "owner":"name or role",
      "ownerType":"company|sitd|dept|external",
      "dueDate":"ISO-8601 or null",
      "dueDateHijri":"string or null",
      "description":"≤ 40 words",
      "priority":"Low|Medium|High",
      "linkedTicket":"{{ticket_ref}} or null"
    }
  ]
}

Rules:
- Extract ONLY explicit action items (verbs of commitment: will, shall, to
  complete, assigned to). Do NOT infer items from general discussion.
- If an owner is ambiguous, set ownerType and leave owner = "UNASSIGNED".
- If a due date is missing or relative ("within two weeks"), compute from
  {{meeting_date}} and tag the field dueDateInferred=true.
- Preserve Hijri dates alongside Gregorian; never silently convert.
- Link each action item to {{ticket_ref}} unless it clearly belongs elsewhere.
- Do not invent decisions; if none are explicit, return an empty decisions[].

5. Tiered AI Assistant (RAG)

The portal provides three tiers of an AI assistant — a Retrieval-Augmented-Generation (RAG) assistant scoped per role. All three share one retrieval backend and one engine abstraction; they differ in corpus, prompt, and UI affordances.

5.1 Tier matrix

5.2 RAG architecture

The retrieval backend is Meilisearch for lexical multilingual search plus a vector index for semantic similarity (Meilisearch supports hybrid search; an external embedding store in the AI service holds the canonical vectors for re-indexing). No pgvector is used; the system of record is MariaDB and vector search is delegated to Meilisearch/AI-service.

Written description. The user's question is embedded by an on-prem embedding model (embeddings never leave the boundary). A hybrid retrieval over Meilisearch combines lexical (multilingual tokenization, good for Urdu/Sindhi) and vector similarity, scoped by the user's RBAC + ABAC so a Staff officer cannot retrieve into another department's confidential tickets and a DG cannot see ticket bodies outside their portfolio. Top-K chunks are reranked (cross-encoder or LLM rerank) to improve precision. The prompt is assembled from a tier-specific system prompt, the retrieved chunks (each carrying its source id: ticket, attachment, KB, MoM, comms), and the question. The LLM produces an answer; the citation binder maps claims back to source ids by span. Post-guardrails enforce: no PII in the answer that wasn't in the (redacted) retrieved context, scope compliance (answer only from in-scope sources), and policy compliance (no invented SLA, no legal advice). The answer is returned with citations the UI renders as clickable links to the source record.

5.3 Per-tier prompts

{{templateId: assistant.staff.v2}}
{{locale: {{user_locale}}}}
You are the staff AI assistant for an officer working ticket {{ticket_ref}} on
the Sindh IT Portal. Answer the officer's question using ONLY the retrieved
context, which is scoped to this ticket and the officer's department SOPs/KB.

Retrieved context (each chunk has a source id):
{{retrieved_chunks_with_ids}}

Officer's question:
{{question}}

Produce JSON:
{
  "answer": "≤ 200 words in {{user_locale}}",
  "citations": [{"type":"ticket|attachment|kb|sop|mom","id":"...","span":[s,e]}],
  "followups": ["optional clarifying question", "..."],
  "confidence": 0.0
}

Rules:
- Cite every factual claim.
- If the answer is not in the retrieved context, say so and suggest where to look.
- Never reveal data from tickets outside this officer's scope.
- Offer to draft a reply or action item when the question implies one.

{{templateId: assistant.dg.v2}}
{{locale: {{user_locale}}}}
You are the DG oversight assistant on the Sindh IT Portal. The DG oversees
departments: {{dg_departments}}. Inputs are SLA/risk rollups and anonymized
ticket summaries within the DG's portfolio.

Retrieved rollups + summaries:
{{retrieved_rollups_with_ids}}

DG's question:
{{question}}

Produce JSON:
{
  "answer": "≤ 250 words, executive register",
  "citations": [{"type":"rollup|ticket|mom","id":"...","span":[s,e]}],
  "atRiskTickets": ["ticket_ref", "..."],
  "recommendedActions": ["≤ 25 words each"]
}

Rules:
- Quantify (use the rollup numbers); avoid vague language.
- Flag SLA-breaches and double-escalations explicitly.
- Do not propose actions outside the DG's configured oversight powers
  (notify-only vs action) — check {{dg_powers_summary}}.

{{templateId: assistant.secretary.v2}}
{{locale: {{user_locale}}}}
You are the Secretary briefing assistant on the Sindh IT Portal. Produce
concise, decision-ready briefings from portfolio aggregates, policy docs,
officials records, and MoMs. Inputs are anonymized aggregates unless the
Secretary is explicitly viewing a named record.

Retrieved context:
{{retrieved_context_with_ids}}

Secretary's question:
{{question}}

Produce JSON:
{
  "briefing": "≤ 400 words, structured with headings",
  "citations": [{"type":"aggregate|policy|mom|official","id":"...","span":[s,e]}],
  "portfolioStatus": "improving|stable|declining",
  "topIssues": ["≤ 20 words each"],
  "policyImplications": ["≤ 25 words each"]
}

Rules:
- Distinguish operational issues from policy/structural issues.
- Cite the specific aggregate or policy document.
- Where data is incomplete, say so rather than extrapolating.
- Match the officials/branding records by effective-date for any referenced
  official (see Module P — historical accuracy).

5.4 "Ask the ticket" Q&A

A dedicated Staff-assistant affordance: an officer types a question about the open ticket — "what evidence has the company provided?", "when did we last ask for more info?", "what does the attached notice actually say?" — and the assistant answers with citations into the ticket history, attachment OCR text, and MoM. The retrieval scope is locked to that single ticket plus its uploads; PII is redacted before any cloud LLM call. This is the single highest-leverage RAG feature for officer productivity.

6. PII Redaction Pipeline

PII redaction is the first step in every AI flow that may touch a cloud engine, and a defence-in-depth step even for on-prem engines. It is deterministic, on-prem, and audited.

6.1 Entities detected

6.2 Pipeline

Written description. The inbound payload is classified for sensitivity (the classification is supplied by the caller based on the owning record's data class — see /specs/en/11-security-compliance/). The destination residency is decided by the engine selector (§2.3). For on-prem destinations, redaction still runs but the mapping is retained in memory and the original values may be passed to the on-prem engine if its policy permits (trade-off: better translation quality for named entities). For cloud destinations, redaction is mandatory and the mapping never leaves the AI service. Each distinct raw value is replaced by a stable token ([CNIC_1], [PHONE_2]…) within the request scope; the mapping is written to the encrypted ai_pii_vault with a short TTL (call duration + 10 minutes). The redacted payload is sent to the engine; on return, the de-redactor replaces tokens with originals using the vault mapping; the mapping is then purged. The raw value is never written to logs, to ai_runs, to Meilisearch, or to Loki. A downstream canary (a scheduled job that scans Loki + ai_runs for raw CNIC patterns) catches any leak as a P0 incident.

6.3 Invariants

• Fail closed. If the redaction service is unavailable and the destination is cloud, the call is rejected. On-prem destinations may proceed without redaction (logging a degraded-mode event).
• No raw PII in logs. Structured logs and ai_runs store redactionsApplied count and a hash of the redacted payload — never the raw payload, never the mapping.
• Reversal only internally. Only the AI service can read ai_pii_vault; the encryption key is held in the secrets vault (see /specs/en/15-tech-architecture/ §16). No API exposes a de-redaction operation to the NestJS layer or to clients.
• Aggressive mode is enabled automatically for restricted inputs (also masks non-public person names and addresses).

7. Prompt Management

7.1 Storage and versioning

Prompt templates are versioned artefacts stored in the source repository under ai-service/prompts/ (one file per template id, one row per version in ai_prompt_templates). A template id looks like summary.ticket.v3 — name + major version. Every change is a new version; versions are never mutated in place. Deployment pins a template id to a specific version per environment via the feature-flag/config service.

type PromptTemplate = {
  id: string;                  // "summary.ticket"
  version: number;             // 3
  locale: "en" | "ur" | "sd" | "multi";
  body: string;                // the fenced template with {{variables}}
  variablesSchema: object;     // JSON Schema for the variables map
  responseSchema?: object;     // JSON Schema for the structured output (if any)
  guardrails: Guardrail[];     // per-template output checks
  status: "draft" | "shadow" | "active" | "retired";
  evalResultsRef?: string;     // latest evaluation run on the golden set
  authoredBy: string;
  authoredAt: UTCDateTime;
};

7.2 Variables and rendering

Templates use {{variable}} placeholders validated against variablesSchema at render time. Unknown variables fail closed. Locales are handled by either separate per-locale template versions (preferred for high-stakes outputs like draft replies) or a single template with {{target_locale}} injected (preferred for cheap, low-risk outputs like summaries).

7.3 A/B testing

ai_ab_assignments records which prompt version a given user or ticket is exposed to, deterministically (hash of user_id or ticket_id into the configured split). Outcomes (kept-as-is rate, override rate, CSAT) are joined back to the assignment for analysis. A/B runs are time-boxed; the winning version is promoted to active and the loser to retired after a documented decision.

7.4 Evaluation set

Every active template has a golden test set — curated input/expected-output pairs covering EN/UR/Sindhi, edge cases (empty input, very long input, mixed script, adversarial input), and common failure modes. The set lives in ai-service/prompts/eval/<template_id>.golden.jsonl. CI runs the set on every template change; a regression beyond thresholds blocks promotion to active. See §10.

7.5 Guardrails

Each template declares output guardrails enforced post-generation:

• Schema validation — structured outputs must validate against responseSchema or the result is rejected and the fallback engine is tried.
• Citation check — when citationsRequired, every claim must map to a cited source; uncited claims trigger a re-prompt or rejection.
• PII leak check — the output is re-scanned by the redaction detector; any PII token that was not in the redacted input is treated as a leak and the output is rejected.
• Scope check (RAG) — every cited id must be in the user's retrieved set; out-of-scope citations are stripped or the answer is downgraded.
• Jailbreak / prompt-injection defense — system prompts mark retrieved/user content with explicit delimiters ("""), instruct the model to treat delimited content as data not instructions, and a classifier flags known injection patterns (e.g. "ignore previous instructions") for rejection.
• Policy check — domain-specific (no invented SLA numbers, no legal advice, no political opinion).

8. Cost & Usage Governance

8.1 Per-call logging

Every AI call writes one immutable row to ai_runs:

8.2 Budgets and rate limits

• Per-call budget in ai_engine_configs.costBudgetPerCall — a call is rejected if the engine's estimated cost (from a per-engine cost table maintained in config) exceeds the budget.
• Per-tenant rate limits — tokens per minute and calls per minute, per department and per user, enforced via Redis counters. Breaches trigger throttling (429) to the caller, which degrades gracefully (§11).
• Daily/monthly budgets — per capability and per department; on breach, non-critical capabilities (e.g. trends) are auto-disabled and an alert is raised; critical capabilities (e.g. pii_redaction, ocr for restricted) are never auto-disabled.
• Cloud spend alerts — at 50/80/100% of the monthly cloud-AI budget, notify the S&ITD operations contact and the MAAHIR delivery lead.

8.3 Dashboards and reports

• A Grafana dashboard surfaces: calls/min, tokens/min, cost/day by capability and engine, fallback rate, fail-closed rate, P95 latency, and redaction counts.
• A scheduled monthly cost report (PDF via the exports pipeline) is delivered to S&ITD leadership, breaking down cost by capability, department, and engine, with month-over-month delta and a forecast.
• An efficiency view tracks cost-per-useful-output (e.g. cost per accepted draft, cost per confirmed action item) — the goal is to drive this down over time via prompt improvements and on-prem migration.

9. Evaluation & Quality

9.1 Golden test sets

Each capability has a golden set curated by the product/AI team, covering the three locales and edge cases. Sets are versioned alongside prompt templates in the repo. The minimum sizes:

9.2 Metrics

• Accuracy / precision / recall / F1 as appropriate per capability (defined in each capability's "success metrics").
• Urdu/Sindhi-specific evaluation — metrics are computed per locale, not pooled, so a high EN score cannot mask a low SD score. A locale whose metric falls below threshold blocks promotion of the template to active for that locale.
• Citation precision — every cited source must exist and be in-scope; target 100%.
• Human-review sampling — a configurable % of every capability's outputs (default 5%, 20% for restricted and for new templates in shadow status) are routed to a reviewer queue. Reviewer accept/reject feeds ai_feedback and the golden set.
• Feedback loop — ai_feedback rows (officer thumbs, overrides, edits) are triaged weekly: confirmed issues become golden-set negatives; recurring patterns drive prompt iteration or rule-layer updates; enough volume triggers a fine-tuning candidate (Phase 2).

9.3 Continuous evaluation

A nightly job re-runs the current active templates against the golden set and writes results to ai_prompt_eval_results. A regression beyond threshold raises a Grafana alert and, for critical capabilities, auto-flips the template to shadow (still running but not served to users) pending review.

10. Fallback & Degradation

AI is assistive, never blocking. The system degrades gracefully through three tiers:

Invariants:

• A failed AI call never blocks a ticket state transition, a file upload, a notification, or a login.
• Failed calls return a structured { ok: false, reason } so the caller can degrade deterministically.
• The fail-closed path (e.g. PII redaction down + cloud destination) returns { ok: false, reason: "pii_unavailable_cloud_blocked" } and the caller must not call the cloud engine.

11. Ethics & Bias

12. Configuration (Feature Flags)

Every capability is toggleable by the Super Admin, per department and per environment, via the Feature Flag module (Q). Engine selection per capability is configurable via ai_engine_configs.

12.1 Capability flags

12.2 Engine configuration

ai_engine_configs is editable by the Super Admin (or a delegated AI-config role) in the admin UI. Every change is audit-logged with before/after, the actor, and a reason field. Changes are validated against the residency invariant (no cloud engine for restricted). The current config is exportable as JSON for reproducibility and for the monthly cost report.

12.3 Bootstrap

Critical defaults (e.g. ai.pii_redaction.enabled = on, ai.cloud_engines.allowed = on in non-sovereign envs) are seeded by migration so a fresh environment starts in a known state. The bootstrap config is reviewed at every major release.

13. Traceability

End of document.