Knowledge Base & SOP

The authoritative specification for the Sindh IT Portal — Facilitation Desk (SITP) Knowledge Base: content types, versioned SOPs, the AI semantic search layer, deflection tracking, the service catalog, the authoring/publishing workflow, content governance, multilingual content, the forms library, taxonomies, in-flow article suggestions, and KB analytics.

1. Purpose & Scope

This document is the single source of truth for how the SITP Knowledge Base (KB) is authored, versioned, searched, surfaced, governed, and measured. The KB is Module J of the Portal (_context.md §4) and serves three audiences with one shared corpus:

1. The public — IT companies, freelancers, startups, and citizens seeking answers before or instead of filing a ticket.
2. Company Representatives — filers and requesters who get article suggestions while composing a ticket.
3. S&ITD and department staff — officers, facilitators, triage, and oversight roles who get the same article suggestions while replying, for consistent, evidence-backed resolutions.

1.1 Why the KB exists

The KB exists for three locked reasons (Module J, _context.md §4):

• Self-service. Visitors and companies find authoritative answers in EN, Urdu, and Sindhi without needing to file a ticket, register, or wait for business hours.
• Deflection. A meaningful share of would-be tickets are answered by a single well-placed article surfaced at search time or at filing time, reducing avoidable load on departments and on S&ITD triage.
• Consistent resolutions. Officers across departments cite the same SOPs, forms, and legal references when replying, so two tickets on the same issue receive the same answer.

1.2 What is in scope

• Content types, the taxonomy, versioned SOPs, the service catalog, and the forms library.
• The AI semantic search layer (Meilisearch hybrid index, on-prem embeddings, multilingual matching, relevance ranking, "did you mean", filters).
• Deflection tracking and the "was this helpful" feedback loop.
• In-flow article suggestions to officers (reply drafting) and to companies (filing/compose time).
• The authoring/publishing workflow, roles, scheduled publish/expiry, and content governance.
• Multilingual content production (AI-translate + human review, glossary-driven).
• KB analytics and the functional/traceability mapping.

1.3 What is out of scope

• The full AI engine abstraction, prompt versioning, and PII-redaction pipeline are specified in /specs/en/07-ai-ocr-spec/ and are referenced, not re-specified, here.
• The public chatbot's conversational behaviour lives in /specs/en/07-ai-ocr-spec/ §4.8; the KB only supplies its retrieval corpus.
• RBAC detail (who may author/publish) is summarized in §11 and fully specified in /specs/en/04-roles-permissions/.
• The data model tables are summarized in §17; full schemas live in /specs/en/05-data-model/.

1.4 KB principles

Every KB behaviour obeys the following principles, referenced by ID (KB-1 … KB-8) in acceptance criteria throughout.

2. Content Types

The KB holds seven content types. Each type has its own authoring fields, lifecycle, freshness window, and search behaviour. The kb_content_type enum on every KB item is fixed.

Common fields on every KB item (in addition to type-specific fields): id, title (per locale), summary (per locale, ≤ 60 words, shown in search results), body (per locale, Markdown), content_type, owning_department, categories[], tags[], audience[], locale_status{en,ur,sd}, version, status (draft/in_review/scheduled/published/retired), effective_from, expires_at, last_reviewed_at, freshness_window_days, helpful_votes, unhelpful_votes, view_count, deflection_count, created_by, updated_by.

3. Taxonomy

The KB taxonomy is three-layered and shared across all content types. It is also the spine of the service catalog (§5), the in-flow suggestions (§7), and the public FAQ/help center.

Rules.

• The category tree mirrors the ticket category tree (see /specs/en/06-ticket-workflow/ §4.1). A content item's category must be a leaf in that tree, so an article can be unambiguously linked to the tickets it concerns.
• Tags are drawn from a controlled vocabulary maintained by S&ITD KB editors; free-text tags are normalized on save and flagged for review if novel.
• Audience scoping is enforced at the search layer (§6) and at retrieval: an item whose audience does not include the requester's role is never returned, never embedded into a suggestion, and never counted toward deflection for that requester.

4. Versioned SOPs

SOPs are the most consequential content type: an officer cites one when replying, and a company relies on one when deciding whether to file. They therefore carry the strictest versioning and approval discipline.

4.1 What makes an SOP versioned

Every sop item is versioned by construction (KB-5, FR-KB-002). A published change never overwrites; it creates a new version row, and every prior version remains archived, retrievable, and citable. An SOP row carries:

4.2 SOP versioning matrix

4.3 Change log

Every version carries a structured change_log entry appended (never overwritten). The change log is itself rendered on the SOP page and is exportable for audit.

4.4 Approval workflow

A substantive SOP change requires a three-role approval (KB-3, §11.2):

4.5 Linking SOPs to ticket categories

Each SOP declares linked_categories[] drawn from the same tree the ticketing module uses (/specs/en/06-ticket-workflow/ §4.1). This link is what powers in-flow suggestions (§7) and deflection attribution (§9): when an officer or filer is on a ticket in category SRB/Refund, the system retrieves the SOP(s) linked to that category — and to its ancestors — ranked by relevance.

Cardinality. A category may have 0–N linked SOPs; an SOP may be linked to 1–N categories. When zero SOPs are linked, the suggestion layer falls back to semantic search over the whole KB scoped by department.

5. Service Catalog

The service catalog is the public, filterable directory of what the Portal handles, per department and category. It is published on the public site (/specs/en/02-functional-reqs/ FR-PUB-007, FR-KB-007) and is the primary entry point for both self-service and filing.

5.1 Schema

Each catalog entry is one row keyed by (department_code, category_code). The schema below is the canonical contract between the catalog, the KB, the intake form, and analytics.

5.2 Per-department entries

Each Government of Sindh department on the Portal owns its catalog entries. The catalog is filterable by department and by category (FR-KB-007), and each entry links to its SOPs, forms, and SLA expectation. A representative slice (illustrative, not exhaustive — the full catalog is maintained in kb_service_catalog):

5.3 Freshness and SLA mirroring

• The expected_resolution_time shown publicly is derived from sla_definitions (KB-6). When an SLA row changes, the dependent catalog entries are flagged for editorial review so the public number never silently diverges from the operational one.
• Catalog entries inherit the freshness cadence of their parent department and surface on the content-governance dashboard (§12).

6. AI Semantic Search

Search is the KB's front door. It powers the public help center (FR-PUB-003), the public chatbot's retrieval corpus (/specs/en/07-ai-ocr-spec/ §4.8), the in-flow suggestions (§7), and the staff assistant (§5 of the AI spec). Search is multilingual by construction (KB-1) and respects scope (KB-7).

6.1 Index: Meilisearch hybrid (lexical + vector)

The search index is Meilisearch (locked in _context.md §3, AI spec §5.2). Meilisearch provides multilingual lexical tokenization (good Urdu/Sindhi handling) and hybrid vector similarity in one index. No pgvector dependency exists anywhere in the stack (KB-2).

• Lexical layer — multilingual tokenization with locale-aware analyzers for EN, UR (Nastaliq), and SD (Naskh). Handles digit-form normalization (Western ↔ Arabic-Indic) at index and query time.
• Vector layer — each published KB item is embedded by an on-prem embedding model (the AI service's LLMClient.embed interface; embeddings never leave the boundary). Embeddings are recomputed on publish and on substantive edit.
• Canonical source — MariaDB tables kb_items, kb_item_versions, kb_service_catalog, kb_forms are the source of truth. Meilisearch is a derived index rebuilt from them; a re-index worker can rebuild Meilisearch from MariaDB at any time.

6.2 Query flow

Written description. A query arrives carrying its text, the user's locale, role, and scope filters (department/category if known, audience, content type). The gatekeeper first checks the semantic-search feature flag: if disabled, the query degrades to lexical-only search and never errors (KB-8). When enabled, the query text is embedded by the on-prem embedding model (embeddings never leave the boundary) and tokenized by the locale-aware analyzer, which normalizes Western ↔ Arabic-Indic digits and handles Nastaliq/Naskh shaping. A "did you mean" pass generates spell- and script-normalized candidate forms (e.g. catches a Roman-Urdu query against a Nastaliq index, or a transliterated Sindhi term). Meilisearch performs hybrid retrieval combining lexical matching and vector similarity; results are then filtered by scope (department, category, audience, locale, content_type) so an officer never receives an article outside their published SOPs and a public visitor never receives staff-only content. Top-K candidates are reranked by a cross-encoder or LLM rerank pass for precision. Each result carries its signals (type, department, freshness, view count, helpful ratio, canonical flag) which feed the ranking and the UI. The search event is logged with query, locale, result count, has-results flag, and session id — feeding the analytics in §13 (notably the search-no-results rate).

6.3 Relevance ranking

Ranking is a weighted blend. Weights are configurable per surface (public help center vs officer suggestion) and tuned from click-through and helpfulness signals (§10).

6.4 "Did you mean"

For any query returning weak or zero results, the search layer proposes alternative forms using:

• Spell correction against the controlled vocabulary and the KB corpus.
• Script normalization (Roman Urdu ↔ Nastaliq; transliterated Sindhi).
• Synonym/term expansion from _glossary.md (e.g. "NTN" ↔ "National Tax Number" ↔ "قومی ٹیکس نمبر").
• Category-suggestion when the query matches a service-catalog entry directly ("how to file labour complaint" → the LBR catalog entry + the top SOP).

"Did you mean" is rendered above the results and is itself a logged event so synonym gaps can be added to the glossary.

6.5 Filters

The public help center and the officer suggestion surface both expose filters. Default filter facets: department, category, content type (how-to / FAQ / SOP / form / guide / legal / video), audience, and language. Officer-facing search additionally exposes a "my department only" toggle (KB-7). Filters are reflected in the search log and in analytics.

7. Embeddings in the Ticket Flow

The same KB corpus and embedding index that serve search are reused inside the ticket flow to surface relevant articles at two high-leverage moments. This is the operational expression of KB-7 and aligns with the smart-filing-assistant (US-TKT-004, FR-TKT-021) and draft-reply (US-AI-004, FR-AI-005) decisions.

7.1 To the company while filing (deflection-side)

While a Company Representative composes a ticket, the title and description are embedded on-prem and matched against the KB scoped to the likely department/category (which itself comes from the AI routing suggestion, /specs/en/07-ai-ocr-spec/ §4.3).

• What is shown: up to 3 ranked articles (SOPs, FAQs, how-tos) with title, summary, and a "this answers my issue" action.
• Deflection action: if the representative confirms an article answers their issue, the draft is discarded and a deflection event is recorded (§9). No ticket is created.
• Proceed: if none answer the issue, the representative continues filing; the fact that articles were offered (and not accepted) is recorded for analytics.
• Scope: only articles with audience including public or company are eligible.

7.2 To the officer while replying (consistency-side)

While an assigned officer drafts a reply, the ticket's category + history are used to retrieve the relevant SOP(s) and FAQs. This is layered onto the draft-reply capability (/specs/en/07-ai-ocr-spec/ §4.5): the retrieved KB chunks become the cited context the draft is grounded in.

• What is shown: the SOP(s) linked to the ticket's category (§4.5), plus semantically similar FAQs/guides, each with a "cite in reply" action.
• Citation: when the officer cites an article, the citation (type: kb, id) is attached to the sent message and to any AI draft (AI spec §2.1 Citation).
• Scope: scoped by the officer's department and the ticket's category; cross-department articles appear only if explicitly cross-linked.
• Consistency loop: cited articles and their acceptance feed the helpfulness and consistency analytics (§13).

7.3 Failure modes

• If the embedding model is unavailable, both surfaces degrade to lexical search over the KB (KB-8).
• If the whole search index is unavailable, the suggestion surfaces hide gracefully; filing and replying proceed without suggestions.

8. Forms Library

The forms library is the always-current home for downloadable government forms referenced by the service catalog and SOPs. It satisfies FR-KB-004 (downloadable without login) and KB-4 (always-current links).

Always-current links. The canonical URL (/kb/forms/<form_id>) resolves to the latest published version regardless of how many times the form is updated. External references — on other government sites, in printed onboarding packs, in SOPs — never break. A retired form keeps its URL, returning a 410 Gone with a "superseded by" notice and a link to the successor.

Versioning and audit. Re-uploading a form creates a new version (KB-5); the prior version is archived and downloadable by authorized staff. The forms librarian role and the owning department are notified when a form's freshness window expires.

9. Deflection Tracking

Deflection is the KB's measurable contribution to reducing avoidable tickets. A deflection event is recorded when a user finds the answer in the KB and does not proceed to file a ticket they otherwise would have. Deflection is a Should capability (FR-KB-006) and feeds the analytics engagement family (KPI-COM-012).

9.1 Deflection flow

Written description. A visitor or company representative searches the KB or begins composing a draft ticket. The system offers ranked articles scoped to the likely department and category. When the user opens an article (or the public chatbot answers from the KB corpus) and then, within a configurable attribution window, does not file a ticket, a deflection event is recorded and attributed to the article(s) offered, the original query, the locale, and the inferred department/category. Two attribution paths are recognized: (a) the user explicitly marks the article "was this helpful" → Yes and does not file within the window; and (b) the chatbot resolves a session without handoff to a human (/specs/en/07-ai-ocr-spec/ §4.8, KPI-AIM-008). If the user proceeds to file despite the offer, an "offer-shown, not-accepted" event is still logged (without counting as a deflection) so the content-improvement loop can detect articles that are surfacing but not landing. Deflection events are PII-free at the event level (no ticket body, no personal identifier beyond an opaque session/company-band) and feed the analytics engagement family (KPI-COM-012) and the KB analytics in §13.

9.2 Attribution window and rules

• Attribution window — default 24 hours from the offer/answer to "did not file". Configurable per environment.
• Attribution targets — the article(s) offered, the query, the locale, the inferred department/category, and the surface (help center / filing-flow / chatbot).
• No double-counting — a session is deflected at most once, attributed to the highest-ranked article marked helpful (or, for the chatbot path, the cited article).
• Conservative by default — only explicit "helpful → not filed" and "chatbot resolved" paths count toward the headline deflection rate. "Offer-shown, not-accepted, then not filed" is reported separately as a potential deflection, not summed into the headline.

9.3 Metrics

Deflection feeds these metrics (detailed in /specs/en/17-analytics-kpis/):

10. "Was this helpful" Feedback Loop

Every public-facing KB item carries a "was this helpful" control (Yes / No, with an optional free-text reason). This is FR-KB-005 and is the primary content-improvement signal alongside search analytics.

10.1 Capture

• Per-article — Yes/No vote on the article page; optional reason text. One vote per session per article to reduce noise.
• On suggestion surfaces — when an article is offered in-flow (§7) and the user acts on it (cites it in a reply, or marks "this answers my issue"), that interaction is captured as an implicit helpful/unhelpful signal.
• On chatbot answers — the chatbot's thumbs-up/down (ai_feedback) is linked back to the cited KB article.

10.2 Aggregation

Votes are aggregated per article, per locale, and per category, and are visible to KB Authors and Editors on the authoring dashboard. Aggregations include: helpful count, unhelpful count, helpful ratio, trend over the freshness window, and the top free-text reasons (clustered).

10.3 Improvement loop

The feedback loop closes into authoring and governance:

Every content-improvement action feeds the change log of the affected item (§4.3) so the loop is auditable end-to-end.

11. Authoring & Publishing Workflow

Authoring is a gated, role-based workflow with scheduled publish and expiry. It satisfies FR-KB-001 and FR-KB-002.

11.1 Roles

Full RBAC, including granular per-permission overrides, is in /specs/en/04-roles-permissions/.

11.2 Authoring + publish workflow

Written description. An Author creates an item as a Draft and may autosave freely. On submit, the item enters InReview and a KB Editor is notified. The Editor either requests revisions (back to Draft) or approves (to Approved). From Approved, the Publisher either publishes immediately (Published) or schedules publication at a future effective_from (Scheduled, auto-publishing when the date arrives — §11.4). A Published item can be retired (Retired, with expires_at and an optional successor), or opened for a substantive change, which creates a new version whose draft state is Draft while the prior version remains the current published one until the new version is published (KB-5). A retired item can be restored, which publishes a new version. Terminal states are Published (current) and Retired. Every transition writes an audit event and, on publish, a version/change-log row and a re-index job.

11.3 Multilingual readiness gate

Before an item may move Approved → Published in a given locale, that locale must be marked ready: its translation exists, has passed human review (§14), and uses approved _glossary.md terms. An item may be Published in EN while still Draft in UR/SD; the public surfaces show only the locales that are published for that item, with a visible "translation in progress" note on the others rather than a fallback to English content.

11.4 Scheduled publish and expiry

• Scheduled publish — a Publisher may set effective_from to a future date; the item moves to Scheduled and auto-publishes at that timestamp. Useful for policy changes aligned to a notification date.
• Expiry — any published item may carry an expires_at; on that date it auto-retires (with a successor if declared). Expiry is typically used for time-bound circulars and notices rather than SOPs.
• Time zones — all scheduling is stored in UTC and rendered in Asia/Karachi (PKT) with Hijri dates per _context.md §2.

12. Content Governance

Governance keeps the corpus accurate, owned, and fresh. It satisfies FR-KB-008 (outdated-content flagging).

12.1 Ownership

Every KB item has an owning department and, within it, an accountable owner (a named KB Publisher). Ownership determines who is notified for review, who can publish, and whose dashboard the item appears on. Items without a current owner are flagged "orphaned" and surface on the Super Admin governance view.

12.2 Review cadence

Each content type has a default freshness window (§2). On window expiry, the item is flagged Needs Review and its owner is notified. The cadence is configurable per department and per content type.

12.3 Stale-content flags

An item is flagged stale (Needs Review) when any of:

• Its freshness window has elapsed since last_reviewed_at.
• A linked SLA row (sla_definitions) has changed since the item was last reviewed.
• A linked statute/notification (for legal items) has a recorded amendment after last_reviewed_at.
• Its helpful ratio has dropped below threshold for two consecutive windows (§10.3).
• Its owning department has been reorganized (category tree change) and the item's category no longer resolves.

Stale items appear on the owner's and Editor's review queues, are visually marked on the public page ("under review — content may be updated"), and are excluded from the top of search ranking until reviewed.

13. Analytics

KB analytics are part of the engagement/comms metric family in /specs/en/17-analytics-kpis/ and are visible on the Super Admin and S&ITD dashboards.

13.1 KB metrics

13.2 Dashboards

KB metrics surface on: the Super Admin dashboard (full KB family + freshness + gaps); the S&ITD Editor dashboard (review queue, stale items, feedback clusters, translation backlog); and the per-department Publisher dashboard (their items' views, helpfulness, deflection, freshness). All obey role scoping (AP-3 in /specs/en/17-analytics-kpis/).

13.3 Public transparency

Aggregate, anonymized KB metrics (total deflections, overall helpfulness, search-success rate) are eligible for publication on the public transparency dashboard (/specs/en/02-functional-reqs/ FR-PUB-008), sourced from anl_pub_* with small-cell suppression, never from live queries.

14. Multilingual Content

Multilingual content is produced AI-first with mandatory human review for published material, driven by _glossary.md. This satisfies KB-1 and aligns with the translation capability (/specs/en/07-ai-ocr-spec/ §4.6).

14.1 Pipeline

1. Author in EN — the master locale. The Author writes the EN body and marks terms that must use the glossary mapping.
2. AI pre-translation — the translation capability (glossary-injected LLM, on-prem Qwen 2.5 preferred for UR/SD per AI spec §3.1) produces UR and SD drafts.
3. Human review — a bilingual reviewer edits each locale. Citizen-facing legal text, RTI responses, and official-letter-domain content always require human review regardless of engine (AI spec §4.6).
4. Glossary enforcement — terms are checked against _glossary.md; deviations are flagged. Novel terms are added to the glossary on publish.
5. Locale readiness — each locale is independently marked ready (§11.3); an item may publish in EN before UR/SD.

14.2 Quality

• Translation quality is measured per locale (AI spec §9.2) so a strong EN score cannot mask a weak SD score.
• A locale whose helpful ratio is markedly lower than EN's (§10.3) is flagged for translation review even within its freshness window.
• RTL rendering for UR (Noto Nastaliq Urdu) and SD (Noto Naskh Arabic) is enforced per _context.md §2 and /specs/en/09-i18n-localization/.

15. Functional Requirements Mapping

This document realizes the KB functional requirements in /specs/en/02-functional-reqs/ §KB. Cross-references:

Additional KB behaviours specified here that trace to user stories in /specs/en/01-prd/:

16. Feature Flags

KB capabilities are toggleable by the Super Admin per environment/department via the Feature Flag module (Q).

Engine selection for the embedding and rerank steps is configured via ai_engine_configs (AI spec §2.2); embeddings are always on-prem.

17. Data Model (Summary)

KB-specific tables in MariaDB (full schemas in /specs/en/05-data-model/). Meilisearch is a derived index rebuilt from these.

The embedding vectors are held in the AI service's embedding store and in Meilisearch; ai_embeddings (AI spec §2.4) is the source of truth for re-indexing. No pgvector.

18. Traceability

End of document.