Keystone Applied Intelligence Governed RAG for Regulated Industries

Governed RAG for Regulated Industries

Evidence-backed answers. Fail-closed refusal. On your hardware.

Every response scored for factual consistency. Query-time role-based access control. Hash-chained, tamper-evident audit trails. Operational deployment and public demo with quantitative evaluation results.

demo.getkeystone.ai (live demo), github.com/getkeystone (source).

Demo access demo.getkeystone.ai. Log in as operator1 / demo123. Try: "What atmospheric testing is required before entering a confined space?"
The Problem

Safety-critical knowledge is scattered, unfindable, and unauditable

Hundreds of SOPs live across SharePoint folders, shared drives, and filing cabinets. When a regulator asks which version of a confined space entry procedure was in effect on the day of an incident, finding the right document takes hours. When an internal audit asks who accessed a specific safety procedure last Tuesday, the honest answer is often: we don't know.

Generic AI tools make this worse. They answer with confidence whether or not the evidence supports it, send your operational documents to third-party cloud providers, and produce no audit trail of what was accessed or by whom.

Keystone was built to address this.

Architecture

What the system enforces

Keystone enforces architectural properties simultaneously. These are not prompt instructions or configuration options. They are structural constraints in the retrieval pipeline, database schema, and API layer.

Evidence-backed answers with citations to source documents
Every answer cites the specific document and section it draws from. Operators see the source material alongside the synthesized response. No black box. No unsupported assertions.
Refuses to answer when evidence is insufficient or query is out of corpus scope
When retrieved evidence does not support an answer, or when a query is outside the corpus topic scope, the system refuses rather than generating an unsupported response. Both failure modes are caught with separate, audited refusal reasons.
Access control enforced at the retrieval layer, not the UI
User roles are checked before retrieval runs. Unauthorized documents are excluded at the database level and never enter the query context. Role-restricted content cannot be surfaced by reformulating a query.
Tamper-evident audit logging
Every query writes an audit record: user identity, role at query time, documents retrieved, permission decision, and the answer generated. Records are hash-chained with an INSERT-only database role. The application cannot modify or delete them after the fact.
Every response scored for factual consistency
A dedicated hallucination detection model independently verifies that every generated answer is supported by the retrieved evidence. Answers that score below threshold are automatically withheld. This creates a double safety net: the system refuses when evidence is insufficient, and independently verifies when evidence is present.
Runs entirely on your hardware. Air-gap capable.
No external API calls for core operation. No data sent to cloud providers. Deployable on customer infrastructure without internet access. Your documents do not leave your network.
Governed learning loop: field feedback improves procedures through auditable review
When operators flag an answer as unhelpful, a review task is automatically created for a custodian. When a custodian publishes an improved procedure version, re-queries return the improved answer. The complete chain (query, feedback, review, publication) is reconstructable from an append-only audit trail.
Document version tracking: point-in-time procedure retrieval for audit scenarios
Every procedure version is tracked with effective dates. The system can answer "which version of this procedure was active on the day of the incident?" The database enforces at-most-one-active-version per document. Superseded versions are retained, never deleted.
Separation of duties: enforced at the system level, not just policy
The person who flags a knowledge gap cannot declare it resolved. The person who creates a procedure version cannot approve it for publication. Both constraints are enforced at the API layer with 403 responses, not documented in a policy that can be bypassed.
How It Works

Your documents. Your hardware. Cited answers and a full audit trail.

01
Load your documents
SOPs, regulations, and safety procedures are ingested and indexed on your hardware. Documents are assigned access control metadata specifying which roles are permitted to retrieve each one. Nothing leaves your network at any stage.
02
Ask in plain language
Users query the corpus the way they would ask a colleague. No Boolean syntax, no folder navigation, no knowledge of where documents are stored.
03
Access is filtered at query time
Before retrieval runs, the system checks the user's role. Documents the user is not permitted to access are excluded at the database level. This happens at every query, not once at login.
04
Get cited answers with an audit trail
Every response includes citations to specific source documents and sections. Every query generates a tamper-evident log record. If the evidence is insufficient to answer, the system says so rather than guessing.
Continuous Improvement

Closes the loop between written procedure and field reality

After every answer, workers can flag what was useful, what was missing, and what was wrong. Those signals feed a governed review process. When a procedure owner validates a change, the next person who asks the same question gets a better answer. Your procedures improve because your people use them.

Live demo at demo.getkeystone.ai (operator1 / demo123). Eval baseline (KDAT-001B, 2026-04-11): P@1=0.75, MRR=0.79. Adversarial ACL: 8/8 blocked, 0 leaks. Fail-closed: 5/6 (83%). Audit chain intact.

Evaluation

Quantitative evaluation baseline

All capability claims are backed by evaluation evidence. KDAT-001B (2026-04-11): governed retrieval, 53-document Alberta OHS corpus, 2,674 chunks. KDAT-002D (2026-05-20): governed agent extension, 135-document corpus (Alberta OHS plus supplementary regulatory material), 23,684 chunks.

Retrieval Quality
  • Precision at 1 (P@1): 0.75
  • Mean Reciprocal Rank (MRR): 0.79
  • Corpus: 53 documents, 2,674 chunks
  • Hybrid retrieval: pgvector + full-text search
Security and Safety
  • Adversarial ACL testing: 8/8 blocked, 0 leaks
  • Fail-closed accuracy: 5/6 (83%)
  • Audit chain: intact, immutable
  • INSERT-only DB role enforced
FC-005 remediation deployed 2026-05-17 (v0.5.2-fc005). Pre-retrieval domain scope guard refusing out-of-corpus queries (emissions regulations, workers comp, federal tax, IT procurement). Closes the KDAT-001B FC-005 failure where a TIER greenhouse gas query returned Part 36 mine gas chunks via embedding overlap. Validated manually against the probe set. Full taxonomy-based two-stage domain gate is not yet implemented; partial domain-scope remediation (FC-005, 2026-05-17) is in place. See eval ledger.
KDAT-002D (2026-05-20)
  • Governed agent extension
  • 186 eval cases, 12 categories
  • 558 executions, 0 failures
  • 153 strict pass, 33 characterization
  • Spec: KDAT-002-SPEC v1.2
Agent Governance Controls
  • Severity-tier HITL routing
  • Per-step evidence gating
  • HMAC action audit chain
  • Eval identified 4 system bugs; all fixed and re-verified
  • Failing run (KDAT-002C) published
Not Claimed
  • Enterprise HA or disaster recovery
  • Multi-node or distributed deployment
  • OIDC/SAML production identity integration
  • Third-party penetration testing
  • WCAG accessibility compliance
Hackathon Demo

Hackathon Demo

Generative UI for governed agent actions, built at AI Tinkerers Global Hackathon May 2026.

governed-incident-agent on GitHub

About

Built from operational and infrastructure experience

Arnaldo Sepulveda built Keystone Applied Intelligence after nearly 13 years delivering and supporting enterprise platforms at Genesys for regulated and public sector customers. Those environments required production reliability, security controls, and the ability to prove what happened, when, and who authorized it.

That operational background is why Keystone is built the way it is. Every design decision maps to a documented requirement. Every capability claim maps to a demonstrated proof artifact. No overclaims, no roadmap presented as current capability.

Based in Canada.

More about Arnaldo: arnaldosepulveda.com

Public demonstration system at demo.getkeystone.ai (53 Alberta OHS documents, 2,674 indexed chunks). Eval baseline (KDAT-001B, 2026-04-11): P@1=0.75, MRR=0.79. Adversarial ACL: 8/8 blocked, 0 leaks. Fail-closed: 5/6 (83%) — FC-005 remediated 2026-05-17.
Stack

Built with

Python, FastAPI, PostgreSQL 16 with pgvector, Ollama (nomic-embed-text, qwen2.5:7b-instruct), React/TypeScript/Tailwind, Docker Compose, Caddy, Cloudflare Tunnels. No cloud dependency for core operation.

Source and evaluation evidence: github.com/getkeystone