Rule Agent & AI Governance Dashboard: Real-Time LLM Compliance Auditing

The Problem: LLMs Without Governance Are Liability

When an enterprise deploys a large language model for internal decision support, every response carries regulatory, reputational, and strategic risk. Traditional LLM deployments have no mechanism to enforce organizational policies on outputs — they rely entirely on prompt engineering and hope.

ArcaQ's answer is the Rule Agent: a governance interceptor that runs between the LLM and the user. Before any response reaches the interface, the Rule Agent checks it against a configurable set of DataGovernanceRules stored in the Knowledge Graph, computes a compliance score, and — if the score falls below threshold — automatically rewrites the response.

Every audit event is written as RDF triples to a dedicated audit graph, creating a permanent, SPARQL-queryable record of every governance decision the system has ever made.

The Rule Agent: Architecture

The Rule Agent pipeline executes in five steps for every LLM response:

1. Rule retrieval: Governance rules are loaded from Fuseki with a 300-second TTL cache. Hot path latency is <1ms for cached rules.
2. Compliance scoring: The LLM response is evaluated against each rule. A composite compliance score (0–100%) is computed. Scores ≥80% are high compliance; 50–79% are medium risk; below 50% trigger mandatory correction.
3. Automatic correction: When the score is below threshold, the Rule Agent instructs the LLM to rewrite the response with explicit constraints. Both original and corrected snippets are captured.
4. RDF audit logging: A arcaq:ComplianceViolation node is inserted into <http://arcaq.com/audit> with predicates for timestamp, intent type, score, original snippet, corrected snippet, and number of rules checked.
5. Badge in UI: The chat interface displays a governance badge (🛡️) with a tooltip explaining the rule check outcome, giving users immediate transparency.

The Policy Agent: Managing Governance Rules

Governance rules are not hardcoded. The Policy Agent provides a full CRUD interface for compliance administrators:

• Create new rules with a label, description, severity (low/medium/high/critical), and active/inactive status
• Toggle rules on/off without deleting them — useful for testing new policies before enforcement
• Edit existing rules and flush the Rule Agent cache instantly
• All operations persist as RDF triples in Fuseki — versioned, queryable, auditable

The rule cache (TTL 300s) means new rules are effective within 5 minutes of creation — or immediately if you flush the cache from the Policy Agent dashboard. This design separates governance policy from platform configuration, allowing compliance teams to manage AI behavior without engineering intervention.

The AI Governance Dashboard: Live Violation Monitoring

The Governance Dashboard gives compliance officers a real-time view of AI behavior across the platform. The dashboard is organized in four panels:

• KPI row: Total violations logged, compliance rate (%), number of intent types, and active governance rules — updated on every page load
• Violations by intent: A horizontal bar chart breaking down violations by query intent type (governance, metadata, encyclopedia, etc.), surfacing which AI behaviors are most problematic
• Most triggered rules: Which governance rules are firing most often — helping administrators tune thresholds or refine rule language
• Violations table: A filterable, searchable table of the most recent 50 violations, with click-through to a full detail modal

Filter Bar: Slicing the Audit Log

The violations table includes a multi-dimension filter bar that queries Fuseki in real-time via a dedicated REST endpoint. Compliance analysts can filter by:

• Text search: Full-text search across violation descriptions and original AI response snippets — debounced at 300ms for responsive filtering
• Intent type: Dropdown populated from actual violation data — filter to a specific AI behavior domain
• Score range: High (≥80%), Mid (50–79%), Low (<50%) — focus on the most critical compliance failures
• Time period: Today, last 7 days, last 30 days, or all time — enabling trend analysis and incident investigation
• Corrected status: Filter to violations that were auto-corrected vs. those that were not — measure Rule Agent effectiveness

Each filter change triggers a SPARQL query against the Fuseki audit graph. Filters are dynamically composed into FILTER clauses with sanitized inputs (injection-safe), returning results in under 50ms on typical datasets. The result count is displayed as a live badge in the panel header.

Rule Details Modal: Full Provenance

Clicking any violation row opens a Rule Details modal with the complete audit record:

• Compliance score gauge with color-coded severity
• Full violation description (not truncated)
• Side-by-side original AI response snippet (amber) and corrected response snippet (green)
• Number of governance rules checked in this audit
• SPARQL provenance URI — the unique identifier of this violation node in the audit graph, with a one-click copy button

The provenance URI allows compliance teams to query the audit graph directly with SPARQL, retrieve the full RDF context of any violation, and attach it to compliance reports or regulatory filings as verifiable evidence of AI governance controls.

Why This Matters for Regulated Industries

The EU AI Act (Article 9), ISO 42001, and sector-specific regulations (GDPR, DORA, MiFID II) all require organizations to demonstrate that AI systems are subject to human oversight and that AI decisions are explainable and auditable. The ArcaQ governance architecture directly addresses these requirements:

• Auditability: Every AI governance decision is stored as RDF with a permanent identifier — regulators can query the full history
• Explainability: The Rule Details modal shows exactly which rules were checked, what the original response said, and what correction was applied
• Controllability: Compliance teams can modify governance rules in real-time without code changes — the Policy Agent gives non-technical administrators full control
• Sovereignty: All audit data remains on-premise in your Fuseki instance — zero data leaves your infrastructure