LLM Shield Guardrails - On-Premises Deployment Guide
RedHat Enterprise + H200 GPU Infrastructure
📋 Executive Summary
This document provides comprehensive guidance for deploying LLM Shield Guardrails in an on-premises environment using RedHat Enterprise Linux with NVIDIA H200 GPUs via MIG (Multi-Instance GPU) slicing. The deployment supports government and enterprise environments requiring air-gapped operation with multi-tenant security controls.
🏗️ Application Architecture
Complete System Architecture
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Government Internal Network (GIN) │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ Client Layer │
│ ┌─────────────┐┌─────────────┐┌─────────────┐┌─────────────┐┌─────────────┐┌───────────┐ │
│ │ RTA App ││ DP App ││ DEWA App ││Municipal App││ DH App ││Courts App │ │
│ │(Transport) ││(Police) ││(Utilities) ││(Municipal) ││(Health) ││(Justice) │ │
│ │ ││ ││ ││ ││ ││ 100+ │ │
│ └─────────────┘└─────────────┘└─────────────┘└─────────────┘└─────────────┘└───────────┘ │
│ │ │ │ │ │ │ │
│ └───────────────┼───────────────┼───────────────┼───────────────┼───────────┘ │
│ │ │ │ │ │
├─────────────────────────┼───────────────┼───────────────┼───────────────┼─────────────────┤
│ ┌─────────────────────────────────────────────────────┐ │
│ │ PA Firewall │ │
│ │ • Rate Limiting • DDoS Protection • SSL Offload │ │
│ └─────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ AI Factory │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ API Gateway Layer │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ API Gateway │ │ │
│ │ │ • Load Balancing • SSL Termination • Authentication │ │ │
│ │ │ • Request Routing • Health Checks • Circuit Breaker │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Security & Guardrails Layer │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ LLM Shield Guardrails Engine │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │
│ │ │ │ TIER 1 FAST │ │ TIER 2 SLOW │ │ RBAC │ │ │ │
│ │ │ │ CPU-based │ │ LLM-based │ │ & Policies │ │ │ │
│ │ │ │ <5ms │ │ 200-800ms │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ │ │• Keyword │ │• Adversarial│ │• Multi-Tenant│ │ │ │
│ │ │ │• PII │ │• Safety │ │• Rate Limits │ │ │ │
│ │ │ │• Regex │ │• Topic │ │• Tool Access │ │ │ │
│ │ │ │• Rate Limit │ │• Bias │ │• Data Scope │ │ │ │
│ │ │ │• Sentiment │ │• Factual │ │• Audit Trail │ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Redis Cluster │ │ │
│ │ │ • Tenant State • Rate Limiting • Session Management │ │ │
│ │ │ • Guardrail Cache • Real-time Metrics • Multi-tenant Isolation │ │ │
│ │ │ │ │ │
│ │ │ [Master-1] ←→ [Master-2] ←→ [Master-3] │ │ │
│ │ │ │ │ │ │ │ │
│ │ │ [Replica-1] [Replica-2] [Replica-3] │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Model Serving Layer │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Lite LLM Gateway │ │ │
│ │ │ • Model Routing • Load Balancing • Request Queuing │ │ │
│ │ │ • Token Management • Context Caching • Fallback Handling │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Model Workers (llama.cpp) │ │ │
│ │ │ │ │ │
│ │ │ Worker-1:8000 Worker-1:8001 Worker-2:8000 Worker-2:8001 │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │Qwen3.5-9B│ │Qwen0.8B │ │Qwen3.5-9B│ │Qwen0.8B │ │ │ │
│ │ │ │(Main) │ │(Draft) │ │(Main) │ │(Draft) │ │ │ │
│ │ │ │MIG-0 │ │MIG-1 │ │MIG-0 │ │MIG-1 │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ │ ... ... ... ... │ │ │
│ │ │ (Up to 103 GPU instances) │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Observability & Telemetry │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Votal I/O │ │ │
│ │ │ • Request Tracing • Performance Metrics • Security Events │ │ │
│ │ │ • Audit Logging • Compliance Reports • Real-time Dashboards │ │ │
│ │ │ │ │ │
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │
│ │ │ │Elasticsearch│ │ Splunk │ │ Grafana │ │ AlertMgr │ │ │ │
│ │ │ │ SIEM │ │ HEC │ │ Metrics │ │ Notifications│ │ │ │
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
├─────────────────────────────────────────────────────────────────────────────────────────┤
│ GPU Infrastructure │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ OpenShift + Ray Cluster │ │
│ │ │ │
│ │ Node-1: H200 (MIG 4x20GB) │ Node-2: H200 (MIG 4x20GB) │ ... │ Node-26: H200 │ │
│ │ ┌─────┐┌─────┐┌─────┐┌─────┐ ┌─────┐┌─────┐┌─────┐┌─────┐ ┌─────┐┌─────┐┌───┐ │ │
│ │ │MIG-0││MIG-1││MIG-2││MIG-3│ │MIG-0││MIG-1││MIG-2││MIG-3│ ... │MIG-0││MIG-1││...│ │ │
│ │ │20GB ││20GB ││20GB ││20GB │ │20GB ││20GB ││20GB ││20GB │ │20GB ││20GB ││ │ │ │
│ │ └─────┘└─────┘└─────┘└─────┘ └─────┘└─────┘└─────┘└─────┘ └─────┘└─────┘└───┘ │ │
│ │ │ │
│ │ Total: 103 GPU Instances │ │
│ │ Theoretical Peak: ~2,369 req/s @ 200ms │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────┘
Data Flow Architecture
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Client │───▶│ API Gateway │───▶│LLM Shield │───▶│ Lite LLM │───▶│GPU Workers │
│ Application │ │ │ │ Guardrails │ │ │ │(llama.cpp) │
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Audit │ │ Redis │ │ Ray │ │
│ │ Logs │ │ Cluster │ │ Scheduler │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│ Votal I/O Telemetry │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │Request Trace│ │Security │ │Performance │ │Compliance │ │Real-time │ │
│ │& Correlation│ │Events & │ │Metrics & │ │Reports & │ │Dashboards & │ │
│ │ IDs │ │Violations │ │Latency │ │Audit Trails │ │ Alerts │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────┘
Component Integration Details
1. Client Application Layer
# Application Categories & Use Cases
government_applications:
transport_authority: # RTA
- route_planning_assistant
- traffic_violation_queries
- public_transport_info
- driver_license_support
police_department: # DP
- incident_report_assistant
- case_management_queries
- public_safety_information
- emergency_response_guidance
utilities_authority: # DEWA
- billing_inquiry_assistant
- service_request_handler
- outage_information_bot
- energy_efficiency_advisor
municipal_services:
- permit_application_helper
- city_services_navigator
- complaint_resolution_bot
- public_records_assistant
health_department: # DH
- patient_information_system
- appointment_scheduling_bot
- medical_records_assistant
- public_health_advisor
data_center: # DC
- infrastructure_monitoring
- capacity_planning_assistant
- incident_response_coordinator
- compliance_checker
courts_system:
- case_status_inquiry
- legal_document_assistant
- hearing_scheduling_bot
- public_records_access
2. Security & Policy Enforcement
# Multi-Tenant Security Model
tenant_isolation:
network_level:
- dedicated_api_keys_per_tenant
- rate_limiting_by_tenant_id
- network_policies_via_rbac
data_level:
- encrypted_data_at_rest
- tenant_specific_encryption_keys
- data_retention_policies
- pii_redaction_by_clearance_level
application_level:
- rbac_roles_per_department
- tool_access_restrictions
- context_window_limitations
- audit_trail_per_tenant
# Example RBAC Configuration
rbac_policies:
rta_transport_agent:
clearance_level: "internal"
allowed_tools:
- search_route_database
- get_traffic_updates
- access_public_schedules
denied_tools:
- modify_traffic_systems
- access_police_records
max_context: 4096
rate_limit: "120/min"
courts_case_assistant:
clearance_level: "confidential"
allowed_tools:
- search_case_records
- schedule_hearings
- generate_legal_summaries
denied_tools:
- modify_court_decisions
- access_sealed_records
max_context: 8192
rate_limit: "60/min"
health_patient_bot:
clearance_level: "restricted"
allowed_tools:
- search_medical_protocols
- book_appointments
- access_public_health_data
denied_tools:
- access_patient_records
- prescribe_medications
max_context: 2048
rate_limit: "90/min"
3. Request Flow & Processing Pipeline
graph TD
A[Client Request] --> B{API Gateway}
B -->|Load Balance| C[LLM Shield Instance]
C -->|Authenticate| D{Auth Check}
D -->|✓ Valid| E[Tier 1 Fast Guardrails]
D -->|✗ Invalid| Z[Reject Request]
E -->|Pass| F{Need Deep Check?}
E -->|Block/Warn| G[Log & Respond]
F -->|Yes| H[Tier 2 Slow Guardrails]
F -->|No| I[Route to LiteeLLM]
H -->|Pass| I
H -->|Block| G
I -->|Queue| J{Ray Scheduler}
J -->|Select| K[Available GPU Worker]
K -->|Process| L[Model Inference]
L -->|Response| M[Output Guardrails]
M -->|Clean| N[Return to Client]
M -->|Violation| O[Redact/Block]
O --> N
G --> P[Votal Telemetry]
N --> P
P --> Q[Redis Cache]
P --> R[Audit Logs]
P --> S[Real-time Metrics]
4. GPU Resource Management
# OpenShift + Ray Cluster Configuration
gpu_allocation_strategy:
primary_models: # Qwen3.5-9B (Main inference)
instances_per_node: 2 # 2x MIG-20GB per H200
total_instances: 52 # 26 nodes × 2
concurrent_requests: 416 # 52 × 8 req/instance
draft_models: # Qwen0.8B (Speculative decoding)
instances_per_node: 2 # 2x MIG-20GB per H200
total_instances: 51 # 25.5 nodes × 2 (rounded)
concurrent_requests: 1020 # 51 × 20 req/instance
performance_estimates:
peak_throughput: "2,369 req/s"
average_latency: "200ms"
gpu_utilization: "85-95%"
memory_efficiency: "75-85%"
# Ray Auto-scaling Configuration
ray_cluster:
head_node:
cpu: 32
memory: "128GB"
role: "scheduler"
worker_nodes:
cpu: 64
memory: "256GB"
gpus: "4x H200-MIG-20GB"
replicas: 26
auto_scaling:
min_workers: 10
max_workers: 26
target_utilization: 80
scale_up_threshold: 85
scale_down_threshold: 60
Application-Specific Configurations
Based on your government use cases, here are tailored configurations:
RTA (Transport Authority) Configuration
rta_config:
tenant_id: "rta_transport_ae"
guardrails:
topic_enforcement:
allowed_topics: ["transportation", "traffic", "public_transit", "roads"]
blocked_topics: ["politics", "religion", "military"]
pii_detection:
entities: ["LICENSE_NUMBER", "VEHICLE_PLATE", "PASSPORT_ID"]
action: "redact"
tools:
- search_route_database
- get_traffic_updates
- access_public_schedules
- calculate_fare_estimates
DEWA (Utilities) Configuration
dewa_config:
tenant_id: "dewa_utilities_ae"
guardrails:
topic_enforcement:
allowed_topics: ["electricity", "water", "utilities", "billing"]
blocked_topics: ["politics", "competitors"]
financial_data:
redact_amounts: true
mask_account_numbers: true
tools:
- query_consumption_data
- billing_lookup
- service_request_status
- outage_information
Courts System Configuration
courts_config:
tenant_id: "courts_justice_ae"
security_level: "confidential"
guardrails:
topic_enforcement:
allowed_topics: ["legal", "court_procedures", "case_management"]
blocked_topics: ["politics", "bias", "personal_opinions"]
legal_compliance:
attorney_client_privilege: true
sealed_record_protection: true
tools:
- search_case_records
- schedule_hearings
- generate_legal_summaries
- check_statute_limitations
This comprehensive architecture ensures secure, scalable, and compliant AI services across all your government applications while maintaining strict tenant isolation and performance requirements.
🏗️ Corrected 2-Worker + Redis + Guardrail Server Architecture
For initial deployment, here’s the corrected architecture with LiteLLM’s built-in guardrails integration calling a separate Votal AI Guardrail Server:
Corrected 2-Worker Deployment Overview
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Master Node │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Admin & Control Plane │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Admin │ │ Votal I/O │ │ Guardrail Server│ │ │
│ │ │ Portal │ │ Telemetry │ │ (Votal AI) │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ • Multi-Tenant │ │ • Audit Logs │ │ • Policy Engine │ │ │
│ │ │ • Policy Mgmt │ │ • Real-time │ │ • ML Guardrails │ │ │
│ │ │ • Usage Stats │ │ Monitoring │ │ • Per-Tenant │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Redis Cluster Hub │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Redis Master-1 │ │ Redis Master-2 │ │ Redis Master-3 │ │ │
│ │ │ Port: 7000 │ │ Port: 7001 │ │ Port: 7002 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Tenant Configs │ │ Rate Limiting │ │ Session State │ │ │
│ │ │ Guardrail Cache │ │ Request Queue │ │ Model Cache │ │ │
│ │ │ Policy Rules │ │ Worker Health │ │ Metrics Buffer │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Redis Replica-1 │ │ Redis Replica-2 │ │ Redis Replica-3 │ │ │
│ │ │ Port: 7003 │ │ Port: 7004 │ │ Port: 7005 │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
│
┌───────────────┼───────────────┐
│ │ │
┌─────────────────────────▼─────────────────────────────────▼─────────────────────────┐
│ GPU Worker Nodes (LiteLLM + Models) │
├─────────────────────────┬─────────────────────────────────┬─────────────────────────┤
│ WORKER-1 │ │ WORKER-2 │
│ ┌─────────────────┐ │ │ ┌─────────────────┐ │
│ │ LiteLLM │ │ │ │ LiteLLM │ │
│ │ Gateway │ │ │ │ Gateway │ │
│ │ Port: 8080 │ │ Redis Cluster │ │ Port: 8080 │ │
│ │ │ │ Communication │ │ │ │
│ │ ┌─────────────┐ │ │ │ │ ┌─────────────┐ │ │
│ │ │ Guardrails │ │ │ ┌─────────────────────────┐ │ │ │ Guardrails │ │ │
│ │ │ Middleware │ │◄──┼────┤ Shared Data Plane ├──┼──►│ │ Middleware │ │ │
│ │ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ Calls to │ │ │ │ • Tenant Policy Sync │ │ │ │ Calls to │ │ │
│ │ │ Votal AI │ │ │ │ • Guardrail Cache │ │ │ │ Votal AI │ │ │
│ │ └─────────────┘ │ │ │ • Cross-Worker LB │ │ │ └─────────────┘ │ │
│ └─────────────────┘ │ │ • Health Monitoring │ │ └─────────────────┘ │
│ │ │ └─────────────────────────┘ │ │ │
│ ┌─────────────────┐ │ │ ┌─────────────────┐ │
│ │ Model Workers │ │ │ │ Model Workers │ │
│ │ (llama.cpp) │ │ │ │ (llama.cpp) │ │
│ │ │ │ │ │ │ │
│ │ ┌─────┬─────┐ │ │ │ │ ┌─────┬─────┐ │ │
│ │ │MIG-0│MIG-1│ │ │ │ │ │MIG-0│MIG-1│ │ │
│ │ │LLM │LLM │ │ │ │ │ │LLM │LLM │ │ │
│ │ │8000 │8001 │ │ │ │ │ │8002 │8003 │ │ │
│ │ └─────┴─────┘ │ │ │ │ └─────┴─────┘ │ │
│ │ ┌─────┬─────┐ │ │ │ │ ┌─────┬─────┐ │ │
│ │ │MIG-2│MIG-3│ │ │ │ │ │MIG-2│MIG-3│ │ │
│ │ │LLM │LLM │ │ │ │ │ │LLM │LLM │ │ │
│ │ │8004 │8005 │ │ │ │ │ │8006 │8007 │ │ │
│ │ └─────┴─────┘ │ │ │ │ └─────┴─────┘ │ │
│ └─────────────────┘ │ │ └─────────────────┘ │
│ │ │ │
│ H200 GPU (4x MIG) │ │ H200 GPU (4x MIG) │
│ 80GB Total VRAM │ │ 80GB Total VRAM │
└─────────────────────────┴─────────────────────────────────┴─────────────────────────┘
Model Distribution Strategy
Worker-1: Primary + Specialized Models
worker_1_config:
node_ip: "10.0.2.10"
gpu_count: 1 # H200 with 4x MIG instances
model_services:
# MIG-0 (20GB): Primary Model
qwen35_primary_8000:
port: 8000
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 8
context_length: 32768
use_cases: ["general_inference", "safety_checks", "topic_enforcement"]
# MIG-1 (20GB): Draft Model
qwen08_draft_8001:
port: 8001
model: "Qwen3.5-0.8B-Q4_K_M.gguf"
gpu_memory: "2GB"
max_concurrent: 32
context_length: 8192
use_cases: ["speculative_decoding", "fast_classification"]
# MIG-2 (20GB): Adversarial Detection
qwen35_adversarial_8004:
port: 8004
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 6
specialized_for: ["adversarial_detection", "jailbreak_prevention"]
guardrails: ["adversarial_detection", "safety_check"]
# MIG-3 (20GB): Bias & Content
qwen35_content_8005:
port: 8005
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 6
specialized_for: ["bias_detection", "tone_enforcement", "factual_grounding"]
guardrails: ["bias_detection", "tone_enforcement", "hallucinated_links"]
# Total Worker-1 Capacity: ~52 concurrent requests
Worker-2: Scaled + Backup Models
worker_2_config:
node_ip: "10.0.2.11"
gpu_count: 1 # H200 with 4x MIG instances
model_services:
# MIG-0 (20GB): Primary Model Replica
qwen35_primary_8002:
port: 8002
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 8
context_length: 32768
role: "primary_backup"
# MIG-1 (20GB): Draft Model Replica
qwen08_draft_8003:
port: 8003
model: "Qwen3.5-0.8B-Q4_K_M.gguf"
gpu_memory: "2GB"
max_concurrent: 32
context_length: 8192
role: "draft_backup"
# MIG-2 (20GB): Topic & PII Specialist
qwen35_topic_8006:
port: 8006
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 6
specialized_for: ["topic_restriction", "pii_leakage", "role_redaction"]
guardrails: ["topic_restriction", "pii_leakage"]
# MIG-3 (20GB): Multi-Purpose
qwen35_multipurpose_8007:
port: 8007
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 8
role: "overflow_general"
handles: ["peak_load", "failover", "general_inference"]
# Total Worker-2 Capacity: ~54 concurrent requests
# Combined System: ~106 concurrent requests
Redis Data Architecture
Redis Key Distribution & Usage Patterns
# Redis Cluster Key Distribution (3 Masters)
redis_data_model:
# Master-1 (7000): Tenant & Configuration Data
tenant_data:
key_patterns:
- "tenant:config:{tenant_id}" # Tenant settings
- "tenant:rbac:{tenant_id}" # RBAC policies
- "tenant:quota:{tenant_id}" # Usage quotas
- "tenant:keys:{tenant_id}" # API keys
- "tenant:audit:{tenant_id}:{date}" # Audit events
data_structure:
tenant_config: "Hash" # Nested config object
rbac_policies: "Hash" # Role-based access rules
api_keys: "Set" # Active API key list
usage_quotas: "Hash" # Current usage vs limits
audit_events: "List" # Time-ordered events
ttl_settings:
config_cache: 3600 # 1 hour
audit_events: 604800 # 7 days
api_key_validation: 300 # 5 minutes
# Master-2 (7001): Request Flow & Rate Limiting
request_flow:
key_patterns:
- "rate_limit:{tenant_id}:{client_ip}" # Rate limiting
- "request:queue:{priority}" # Request queues
- "worker:health:{worker_id}" # Worker status
- "worker:load:{worker_id}" # Current load
- "session:state:{session_id}" # Conversation state
data_structure:
rate_limits: "String" # Token bucket counters
request_queue: "List" # FIFO queue with priorities
worker_health: "Hash" # Status + last seen
worker_load: "Hash" # Current requests + capacity
session_state: "Hash" # Conversation context
ttl_settings:
rate_limit_window: 60 # 1 minute sliding window
worker_health: 30 # 30 seconds
request_queue: 300 # 5 minutes max queue time
session_state: 86400 # 24 hours
# Master-3 (7002): Caching & Metrics
cache_and_metrics:
key_patterns:
- "cache:guardrail:{content_hash}" # Guardrail results
- "cache:embedding:{text_hash}" # Text embeddings
- "metrics:latency:{endpoint}" # Performance data
- "metrics:usage:{tenant}:{date}" # Usage statistics
- "alert:threshold:{metric_name}" # Alert conditions
data_structure:
guardrail_cache: "Hash" # Results + metadata
embedding_cache: "String" # Serialized vectors
latency_metrics: "ZSet" # Time-ordered measurements
usage_metrics: "Hash" # Aggregated counters
alert_state: "String" # Current alert status
ttl_settings:
guardrail_cache: 300 # 5 minutes
embedding_cache: 1800 # 30 minutes
metrics_data: 86400 # 24 hours
alert_state: 3600 # 1 hour
Redis Integration with Workers
# Worker Connection & Data Flow
worker_redis_integration:
connection_config:
# Each worker connects to full Redis cluster
cluster_nodes:
- "redis-master-1:7000"
- "redis-master-2:7001"
- "redis-master-3:7002"
connection_pool:
max_connections_per_node: 50
health_check_interval: 10
retry_on_failure: true
circuit_breaker: true
# Data Flow Patterns
request_lifecycle:
1_authentication:
action: "GET tenant:config:{tenant_id}"
target: "Master-1"
purpose: "Validate API key + get tenant config"
2_rate_limiting:
action: "INCR rate_limit:{tenant_id}:{client_ip}"
target: "Master-2"
purpose: "Check + update rate limit counters"
3_load_balancing:
action: "HGETALL worker:load:*"
target: "Master-2"
purpose: "Find least loaded model worker"
4_guardrail_cache:
action: "HGET cache:guardrail:{content_hash}"
target: "Master-3"
purpose: "Check for cached guardrail results"
5_session_management:
action: "HSET session:state:{session_id} context {data}"
target: "Master-2"
purpose: "Store conversation context"
6_result_caching:
action: "HSET cache:guardrail:{content_hash} result {data}"
target: "Master-3"
purpose: "Cache guardrail results for reuse"
7_metrics_collection:
action: "ZADD metrics:latency:{endpoint} {timestamp} {latency}"
target: "Master-3"
purpose: "Record performance metrics"
8_audit_logging:
action: "LPUSH tenant:audit:{tenant_id}:{date} {event}"
target: "Master-1"
purpose: "Log security events for compliance"
Cross-Worker Communication & Failover
# High Availability & Load Balancing via Redis
ha_configuration:
health_monitoring:
# Each worker updates health status every 10s
worker_heartbeat:
key: "worker:health:{worker_id}"
data:
last_seen: timestamp
status: "healthy|degraded|offline"
current_load: "45/100"
gpu_memory: "75%"
queue_depth: 12
load_balancing:
# Smart routing based on worker capacity + specialization
routing_algorithm:
1_check_cache: "Is result already cached?"
2_check_specialization: "Which worker handles this guardrail?"
3_check_capacity: "Which worker has lowest load?"
4_route_request: "Send to optimal worker"
routing_logic: |
if cached_result:
return cached_result
elif specialized_guardrail:
return route_to_specialist_worker(guardrail_type)
else:
return route_to_least_loaded_worker()
failover_strategy:
# Automatic failover when worker becomes unavailable
detection:
missed_heartbeats: 3 # 30 seconds without heartbeat
connection_timeout: 5 # 5 seconds TCP timeout
health_check_failure: 2 # 2 consecutive health check fails
recovery_actions:
1_mark_offline: "UPDATE worker:health:{id} status offline"
2_drain_queue: "LPOP request:queue:{worker_id} → route to healthy workers"
3_redistribute: "Recalculate load balancing weights"
4_alert: "Send alert to ops team"
recovery_process:
1_detect_return: "Worker heartbeat resumed"
2_health_check: "Validate GPU + model availability"
3_gradual_ramp: "Slowly increase traffic allocation"
4_full_restore: "Return to normal load balancing"
Correct Final Architecture: CPU Services + GPU Models
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Master Node (CPU Only) │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ CPU Services Layer │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Admin │ │ Votal I/O │ │ Guardrail Server│ │ │
│ │ │ Portal │ │ Telemetry │ │ (CPU Service) │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ • Multi-Tenant │ │ • Audit Logs │ │ • Policy Engine │ │ │
│ │ │ • Policy Mgmt │ │ • Real-time │ │ • Redis Cache │ │ │
│ │ │ • Usage Stats │ │ Monitoring │ │ • API Router │ │ │
│ │ │ │ │ │ │ • Calls GPU │ │ │
│ │ │ │ │ │ │ Guard Models │ │ │
│ │ │ │ │ │ │ Port: 9000 │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────────────────────────────┐ │
│ │ Redis Cluster │ │
│ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Redis Master-1 │ │ Redis Master-2 │ │ Redis Master-3 │ │ │
│ │ │ Port: 7000 │ │ Port: 7001 │ │ Port: 7002 │ │ │
│ │ │ │ │ │ │ │ │ │
│ │ │ Tenant Configs │ │ Rate Limiting │ │ Session State │ │ │
│ │ │ Guardrail Cache │ │ Request Queue │ │ Model Cache │ │ │
│ │ │ Policy Rules │ │ Worker Health │ │ Metrics Buffer │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ │ │ │ │ │ │
│ │ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │
│ │ │ Redis Replica-1 │ │ Redis Replica-2 │ │ Redis Replica-3 │ │ │
│ │ │ Port: 7003 │ │ Port: 7004 │ │ Port: 7005 │ │ │
│ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
│ ▲
│ │
┌───────────────▼──────────────┐ │ Guardrail
│ LiteLLM Gateway (CPU) │ │ Server Calls
│ • Middleware Integration │ │ Port 9000
│ • Calls Guardrail Server │ │
└───────────────┬──────────────┘ │
│ │
┌─────────▼─────────┐ │
│ │ │
┌─────────────────────────▼─────────────────────────────────▼─────────────────────────┐
│ GPU Worker Nodes │
├─────────────────────────┬─────────────────────────────────┬─────────────────────────┤
│ WORKER-1 │ │ WORKER-2 │
│ ┌─────────────────┐ │ │ ┌─────────────────┐ │
│ │GPU Model Workers│ │ │ │GPU Model Workers│ │
│ │ (llama.cpp) │ │ │ │ (llama.cpp) │ │
│ │ │ │ Redis Cluster │ │ │ │
│ │ Main + Guard │ │ Communication │ │ Main + Guard │ │
│ │ Models │ │ │ │ Models │ │
│ └─────────────────┘ │ ┌─────────────────────────┐ │ └─────────────────┘ │
│ │ │ │ Shared Data Plane │ │ │ │
│ ┌─────────────────┐ │ │ │ │ ┌─────────────────┐ │
│ │ ┌─────┬─────┐ │ │ │ • Model Health Check │ │ │ ┌─────┬─────┐ │ │
│ │ │MIG-0│MIG-1│ │ │ │ • Load Balancing │ │ │ │MIG-0│MIG-1│ │ │
│ │ │Main │Guard│ │ │ │ • Cache Coordination │ │ │ │Main │Guard│ │ │
│ │ │8000 │8100 │ │ │ │ • Cross-Worker Sync │ │ │ │8002 │8102 │ │ │
│ │ └─────┴─────┘ │ │ └─────────────────────────┘ │ │ └─────┴─────┘ │ │
│ │ ┌─────┬─────┐ │ │ │ │ ┌─────┬─────┐ │ │
│ │ │MIG-2│MIG-3│ │ │ │ │ │MIG-2│MIG-3│ │ │
│ │ │Main │Guard│ │ │ │ │ │Main │Guard│ │ │
│ │ │8004 │8104 │ │ │ │ │ │8006 │8106 │ │ │
│ │ └─────┴─────┘ │ │ │ │ └─────┴─────┘ │ │
│ └─────────────────┘ │ │ └─────────────────┘ │
│ │ │ │
│ H200 GPU (4x MIG) │ │ H200 GPU (4x MIG) │
│ Main LLMs + Guards │ │ Main LLMs + Guards │
└─────────────────────────┴─────────────────────────────────┴─────────────────────────┘
Correct Request Flow (Matching Votal AI Architecture)
┌─────────────────────────────────────────────────────────┐
│ Votal Guardrails API / Policy Engine │
│ Central policy decision plane │
│ │
│ centralized thresholds, actions, policy sets │
│ and decisioning │
│ │
│ POST /classify │ POST /classify_output │
│ (INPUT) │ (OUTPUT) │
└─────────────────────────────────────────────────────────┘
▲ ▲
│ │
│ │
┌─────────────┐ ┌─────────────┐ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Your app │───▶│ LLM Lite │─────▶│ Pre-call │─▶│ LLM layer │─▶│ Post-call │─▶│ Secure │
│ │ │ gateway │ │ │ inspection │ │ │ │ inspection │ │ response │
│ copilot, │ │ │ │ │ │ │ • Qwen │ │ │ │ │
│ agent or │ │ auth • │ │ │ sync input │ │ • Llama │ │ sanitize │ │ clean output│
│ API caller │ │ retries • │ │ │ checks │ │ • GLM │ │ response │ │ back to │
│ │ │ routing │ │ │ block before│ │ • Kiwi │ │ redact, warn│ │ application │
│ │ │ │ │ │ token spend │ │ (GPU) │ │ or block │ │ │
└─────────────┘ └─────────────┘ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
│ │
│ │
│ │
┌────────────────────────────────────────────────────────────┐
│ Runtime data plane │
│ synchronous enforcement path │
│ │
│ ALLOW │ BLOCK │ WARN │ REDACT │ RATE LIMIT │
│ continue │ → 403 │ → log │ → sanitize │ → throttle │
└────────────────────────────────────────────────────────────┘
▲ ▲
│ │
│ │
┌─────────────────────┐ ┌─────────────────────┐
│ Guard Model Workers │ │ Redis │
│ (GPU) │ │ │
│ │ │ • Cache Results │
│ • Input Guards │◄────────│ • Tenant Policies │
│ • Output Guards │ │ • Config Store │
│ • Adversarial │ │ • Runtime Updates │
│ • Safety/Bias │ │ /v1/shield/config │
│ (Ports 8100-8106) │ │ │
└─────────────────────┘ └─────────────────────┘
GPU Model Distribution Strategy
Now with both Main LLMs and Guard Models on same GPU workers:
Worker-1: Mixed Main + Guard Models
worker_1_gpu_allocation:
node_ip: "10.0.2.10"
gpu_count: 1 # H200 with 4x MIG instances (20GB each)
model_services:
# MIG-0 (20GB): Main Model - Qwen
qwen_main_8000:
port: 8000
model: "Qwen2.5-7B-Instruct-Q4_K_M.gguf"
gpu_memory: "16GB"
max_concurrent: 8
context_length: 32768
type: "main_inference"
upstream_name: "qwen"
# MIG-1 (20GB): Guard Model - Input Safety
votal_guard_input_8100:
port: 8100
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 6
specialization: "input_guard"
type: "guard_model"
# MIG-2 (20GB): Main Model - Llama
llama_main_8004:
port: 8004
model: "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
gpu_memory: "16GB"
max_concurrent: 8
context_length: 32768
type: "main_inference"
upstream_name: "llama"
# MIG-3 (20GB): Guard Model - Output Safety
votal_guard_output_8104:
port: 8104
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 6
specialization: "output_guard"
type: "guard_model"
# Total Worker-1: 16 main + 12 guard = 28 concurrent requests
Worker-2: Mixed Main + Guard Models
worker_2_gpu_allocation:
node_ip: "10.0.2.11"
gpu_count: 1 # H200 with 4x MIG instances (20GB each)
model_services:
# MIG-0 (20GB): Main Model - GLM
glm_main_8002:
port: 8002
model: "ChatGLM3-6B-Q4_K_M.gguf"
gpu_memory: "14GB"
max_concurrent: 10
context_length: 8192
type: "main_inference"
upstream_name: "glm"
# MIG-1 (20GB): Guard Model - Input/Output Shared
votal_guard_shared_8102:
port: 8102
model: "Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
gpu_memory: "18GB"
max_concurrent: 8
specialization: "input_guard,output_guard"
type: "guard_model"
# MIG-2 (20GB): Main Model - Kiwi
kiwi_main_8006:
port: 8006
model: "Kiwi-7B-Instruct-Q4_K_M.gguf"
gpu_memory: "16GB"
max_concurrent: 8
type: "main_inference"
upstream_name: "kiwi"
# MIG-3 (20GB): Multi-Model Slot
multi_model_8106:
port: 8106
models:
- "Qwen2.5-7B-Instruct-Q4_K_M.gguf" # Backup Qwen
- "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf" # Backup Llama
gpu_memory: "16GB"
max_concurrent: 8
type: "overflow_main"
# Total Worker-2: 26 main + 8 guard = 34 concurrent requests
# Combined System: 42 main + 20 guard = 62 total concurrent
LiteLLM Configuration for Government Deployment
# /opt/llm-shield/config/litellm-config.yaml
model_list:
# Main LLM Models (GPU Workers)
- model_name: qwen
litellm_params:
model: openai/qwen # Custom provider mapping
api_base: "http://worker-1:8000/v1"
api_key: "dummy"
- model_name: llama
litellm_params:
model: openai/llama
api_base: "http://worker-1:8004/v1"
api_key: "dummy"
- model_name: glm
litellm_params:
model: openai/glm
api_base: "http://worker-2:8002/v1"
api_key: "dummy"
- model_name: kiwi
litellm_params:
model: openai/kiwi
api_base: "http://worker-2:8006/v1"
api_key: "dummy"
# Guardrails Configuration
guardrails:
- guardrail_name: "votal-input-guard"
litellm_params:
guardrail: votal_guardrail.VotalGuardrail
mode: "pre_call"
default_on: true
config:
server_url: "http://guardrail-server:9000"
endpoints:
- "http://worker-1:8100/v1/generate" # Input guard
- "http://worker-2:8102/v1/generate" # Backup input guard
cache_enabled: true
timeout: 800
- guardrail_name: "votal-output-guard"
litellm_params:
guardrail: votal_guardrail.VotalGuardrail
mode: "post_call"
default_on: true
config:
server_url: "http://guardrail-server:9000"
endpoints:
- "http://worker-1:8104/v1/generate" # Output guard
- "http://worker-2:8102/v1/generate" # Backup output guard
cache_enabled: true
timeout: 600
# Multi-tenant routing
router_settings:
routing_strategy: "tenant_aware"
tenant_model_mapping:
courts_tenant: "qwen" # Most capable for legal
healthcare_tenant: "llama" # Good for medical
rta_tenant: "glm" # Fast for transport
dewa_tenant: "kiwi" # Efficient for utilities
default: "qwen" # Fallback
litellm_settings:
set_verbose: true
redis_host: "redis-master-1"
redis_port: 7000
cache_ttl: 300
Guardrail Server Configuration (CPU Service)
The Guardrail Server runs on CPU only and coordinates between Redis and GPU guard models:
# Guardrail Server (Master Node - Port 9000 - CPU Only)
guardrail_server:
port: 9000
type: "cpu_service" # No GPU required
# Redis Integration
redis_config:
cluster_nodes:
- "redis-master-1:7000"
- "redis-master-2:7001"
- "redis-master-3:7002"
cache_ttl: 300 # 5 minutes
policy_refresh: 60 # 1 minute
# GPU Guard Model Endpoints
guard_models:
adversarial_detection:
endpoints:
- "http://worker-1:8100" # Worker-1 MIG-1
- "http://worker-2:8102" # Worker-2 MIG-1 (backup)
load_balancing: "round_robin"
timeout: 800 # ms
safety_check:
endpoints:
- "http://worker-1:8104" # Worker-1 MIG-3
- "http://worker-2:8106" # Worker-2 MIG-3 (backup)
load_balancing: "least_connections"
timeout: 500 # ms
bias_detection:
endpoints:
- "http://worker-1:8104" # Shared with safety
- "http://worker-2:8106" # Shared with safety
timeout: 600 # ms
topic_restriction:
endpoints:
- "http://worker-2:8102" # Worker-2 MIG-1
- "http://worker-1:8100" # Worker-1 MIG-1 (backup)
timeout: 400 # ms
# API Endpoints
endpoints:
- path: "/classify/adversarial"
method: "POST"
guard_model: "adversarial_detection"
cache_key: "adv_{content_hash}"
- path: "/classify/safety"
method: "POST"
guard_model: "safety_check"
cache_key: "safety_{content_hash}"
- path: "/classify/bias"
method: "POST"
guard_model: "bias_detection"
cache_key: "bias_{content_hash}"
- path: "/classify/topic"
method: "POST"
guard_model: "topic_restriction"
cache_key: "topic_{content_hash}"
# LiteLLM Configuration (CPU Service)
litellm_config:
port: 8080
type: "cpu_service" # No GPU required
# Main LLM Model Endpoints
main_models:
qwen35_9b:
endpoints:
- "http://worker-1:8000" # Worker-1 MIG-0
- "http://worker-1:8004" # Worker-1 MIG-2
- "http://worker-2:8002" # Worker-2 MIG-0
- "http://worker-2:8006" # Worker-2 MIG-2
load_balancing: "least_loaded"
# Guardrail Integration
guardrails:
server_url: "http://guardrail-server:9000"
# Pre-request guardrails
pre_call_checks:
- name: "input_safety"
endpoint: "/classify/safety"
timeout: 500
cache_enabled: true
- name: "adversarial_check"
endpoint: "/classify/adversarial"
timeout: 800
enabled_for_tenants: ["all"]
cache_enabled: true
# Post-response guardrails
post_call_checks:
- name: "output_safety"
endpoint: "/classify/safety"
timeout: 500
cache_enabled: true
- name: "bias_detection"
endpoint: "/classify/bias"
timeout: 600
enabled_for_tenants: ["government", "healthcare"]
cache_enabled: true
# Tenant-specific policies (stored in Redis)
tenant_policies:
courts_tenant:
required_pre: ["input_safety", "adversarial_check"]
required_post: ["output_safety", "bias_detection"]
optional_pre: []
optional_post: ["topic_restriction"]
healthcare_tenant:
required_pre: ["input_safety", "adversarial_check"]
required_post: ["bias_detection"]
optional_pre: ["topic_restriction"]
optional_post: ["output_safety"]
rta_tenant:
required_pre: ["input_safety"]
required_post: ["output_safety"]
optional_pre: ["adversarial_check"]
optional_post: ["topic_restriction"]
dewa_tenant:
required_pre: ["input_safety"]
required_post: ["output_safety"]
optional_pre: ["topic_restriction"]
optional_post: ["bias_detection"]
Guardrail Server Request Processing Logic
# Pseudo-code for Guardrail Server (CPU Service)
async def process_guardrail_request(request_type: str, content: str, tenant_id: str):
# 1. Check Redis cache first
cache_key = f"{request_type}_{hash(content)}"
cached_result = await redis.get(cache_key)
if cached_result:
return cached_result
# 2. Get tenant policy from Redis
tenant_policy = await redis.get(f"tenant:policy:{tenant_id}")
# 3. Determine which guard model to call
guard_model_config = get_guard_model_for_request(request_type)
# 4. Call GPU-hosted guard model
guard_model_url = select_endpoint(guard_model_config["endpoints"])
result = await http_client.post(
f"{guard_model_url}/generate",
json={"prompt": build_guardrail_prompt(request_type, content)},
timeout=guard_model_config["timeout"]
)
# 5. Cache result in Redis
await redis.setex(cache_key, 300, result) # 5 min TTL
# 6. Return result to LiteLLM
return result
Performance Characteristics
# Expected Performance with 2-Worker Setup
performance_metrics:
throughput:
combined_capacity: "106 concurrent requests"
peak_throughput: "530 requests/second" # Assuming 200ms avg latency
sustained_load: "400-450 req/s" # With 15% headroom
latency:
fast_guardrails: "< 5ms" # Tier 1 (CPU)
slow_guardrails: "200-400ms" # Tier 2 (GPU)
cache_hit: "< 2ms" # Redis lookup
end_to_end: "250-500ms" # Including network
availability:
worker_redundancy: "2x primary capacity"
redis_cluster: "3-node cluster with replicas"
failover_time: "< 10 seconds"
recovery_time: "< 60 seconds"
target_sla: "99.9% uptime"
resource_utilization:
gpu_memory: "75-85% utilization"
gpu_compute: "80-90% utilization"
redis_memory: "< 50% on each master"
network_bandwidth: "< 20% of 10Gbps"
Deployment Commands for 2-Worker Setup
# 1. Deploy Redis Cluster (on master node)
docker-compose -f docker-compose.redis-cluster.yml up -d
# 2. Initialize Redis Cluster
redis-cli --cluster create \
master-node:7000 master-node:7001 master-node:7002 \
master-node:7003 master-node:7004 master-node:7005 \
--cluster-replicas 1
# 3. Deploy Worker-1
docker-compose -f docker-compose.worker-1.yml up -d
# 4. Deploy Worker-2
docker-compose -f docker-compose.worker-2.yml up -d
# 5. Verify all services
./scripts/health-check-2worker.sh
# 6. Load test the setup
./scripts/load-test-2worker.sh
This 2-worker architecture provides a robust foundation that can handle substantial government workloads while maintaining high availability through Redis-coordinated failover and intelligent load balancing.
💻 Compute Requirements
Recommended Hardware Configuration
Master Node
- CPU: 32+ cores (Intel Xeon or AMD EPYC)
- RAM: 128GB+ DDR4/DDR5
- Storage: 1TB NVMe SSD (OS + container images)
- Network: 10Gbps+ connectivity
- Role: Redis coordination, admin portal, telemetry aggregation
GPU Worker Nodes
- CPU: 64+ cores per node
- RAM: 256GB+ DDR5
- GPU: NVIDIA H200 80GB (with MIG enabled)
- Storage: 2TB NVMe SSD per node
- Network: 25Gbps+ InfiniBand/Ethernet
MIG Configuration (per H200)
| MIG Slice | GPU Memory | Recommended Use | Concurrent Requests |
|---|---|---|---|
| 1g.20gb | 20GB | Qwen3.5-9B Primary | 8-12 |
| 1g.20gb | 20GB | Draft Model (0.8B) | 20-40 |
| 1g.20gb | 20GB | Specialized Models | 8-12 |
| 2g.40gb | 40GB | Large Context (32k+) | 4-8 |
Total per H200: 3-4 MIG instances → 32-72 concurrent requests
Performance Benchmarks
| Configuration | Throughput | Latency | Memory Usage |
|---|---|---|---|
| 1x H200 MIG (20GB) | ~23 req/s | ~200ms | 6-8GB VRAM |
| 4x H200 MIG (80GB) | ~92 req/s | ~200ms | 24-32GB VRAM |
| Full 103 GPU allocation | ~2,369 req/s | ~200ms | 618-824GB VRAM |
🔧 Installation Requirements
Operating System
- RedHat Enterprise Linux 8.8+ or RedHat Enterprise Linux 9.2+
- SELinux: Enforcing mode supported
- Firewalld: Configured with required ports
Container Runtime
# Install Podman (preferred) or Docker
sudo dnf install -y podman podman-compose
# OR
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
NVIDIA Driver & Container Toolkit
# Install NVIDIA driver (latest)
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo
sudo dnf install -y cuda-drivers nvidia-driver-cuda
# Install NVIDIA Container Toolkit
curl -s -L https://nvidia.github.io/nvidia-container-toolkit/stable/rpm/nvidia-container-toolkit.repo | \
sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo dnf install -y nvidia-container-toolkit
# Configure for Podman
sudo nvidia-ctk runtime configure --runtime=podman
sudo systemctl restart podman
Required Packages
sudo dnf groupinstall -y "Development Tools"
sudo dnf install -y \
redis \
python3.11 \
python3.11-pip \
git \
curl \
wget \
htop \
nvidia-smi \
jq
Redis Infrastructure Setup
Redis serves as the backbone for multi-tenant state management, rate limiting, and session storage. For enterprise deployment, a clustered Redis setup is recommended.
Redis Cluster Architecture
┌─────────────────────────────────────────────────────────────┐
│ Redis Cluster │
├─────────────────────────────────────────────────────────────┤
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │Redis Master │ │Redis Master │ │Redis Master │ │
│ │ Node 1 │ │ Node 2 │ │ Node 3 │ │
│ │Port: 7000 │ │Port: 7001 │ │Port: 7002 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │ │ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │Redis Replica│ │Redis Replica│ │Redis Replica│ │
│ │ Node 4 │ │ Node 5 │ │ Node 6 │ │
│ │Port: 7003 │ │Port: 7004 │ │Port: 7005 │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────┘
🚀 Deployment Procedures
1. Pre-Deployment Setup
Network Configuration
# Configure firewall rules
sudo firewall-cmd --permanent --add-port=8080/tcp # LLM Shield API
sudo firewall-cmd --permanent --add-port=8000-8103/tcp # GPU workers
sudo firewall-cmd --permanent --add-port=6379/tcp # Redis
sudo firewall-cmd --permanent --add-port=9200/tcp # Elasticsearch (optional)
sudo firewall-cmd --reload
Create Directory Structure
sudo mkdir -p /opt/llm-shield/{config,data,logs,models}
sudo chown -R $(whoami):$(whoami) /opt/llm-shield
cd /opt/llm-shield
MIG Configuration
# Enable MIG mode on all H200 GPUs
sudo nvidia-smi -mig 1
# Create MIG instances (example for 4x 20GB slices per GPU)
for gpu in {0..6}; do # Assuming 7 H200 GPUs
sudo nvidia-smi mig -cgi 20gb -C
sudo nvidia-smi mig -cgi 20gb -C
sudo nvidia-smi mig -cgi 20gb -C
sudo nvidia-smi mig -cgi 20gb -C
done
# Verify MIG configuration
nvidia-smi -L # Should show MIG instances
2. Base Installation
Clone Repository
git clone https://github.com/your-org/llm-shield.git
cd llm-shield
Download Models (Air-Gapped)
# If internet access available
./scripts/download_models.sh
# For air-gapped environments, transfer models manually:
mkdir -p models/
# Copy Qwen3.5-9B-guardrailed-Q4_K_M.gguf (6.2GB)
# Copy Qwen3.5-0.8B-Q4_K_M.gguf (800MB)
3. Configuration Setup
Master Configuration (/opt/llm-shield/config/production.yaml)
# Multi-tenant enterprise configuration
auth:
enabled: true
admin_key: "admin-secure-key-change-me"
api_keys:
- "tenant-rta-key-abc123"
- "tenant-dewa-key-def456"
- "tenant-courts-key-ghi789"
# Redis cluster for state management
redis:
url: "redis://master-node:6379"
cluster_mode: true
max_connections: 1000
# GPU worker distribution
llm_backend:
model_path: "/models/Qwen3.5-9B-guardrailed-Q4_K_M.gguf"
draft_model_path: "/models/Qwen3.5-0.8B-Q4_K_M.gguf"
servers:
# Node 1: H200 MIG instances 0-3
- url: "http://worker-1:8000"
gpu: "MIG-0"
guardrails: ["adversarial_detection", "safety_check"]
- url: "http://worker-1:8001"
gpu: "MIG-1"
guardrails: ["topic_restriction", "tone_enforcement"]
- url: "http://worker-1:8002"
gpu: "MIG-2"
guardrails: ["bias_detection", "pii_leakage"]
- url: "http://worker-1:8003"
gpu: "MIG-3"
guardrails: ["hallucinated_links"]
# Node 2: H200 MIG instances 4-7
- url: "http://worker-2:8000"
gpu: "MIG-0"
guardrails: ["adversarial_detection", "safety_check"]
# ... repeat for all worker nodes
# Guardrails configuration
guardrails:
# Fast tier (always enabled)
keyword_blocklist:
enabled: true
action: block
settings:
keywords: ["classified", "confidential", "secret", "exploit"]
pii_detection:
enabled: true
action: redact
settings:
entities: ["SSN", "CREDIT_CARD", "PHONE_NUMBER", "EMAIL"]
rate_limiter:
enabled: true
action: block
settings:
requests_per_minute: 60
requests_per_hour: 1000
# Slow tier (LLM-based)
adversarial_detection:
enabled: true
action: block
settings:
confidence_threshold: 0.8
safety_check:
enabled: true
action: warn
settings:
safety_threshold: 0.7
# Tenant-specific RBAC
rbac:
roles:
rta-agent:
allowed_tools: ["search_transport_db", "get_route_info"]
max_tokens_per_request: 4096
rate_limit: "120/min"
data_clearance: "internal"
courts-clerk:
allowed_tools: ["search_cases", "get_court_schedule"]
max_tokens_per_request: 2048
rate_limit: "60/min"
data_clearance: "confidential"
dewa-operator:
allowed_tools: ["query_consumption", "billing_lookup"]
max_tokens_per_request: 2048
rate_limit: "90/min"
data_clearance: "internal"
agents:
rta-bot-1: rta-agent
rta-bot-2: rta-agent
courts-assistant: courts-clerk
dewa-support: dewa-operator
# Telemetry integration
telemetry:
enabled: true
flush_interval_seconds: 5
elasticsearch:
enabled: true
url: "https://elastic.internal.gov:9200"
api_key: "elastic-api-key-here"
index: "llm-guardrails-audit"
# Custom SIEM integration
webhook:
enabled: true
url: "https://siem.internal.gov/api/events"
headers:
Authorization: "Bearer siem-token"
4. Container Deployment
Master Node (Redis + Admin)
# Create master node compose file
cat > docker-compose.master.yml << 'EOF'
version: '3.8'
services:
redis:
image: redis:7-alpine
ports:
- "6379:6379"
volumes:
- redis_data:/data
- ./config/redis.conf:/usr/local/etc/redis/redis.conf
command: redis-server /usr/local/etc/redis/redis.conf
restart: unless-stopped
llm-shield-admin:
image: your-registry/llm-shield:v1.0
build:
context: .
dockerfile: Dockerfile.admin
ports:
- "8080:8080"
environment:
- CONFIG_PATH=/data/production.yaml
- REDIS_URL=redis://redis:6379
- SHIELD_ADMIN_KEY=admin-secure-key-change-me
volumes:
- ./config/production.yaml:/data/production.yaml:ro
- ./logs:/data/logs
depends_on:
- redis
restart: unless-stopped
volumes:
redis_data:
EOF
# Deploy master node
docker-compose -f docker-compose.master.yml up -d
GPU Worker Nodes
# Create worker node template
cat > docker-compose.worker.yml << 'EOF'
version: '3.8'
services:
llm-shield-worker-0:
image: your-registry/llm-shield:v1.0
ports:
- "8000:8000"
environment:
- CONFIG_PATH=/data/production.yaml
- REDIS_URL=redis://master-node:6379
- CUDA_VISIBLE_DEVICES=MIG-GPU-0
- WORKER_ID=0
volumes:
- ./config/production.yaml:/data/production.yaml:ro
- ./models:/models:ro
- ./logs:/data/logs
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["MIG-GPU-0"]
capabilities: [gpu]
restart: unless-stopped
llm-shield-worker-1:
image: your-registry/llm-shield:v1.0
ports:
- "8001:8001"
environment:
- CONFIG_PATH=/data/production.yaml
- REDIS_URL=redis://master-node:6379
- CUDA_VISIBLE_DEVICES=MIG-GPU-1
- WORKER_ID=1
- PORT=8001
volumes:
- ./config/production.yaml:/data/production.yaml:ro
- ./models:/models:ro
- ./logs:/data/logs
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["MIG-GPU-1"]
capabilities: [gpu]
restart: unless-stopped
# Repeat for all MIG instances...
EOF
# Deploy worker nodes
docker-compose -f docker-compose.worker.yml up -d
🔴 Redis Cluster Configuration
1. Redis Cluster Setup
Redis is critical for LLM Shield’s multi-tenant architecture, providing:
- Tenant isolation and configuration storage
- Rate limiting across distributed workers
- Session management for stateful conversations
- Guardrail caching for performance optimization
- Real-time metrics aggregation
Hardware Requirements (per Redis node)
- CPU: 8+ cores
- RAM: 32GB+ (with 16GB allocated to Redis)
- Storage: 500GB SSD for persistence
- Network: 10Gbps+ for cluster communication
Master Nodes Configuration
Create Redis cluster configuration files for each master node:
Node 1 (/etc/redis/redis-7000.conf)
# Network
port 7000
bind 10.0.1.10 # Replace with actual IP
protected-mode yes
tcp-backlog 511
timeout 0
tcp-keepalive 300
# Cluster
cluster-enabled yes
cluster-config-file nodes-7000.conf
cluster-node-timeout 15000
cluster-announce-ip 10.0.1.10
cluster-announce-port 7000
cluster-announce-bus-port 17000
# General
daemonize yes
supervised systemd
pidfile /var/run/redis/redis-server-7000.pid
loglevel notice
logfile /var/log/redis/redis-server-7000.log
databases 16
# Persistence
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump-7000.rdb
dir /var/lib/redis/7000
# Replication
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-ping-replica-period 10
repl-timeout 60
repl-disable-tcp-nodelay no
repl-backlog-size 1mb
repl-backlog-ttl 3600
# Memory Management
maxmemory 16gb
maxmemory-policy allkeys-lru
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
# Security
requirepass "redis-cluster-password-change-me"
masterauth "redis-cluster-password-change-me"
# TLS (recommended for government environments)
tls-port 6380
tls-cert-file /etc/ssl/certs/redis.crt
tls-key-file /etc/ssl/private/redis.key
tls-ca-cert-file /etc/ssl/certs/ca.crt
tls-protocols "TLSv1.2 TLSv1.3"
Nodes 2 & 3: Duplicate above config with different:
port 7001/port 7002bindIP addressescluster-announce-ippidfileandlogfilepathsdbfilenameanddirpaths
Replica Nodes Configuration
Node 4 (/etc/redis/redis-7003.conf)
# Same base config as master, but add:
port 7003
bind 10.0.1.13
cluster-announce-ip 10.0.1.13
cluster-announce-port 7003
cluster-announce-bus-port 17003
pidfile /var/run/redis/redis-server-7003.pid
logfile /var/log/redis/redis-server-7003.log
dbfilename dump-7003.rdb
dir /var/lib/redis/7003
# Replica-specific settings
replica-priority 100
min-replicas-to-write 1
min-replicas-max-lag 10
2. Systemd Service Configuration
Create systemd service files for each Redis instance:
/etc/systemd/system/redis-cluster@.service
[Unit]
Description=Redis Cluster Instance %i
After=network.target
Documentation=http://redis.io/documentation
[Service]
Type=notify
ExecStart=/usr/bin/redis-server /etc/redis/redis-%i.conf
ExecStop=/usr/bin/redis-cli -p %i shutdown
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
RuntimeDirectory=redis
RuntimeDirectoryMode=0755
# Security
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/redis /var/log/redis
[Install]
WantedBy=multi-user.target
Start Redis Cluster
# Create directories
sudo mkdir -p /var/lib/redis/{7000,7001,7002,7003,7004,7005}
sudo mkdir -p /var/log/redis
sudo chown -R redis:redis /var/lib/redis /var/log/redis
# Start all instances
sudo systemctl enable redis-cluster@7000
sudo systemctl enable redis-cluster@7001
sudo systemctl enable redis-cluster@7002
sudo systemctl enable redis-cluster@7003
sudo systemctl enable redis-cluster@7004
sudo systemctl enable redis-cluster@7005
sudo systemctl start redis-cluster@7000
sudo systemctl start redis-cluster@7001
sudo systemctl start redis-cluster@7002
sudo systemctl start redis-cluster@7003
sudo systemctl start redis-cluster@7004
sudo systemctl start redis-cluster@7005
3. Cluster Initialization
# Initialize cluster (run on any node)
redis-cli -a "redis-cluster-password-change-me" --cluster create \
10.0.1.10:7000 \
10.0.1.11:7001 \
10.0.1.12:7002 \
10.0.1.13:7003 \
10.0.1.14:7004 \
10.0.1.15:7005 \
--cluster-replicas 1
# Verify cluster status
redis-cli -c -h 10.0.1.10 -p 7000 -a "redis-cluster-password-change-me" cluster info
redis-cli -c -h 10.0.1.10 -p 7000 -a "redis-cluster-password-change-me" cluster nodes
4. LLM Shield Integration
Update your production.yaml to use the Redis cluster:
redis:
# Cluster mode
cluster_enabled: true
startup_nodes:
- host: "10.0.1.10"
port: 7000
- host: "10.0.1.11"
port: 7001
- host: "10.0.1.12"
port: 7002
# Authentication
password: "redis-cluster-password-change-me"
# Connection pooling
max_connections: 1000
max_connections_per_node: 100
socket_connect_timeout: 5
socket_timeout: 5
# Retry logic
retry_on_timeout: true
health_check_interval: 30
# TLS (if enabled)
ssl: true
ssl_cert_reqs: "required"
ssl_ca_certs: "/etc/ssl/certs/ca.crt"
ssl_certfile: "/etc/ssl/certs/redis-client.crt"
ssl_keyfile: "/etc/ssl/private/redis-client.key"
# Data partitioning strategy
storage:
# Tenant configs: hash by tenant_id
tenant_config_prefix: "tenant:config:"
# Rate limits: hash by client_ip + tenant
rate_limit_prefix: "rate_limit:"
# Sessions: hash by session_id
session_prefix: "session:"
# Guardrail cache: hash by content_hash
guardrail_cache_prefix: "cache:guardrail:"
# Metrics: time-series data
metrics_prefix: "metrics:"
# TTL settings
default_ttl: 3600 # 1 hour
session_ttl: 86400 # 24 hours
cache_ttl: 300 # 5 minutes
5. Redis Monitoring & Maintenance
Monitoring Script (/opt/llm-shield/scripts/redis-monitor.sh)
#!/bin/bash
REDIS_PASSWORD="redis-cluster-password-change-me"
NODES=("10.0.1.10:7000" "10.0.1.11:7001" "10.0.1.12:7002")
echo "=== Redis Cluster Health Check ==="
echo "Timestamp: $(date)"
for node in "${NODES[@]}"; do
host=$(echo $node | cut -d: -f1)
port=$(echo $node | cut -d: -f2)
echo -e "\n--- Node $node ---"
# Test connectivity
if redis-cli -h $host -p $port -a $REDIS_PASSWORD ping > /dev/null 2>&1; then
echo "✅ Connectivity: OK"
# Memory usage
memory_used=$(redis-cli -h $host -p $port -a $REDIS_PASSWORD info memory | grep used_memory_human | cut -d: -f2 | tr -d '\r')
memory_max=$(redis-cli -h $host -p $port -a $REDIS_PASSWORD config get maxmemory | tail -n1)
echo "📊 Memory: $memory_used / $(numfmt --to=iec $memory_max)"
# Connected clients
clients=$(redis-cli -h $host -p $port -a $REDIS_PASSWORD info clients | grep connected_clients | cut -d: -f2 | tr -d '\r')
echo "👥 Clients: $clients"
# Commands per second
ops=$(redis-cli -h $host -p $port -a $REDIS_PASSWORD info stats | grep instantaneous_ops_per_sec | cut -d: -f2 | tr -d '\r')
echo "⚡ Ops/sec: $ops"
# Cluster status
cluster_state=$(redis-cli -h $host -p $port -a $REDIS_PASSWORD cluster info | grep cluster_state | cut -d: -f2 | tr -d '\r')
echo "🔗 Cluster: $cluster_state"
else
echo "❌ Connectivity: FAILED"
fi
done
echo -e "\n=== Cluster-wide Stats ==="
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD cluster nodes | grep master | wc -l | xargs echo "Master nodes:"
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD cluster nodes | grep slave | wc -l | xargs echo "Replica nodes:"
echo -e "\n=== LLM Shield Key Patterns ==="
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD --scan --pattern "tenant:*" | wc -l | xargs echo "Tenant configs:"
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD --scan --pattern "rate_limit:*" | wc -l | xargs echo "Rate limit entries:"
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD --scan --pattern "session:*" | wc -l | xargs echo "Active sessions:"
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD --scan --pattern "cache:*" | wc -l | xargs echo "Cached results:"
Backup Script (/opt/llm-shield/scripts/redis-backup.sh)
#!/bin/bash
BACKUP_DIR="/opt/backups/redis"
REDIS_PASSWORD="redis-cluster-password-change-me"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
mkdir -p $BACKUP_DIR
echo "Starting Redis cluster backup at $(date)"
# Backup each master node
for port in 7000 7001 7002; do
echo "Backing up Redis node on port $port..."
# Trigger BGSAVE
redis-cli -p $port -a $REDIS_PASSWORD BGSAVE
# Wait for backup to complete
while [ $(redis-cli -p $port -a $REDIS_PASSWORD LASTSAVE) = $(redis-cli -p $port -a $REDIS_PASSWORD LASTSAVE) ]; do
sleep 1
done
# Copy RDB file
cp /var/lib/redis/${port}/dump-${port}.rdb $BACKUP_DIR/dump-${port}-${TIMESTAMP}.rdb
# Compress
gzip $BACKUP_DIR/dump-${port}-${TIMESTAMP}.rdb
echo "✅ Node $port backup complete"
done
# Cleanup old backups (keep last 7 days)
find $BACKUP_DIR -name "*.rdb.gz" -mtime +7 -delete
echo "Redis cluster backup completed at $(date)"
6. Redis Security Hardening
TLS Certificate Generation
# Create CA
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 365 -out ca.crt \
-subj "/C=AE/ST=Dubai/L=Dubai/O=Government/CN=Redis-CA"
# Create Redis server certificate
openssl genrsa -out redis.key 2048
openssl req -new -key redis.key -out redis.csr \
-subj "/C=AE/ST=Dubai/L=Dubai/O=Government/CN=redis.internal.gov"
# Sign server certificate
openssl x509 -req -in redis.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out redis.crt -days 365 -sha256
# Create client certificate for LLM Shield
openssl genrsa -out redis-client.key 2048
openssl req -new -key redis-client.key -out redis-client.csr \
-subj "/C=AE/ST=Dubai/L=Dubai/O=Government/CN=llm-shield-client"
openssl x509 -req -in redis-client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out redis-client.crt -days 365 -sha256
# Set permissions
chmod 600 *.key
chmod 644 *.crt
Firewall Configuration
# Redis cluster ports
sudo firewall-cmd --permanent --add-port=7000-7005/tcp # Redis instances
sudo firewall-cmd --permanent --add-port=17000-17005/tcp # Cluster bus
sudo firewall-cmd --permanent --add-port=6380/tcp # TLS port
sudo firewall-cmd --reload
# Restrict access to specific IPs
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" \
source address="10.0.1.0/24" port protocol="tcp" port="7000-7005" accept'
7. Troubleshooting Redis Issues
Common Problems & Solutions
1. Cluster Split-Brain
# Check cluster state
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD cluster info
# Fix split-brain (if needed)
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD cluster reset soft
redis-cli -c -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD cluster meet 10.0.1.11 7001
2. Memory Issues
# Check memory usage
redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD info memory
# Clear expired keys manually
redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD --scan --pattern "*" | \
xargs -I {} redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD TTL {}
3. Connection Pool Exhaustion
# Monitor connections
watch -n 1 'redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD info clients'
# Increase pool size in production.yaml
redis:
max_connections: 2000
max_connections_per_node: 200
4. Slow Query Detection
# Enable slow log
redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD config set slowlog-log-slower-than 10000
redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD config set slowlog-max-len 128
# View slow queries
redis-cli -h 10.0.1.10 -p 7000 -a $REDIS_PASSWORD slowlog get 10
👥 Multi-Tenant Onboarding
1. Tenant Creation
# Create new tenant via API
curl -X POST http://master-node:8080/v1/admin/tenants \
-H "X-Admin-Key: admin-secure-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"name": "Ministry of Health",
"description": "Healthcare applications tenant",
"quota_requests_per_minute": 200,
"quota_requests_per_day": 50000,
"data_clearance": "confidential",
"allowed_models": ["qwen3.5-9b"],
"rbac_role": "healthcare-agent"
}'
2. API Key Generation
# Generate tenant API keys
curl -X POST http://master-node:8080/v1/admin/tenants/ministry-health/api-keys \
-H "X-Admin-Key: admin-secure-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"name": "healthcare-primary-key",
"expires_at": "2027-12-31T23:59:59Z",
"permissions": ["chat", "classify", "admin"]
}'
# Response includes secure API key
# {
# "api_key": "tenant-health-abc123def456",
# "key_id": "health-001",
# "expires_at": "2027-12-31T23:59:59Z"
# }
3. Tenant Configuration
# Update tenant-specific guardrail policies
curl -X PUT http://master-node:8080/v1/admin/tenants/ministry-health/policies \
-H "X-Admin-Key: admin-secure-key-change-me" \
-H "Content-Type: application/json" \
-d '{
"guardrails": {
"pii_detection": {
"enabled": true,
"action": "redact",
"settings": {
"entities": ["SSN", "MEDICAL_RECORD_NUMBER", "PATIENT_ID"]
}
},
"topic_enforcement": {
"enabled": true,
"action": "block",
"settings": {
"allowed_topics": ["medical", "healthcare", "diagnosis"],
"blocked_topics": ["politics", "religion"]
}
}
},
"rbac": {
"max_tokens_per_request": 8192,
"rate_limit": "200/min",
"allowed_tools": [
"search_medical_database",
"get_patient_info",
"schedule_appointment"
]
}
}'
🔧 Worker Configuration
Systemd Service Setup
Create Service File (/etc/systemd/system/llm-shield.service)
[Unit]
Description=LLM Shield Guardrails Service
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/opt/llm-shield
ExecStart=/usr/bin/docker-compose -f docker-compose.worker.yml up -d
ExecStop=/usr/bin/docker-compose -f docker-compose.worker.yml down
TimeoutStartSec=300
[Install]
WantedBy=multi-user.target
Enable Service
sudo systemctl enable llm-shield.service
sudo systemctl start llm-shield.service
Load Balancer Configuration
HAProxy Setup (/etc/haproxy/haproxy.cfg)
global
daemon
maxconn 4096
defaults
mode http
timeout connect 5000ms
timeout client 60000ms
timeout server 60000ms
# LLM Shield API Load Balancer
frontend llm_shield_frontend
bind *:443 ssl crt /etc/ssl/certs/llm-shield.pem
bind *:80
redirect scheme https if !{ ssl_fc }
default_backend llm_shield_workers
backend llm_shield_workers
balance roundrobin
option httpchk GET /health
# Worker node backends
server worker-1 worker-1.internal:8080 check inter 2s fall 3 rise 2
server worker-2 worker-2.internal:8080 check inter 2s fall 3 rise 2
server worker-3 worker-3.internal:8080 check inter 2s fall 3 rise 2
server worker-4 worker-4.internal:8080 check inter 2s fall 3 rise 2
# Admin interface (restricted access)
frontend admin_frontend
bind *:8443 ssl crt /etc/ssl/certs/admin.pem
acl admin_network src 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16
http-request deny unless admin_network
default_backend admin_backend
backend admin_backend
server master master-node.internal:8080 check
📊 Monitoring & Observability
Health Checks
System Health Script (/opt/llm-shield/scripts/health-check.sh)
#!/bin/bash
# Check GPU status
echo "=== GPU Status ==="
nvidia-smi --query-gpu=name,memory.used,memory.total,utilization.gpu --format=csv
echo -e "\n=== MIG Instances ==="
nvidia-smi -L
echo -e "\n=== Service Status ==="
systemctl status llm-shield.service
echo -e "\n=== Container Status ==="
docker ps --filter "name=llm-shield"
echo -e "\n=== API Health ==="
for worker in worker-1 worker-2 worker-3 worker-4; do
echo "Testing $worker..."
curl -s -o /dev/null -w "%{http_code} - %{time_total}s\n" \
http://$worker:8080/health
done
echo -e "\n=== Redis Connectivity ==="
redis-cli -h master-node ping
echo -e "\n=== Recent Errors ==="
tail -n 20 /opt/llm-shield/logs/error.log
Prometheus Metrics
# Add Prometheus endpoint to guardrails
curl http://master-node:8080/metrics
# Key metrics exposed:
# - guardrail_requests_total{guardrail, action, tenant}
# - guardrail_latency_seconds{guardrail, tenant}
# - gpu_memory_usage_bytes{worker_id, mig_instance}
# - tenant_quota_usage{tenant, resource_type}
# - active_connections_total
🔒 Security Hardening
SELinux Policies
# Create custom SELinux policy for containers
sudo setsebool -P container_manage_cgroup true
sudo setsebool -P virt_use_execmem true
# Custom policy for GPU access
cat > llm_shield.te << 'EOF'
policy_module(llm_shield, 1.0)
type llm_shield_t;
type llm_shield_exec_t;
domain_type(llm_shield_t)
files_type(llm_shield_exec_t)
# Allow GPU device access
dev_read_sysfs(llm_shield_t)
dev_rw_generic_chr_files(llm_shield_t)
# Allow network access
corenet_tcp_connect_http_port(llm_shield_t)
corenet_tcp_bind_http_port(llm_shield_t)
EOF
# Compile and install
checkmodule -M -m -o llm_shield.mod llm_shield.te
semodule_package -o llm_shield.pp -m llm_shield.mod
sudo semodule -i llm_shield.pp
TLS Configuration
# Generate certificates
openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/private/llm-shield.key \
-out /etc/ssl/certs/llm-shield.crt -days 365 -nodes \
-subj "/C=AE/ST=Dubai/L=Dubai/O=Government/CN=llm-shield.internal.gov"
# Update configuration for TLS
# Add to production.yaml:
server:
ssl:
enabled: true
cert_file: "/etc/ssl/certs/llm-shield.crt"
key_file: "/etc/ssl/private/llm-shield.key"
🚨 Troubleshooting
Common Issues
1. GPU Memory Issues
# Check GPU memory usage
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# Clear GPU memory
sudo systemctl restart llm-shield.service
# Adjust MIG memory allocation
sudo nvidia-smi mig -dci # Delete compute instances
sudo nvidia-smi mig -dgi # Delete GPU instances
# Recreate with different sizes
2. Model Loading Failures
# Check model file integrity
sha256sum /opt/llm-shield/models/*.gguf
# Verify model permissions
ls -la /opt/llm-shield/models/
chown -R nobody:nobody /opt/llm-shield/models/
# Check container logs
docker logs llm-shield-worker-0
3. Redis Connectivity
# Test Redis connection
redis-cli -h master-node ping
# Check Redis logs
docker logs redis-container
# Verify firewall rules
sudo firewall-cmd --list-ports
4. Performance Issues
# Monitor GPU utilization
watch -n 1 nvidia-smi
# Check CPU usage
htop
# Test API latency
curl -w "@curl-format.txt" -s -o /dev/null \
http://worker-1:8080/health
# Where curl-format.txt contains:
# time_namelookup: %{time_namelookup}\n
# time_connect: %{time_connect}\n
# time_appconnect: %{time_appconnect}\n
# time_pretransfer: %{time_pretransfer}\n
# time_redirect: %{time_redirect}\n
# time_starttransfer: %{time_starttransfer}\n
# ----------\n
# time_total: %{time_total}\n
📋 Pre-Deployment Checklist
Infrastructure Requirements
- RedHat Enterprise Linux 8.8+ installed
- NVIDIA H200 GPUs installed and detected
- MIG mode enabled and configured
- Container runtime (Podman/Docker) installed
- NVIDIA Container Toolkit configured
- Network connectivity between nodes verified
- Firewall rules configured
- SSL certificates generated
Software Requirements
- LLM Shield codebase deployed
- Model files downloaded/transferred
- Configuration files customized
- Tenant configurations prepared
- Monitoring stack deployed
- Backup procedures established
Security Requirements
- SELinux policies applied
- TLS encryption enabled
- API keys generated securely
- Admin access restricted
- Audit logging configured
- SIEM integration tested
Performance Testing
- Load testing completed
- Latency benchmarks verified
- GPU utilization optimized
- Memory usage validated
- Throughput targets met
- Failover scenarios tested
📞 Support Contacts
Technical Support
- Infrastructure Team: infra@internal.gov
- Security Team: security@internal.gov
- AI/ML Team: aiml@internal.gov
Emergency Contacts
- On-Call Engineer: +971-XXX-XXXX
- System Administrator: admin@internal.gov
- CISO Office: ciso@internal.gov
📚 Additional Resources
- LLM Shield API Reference
- Multi-Tenant Architecture Guide
- Agentic Guardrails Guide
- Integration Guide
- Compliance Mapping
Portal SSO and your internal CA
The tenant portal can authenticate humans against your own IdP over OIDC. This section is about the part that is specific to running behind a corporate PKI, because that is where an otherwise-correct setup fails with an error that does not say why.
Do you need a CA bundle at all?
Run this against your IdP from a host on the same network:
openssl s_client -connect keycloak.internal:443 -servername keycloak.internal </dev/null 2>/dev/null | grep "Verify return code"
Verify return code: 0 (ok) means skip this entire section. Your IdP uses
a publicly trusted CA and Shield needs no extra configuration.
Any other code (19, 20, 21) means the certificate chains to a CA that the
default trust store does not know, which is the normal situation for an
internal Keycloak, ADFS or an internal Entra endpoint.
Why the default does not work
Shield’s outbound HTTP client builds its trust store from a bundled CA list,
not from the operating system’s. A certificate your servers trust everywhere
else will still be rejected here. Setting SSL_CERT_FILE is not a reliable
workaround: Python silently ignores a missing file and falls back to the
default store, so a typo becomes a certificate error that mentions no path.
Shield therefore takes the bundle explicitly and validates it when it is read.
Getting the bundle
Ask whoever runs your PKI for the internal root CA certificate. It is the same file already distributed to your servers, usually one of:
ls /usr/local/share/ca-certificates/ /etc/pki/ca-trust/source/anchors/ 2>/dev/null
Include the root, not only the intermediate. An intermediate alone does not verify, and this is the most common mistake.
For a lab or a proof of concept you can take the chain from the server itself:
openssl s_client -showcerts -connect keycloak.internal:443 -servername keycloak.internal </dev/null 2>/dev/null | awk '/BEGIN CERT/,/END CERT/' > internal-ca.pem
Understand what that does before using it in production: you are trusting the certificate presented by the server you are trying to verify. Anyone able to intercept that connection can hand you their own CA and you will install it.
Verify the bundle before configuring Shield
This separates “wrong CA file” from “Shield misconfigured”, which are otherwise indistinguishable from the logs:
openssl s_client -connect keycloak.internal:443 -servername keycloak.internal -CAfile internal-ca.pem </dev/null 2>&1 | grep "Verify return code"
You need 0 (ok). If not, the file is the wrong CA and no amount of Shield
configuration will help.
Configure Shield
Two ways, depending on whether your platform can mount files.
A mounted file, for Docker, Podman, Kubernetes and systemd:
docker run -p 8080:8080 \
-e SHIELD_OIDC_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem \
-v /etc/pki/internal-ca.pem:/etc/ssl/certs/internal-ca.pem:ro \
shield-admin
Inline, for platforms that only offer environment variables:
SHIELD_OIDC_CA_PEM="-----BEGIN CERTIFICATE-----
MIIDxTCCAq2gAwIBAgIBADANBg...
-----END CERTIFICATE-----"
If both are set the mounted path wins, on the assumption that an operator who mounted a file meant it.
Confirm the container can read it
The failure that wastes an afternoon is a bundle present on the host but unreadable by the container user:
docker run --rm \
-e SHIELD_OIDC_CA_BUNDLE=/etc/ssl/certs/internal-ca.pem \
-v /etc/pki/internal-ca.pem:/etc/ssl/certs/internal-ca.pem:ro \
shield-admin \
python3 -c "from core.oauth.tls_trust import httpx_verify; print('bundle OK:', httpx_verify())"
Printing the path means Shield will use it. Anything else names the specific problem: missing, a directory, or not readable.
Rotation
The CA bundle is a trust anchor, not a certificate. It does not follow your server certificate’s renewal cycle, and this distinction is what makes the operational answer simple:
| Event | Update the bundle? |
|---|---|
| IdP’s TLS certificate renewed, same CA | No. This is the point of a CA. |
| IdP moved to a different hostname | No, provided the CA is unchanged |
| Internal intermediate CA re-issued | Yes |
| Internal root CA re-issued or rolled over | Yes |
| Migrating the IdP to a public CA | No — remove the setting instead |
Root CAs typically live 10 to 20 years; intermediates 3 to 5. The realistic trigger is an intermediate rotation, and the usual symptom is that logins keep working for existing sessions and fail for new ones, because the JWKS fetch is what breaks.
Two things make this survivable:
- Include the root, not just the intermediate. A bundle anchored at the root keeps verifying when the intermediate is re-issued from it.
- Rotation in place needs no restart. Shield keys its cached TLS context on
the file’s identity, so replacing the file — cert-manager, a config-map
remount, an Ansible run — takes effect on the next outbound call. A bundle
supplied inline via
SHIELD_OIDC_CA_PEMis materialised once per process, so changing that variable does require a restart, which platforms that only offer environment variables do anyway.
Track your CA’s expiry alongside your other certificates:
openssl x509 -in internal-ca.pem -noout -subject -enddate
Air-gapped deployments
Nothing in the OIDC path contacts a vendor endpoint. The only hosts reached are the issuer configured for that tenant, for discovery and JWKS. There is a test asserting that no module in this path references a vendor domain, because a hardcoded URL would break an air-gapped install and no functional test would catch it.
Requirements are therefore only: the IdP is reachable from Shield, its CA is trusted as above, and clocks are within the token validation leeway.
Document Version: 1.0
Last Updated: April 6, 2026
Author: LLM Shield Engineering Team
Review Cycle: Quarterly