Secure On-Premise RAG: Intelligent Search for Enterprise FinTech (Air-Gapped)

Executive Summary

This document presents the architectural validation and deployment results of an enterprise RAG system (Retrieval-Augmented Generation) within an air-gapped perimeter of a Top-20 bank. The system enables intelligent search across 100,000+ internal documents without transmitting data to external clouds.

Key Business Outcomes:

• Privacy: 100% of data remains within the perimeter — regulatory compliance achieved
• Accuracy: 92% Faithfulness (RAGAS) — hallucination minimization is critical for compliance
• Performance: TTFT < 0.4 sec (instant UI response), E2E 3–5 sec for detailed legal queries on proprietary GPU infrastructure (NVIDIA A100)
• Security: Active Directory integration — users only see documents they're authorized to access

Market Context: Gartner forecasts that by 2026, 80% of enterprise organizations will migrate from public LLM APIs to self-hosted solutions due to regulatory requirements.

1. Problem Statement: Public LLM Risks

1.1 Initial State

The bank utilized full-text search (Elasticsearch) across its corporate knowledge base. When querying "What is the transfer limit for corporate accounts?", the system returned 200+ document links — employees spent up to 30 minutes locating the correct regulation.

1.2 Why OpenAI/Anthropic Cannot Be Used

Table 1. Regulatory Constraints for Financial Sector

1.3 Limitations of Pure Vector Search

Table 2. Dense Retrieval Constraints

Conclusion: Banking documents require Hybrid Search (BM25 + Vectors).

2. Architectural Decisions

2.1 Self-Hosted Inference Stack

Fig. 1. Secure RAG Architecture. The red dashed boundary denotes the Air-Gapped perimeter. All ML components are deployed on the bank's proprietary servers.

2.2 Technology Stack Rationale

2.2.1 LLM Selection: Llama 3 vs Alternatives

Table 3. Self-Hosted LLM Comparison

2.2.2 Vector DB Selection: Qdrant vs Alternatives

Table 4. Vector Database Comparison

2.2.3 Inference Engine: vLLM vs Alternatives

Table 5. Inference Server Comparison

3. Reliability and Accuracy Mechanisms

3.1 Hybrid Search Pipeline

Fig. 2. Hybrid Search Pipeline. ACL filter is applied at the retrieval stage, not post-filtering — critical for security and performance.

Key Mechanism: ACL filter is applied at the index level (pre-filtering), not after obtaining results. This guarantees that users physically cannot retrieve documents without appropriate access.

async def secure_hybrid_search(query: str, user_groups: list[str], top_k: int = 20):
    # ACL Filter at index level
    acl_filter = models.Filter(
        must=[models.FieldCondition(
            key="allowed_groups",
            match=models.MatchAny(any=user_groups)
        )]
    )
    
    # Dense + Sparse search with pre-filtering
    dense_results = await qdrant_client.search(
        collection_name="bank_docs",
        query_vector=await embed_query(query),
        query_filter=acl_filter,
        limit=top_k
    )
    
    # RRF + Reranking
    merged = reciprocal_rank_fusion(dense_results, sparse_results, k=60)
    return await reranker.rerank(query, merged[:20])[:3]

3.2 Reranking for Accuracy Enhancement

Bi-Encoder Problem: Embedding models (E5, BGE) are fast but less accurate — query and document are embedded independently.

Cross-Encoder Solution: BGE-Reranker processes the (query, document) pair together, yielding better relevance understanding.

Table 6. Reranking Impact on Metrics

3.3 Structured Citations (Hallucination Prevention)

To minimize hallucinations, a strict system prompt with rules is used:

• Answers only based on provided context
• Mandatory source citation in format [Document: name, section X.X]
• Prohibition on inferring information

Output is structured via Pydantic models: response, list of citations with document/section/page, and confidence score based on semantic similarity.

4. Results and Metrics

4.1 Comparative Analysis

Table 7. Key Metrics: OpenAI Wrapper vs Secure RAG

4.2 Economic Impact

Table 8. TCO Analysis (3 years)

¹ Compliance Risk = BLOCKING: Risk of regulatory sanctions, fines up to 6% of revenue under GDPR. TCO calculation is based on GPT-4 class models (High Intelligence). Using simplified models (gpt-4o-mini and analogues) in FinTech is unacceptable due to critical hallucination risk when working with legal and financial data.

4.3 Business Results

• Search Time: from 30 minutes to 15 seconds (120x faster)
• L2 Support: ticket processing time reduced by 40%
• Compliance: successfully passed security audit and Penetration Test
• Adoption: 3,500 active users within 3 months

5. Infrastructure and Scaling

5.1 Hardware Requirements

Table 9. Server Specifications

5.2 Kubernetes Deployment

For high availability in the banking perimeter, vLLM is deployed with a minimum of 2 replicas (SPOF elimination). Horizontal scaling is configured on vllm_requests_running metric with a threshold of 10 concurrent requests.

6. Conclusions and Recommendations

This deployment confirmed that Self-Hosted RAG based on Llama 3, Qdrant, and vLLM is the only path to enterprise-grade AI search in regulated industries.

Key Takeaways:

1. Self-Hosted LLM is mandatory for GDPR compliance and financial regulations
2. Hybrid Search (BM25 + Vectors) is critical for accurate search across structured documents
3. Reranking improves accuracy by 30-50% — a mandatory component
4. ACL Pre-Filtering must be at retrieval level, not post-processing
5. vLLM with PagedAttention provides production-grade throughput

Recommendation: This architecture is applicable to any organization with data residency requirements: government sector, healthcare, defense, telecom.