High-Load Loyalty Engine: Processing 10M+ Transactions Daily

Executive Summary

This document presents the architectural validation and development results of a loyalty program transactional core for a federal electronics retail chain (500+ stores). The system processes 10M+ operations daily with ACID financial guarantees.

Softenq team served as a dedicated R&D core (Core Engineering Unit) — we designed and implemented transactional logic, not just "forms".

Key Business Results:

• Performance: ~1,500 TPS at peak (Black Friday), 12,000 TPS — confirmed capacity (10× headroom)
• Reliability: 0% data loss over 2 years of operation
• Fraud Prevention: Double Spending blocking in real-time
• Financial Accuracy: 100% reconciliation with accounting (Double-Entry Ledger)

Market Context: According to Gartner, 1 hour of POS system downtime in retail costs $100k-$500k. Incorrect bonus accruals lead to lawsuits and reputational losses.

1. Problem Statement: Monolithic Architecture Limits

1.1 Initial State

The customer used a loyalty module based on 1C-Bitrix. During Black Friday, the monolithic architecture reached its vertical scaling limit, causing mass failures at cash registers. Fraud cases were recorded: users redeemed the same points twice through App and Web simultaneously.

1.2 Legacy System Technical Limitations

Table 1. Monolithic Solution Problem Matrix

1.3 Threat Model: Double Spending

Attack Scenario (Race Condition):
1. User: balance 1000 points
2. T0: Request from App: "Redeem 800"
3. T1: Request from Web: "Redeem 800" (parallel)
4. T2: App reads balance: 1000 ✓
5. T3: Web reads balance: 1000 ✓ (not updated yet)
6. T4: App writes: 1000 - 800 = 200
7. T5: Web writes: 1000 - 800 = 200 (overwrites!)
 
Result: 1600 points redeemed with 1000 balance

Conclusion: "Current balance" architecture without locking allows fraud. Transactional core required.

2. Architectural Solutions

2.1 Double-Entry Ledger

We rejected the "current balance" model in favor of Immutable Ledger Architecture with accounting double-entry pattern.

Key Principles:

• Every balance change creates two entries: debit and credit
• Balance is calculated as sum of all ledger entries
• Entries are immutable — no updates, only appends
• Concurrent operations handled via pessimistic locking

2.2 Sharding Strategy (Citus)

To achieve horizontal scalability, we implemented PostgreSQL sharding via Citus:

• Shard Key: customer_id (co-located with transactions)
• Distribution: consistent hashing across 32 shards
• Replication: synchronous replication for durability

2.3 Double Spending Protection

Protected Transaction Flow:
1. SELECT ... FOR UPDATE on customer row (pessimistic lock)
2. Validate balance >= redemption amount
3. INSERT debit entry
4. INSERT credit entry
5. COMMIT (releases lock)
 
Concurrent request waits at step 1 until lock released

3. Results and Metrics

3.1 Performance Improvements

3.2 Financial Impact

The transactional core eliminated fraud losses (~$600K/year) and ensured 100% reconciliation with accounting systems, providing full auditability for financial compliance.