Horizontal Scaling

Published March 2026

The Single-Process Ceiling

The TSX stress test proved AgDR at 100 million decisions, reaching a ceiling of approximately 2.85 billion decisions in a single process (memory-bound). At 253,807 decisions per second, that ceiling is reached in roughly 3.1 hours of sustained operation.

For high-velocity environments such as trading desks, clinical systems, government adjudication at scale, and autonomous vehicle fleets, the answer is horizontal scaling: multiple AgDR processes, each maintaining its own atomic kernel chain, coordinated through a Root Merkle Tree that provides a single, unified, court-provable record of the entire distributed operation.

Architecture

                    ┌─────────────────────────────────────┐
                    │        Root Merkle Coordinator       │
                    │   (aggregates shard roots hourly)    │
                    │   Produces: Global Merkle Root        │
                    └──────┬──────────┬──────────┬─────────┘
                           │          │          │
              ┌────────────┘  ┌───────┘  ┌───────┘
              ▼               ▼           ▼
        ┌──────────┐   ┌──────────┐   ┌──────────┐
        │  Shard 0 │   │  Shard 1 │   │  Shard N │
        │  AKI     │   │  AKI     │   │  AKI     │
        │  Process │   │  Process │   │  Process │
        └──────────┘   └──────────┘   └──────────┘
        0–2.85B        0–2.85B        0–2.85B
        decisions      decisions      decisions

Each shard is an independent AgDR process with its own:

• Kernel atomic section
• Merkle chain (forward-secret)
• Ed25519 signing key

The Root Merkle Coordinator aggregates shard roots on a defined interval and produces a Global Merkle Root. This single hash commits to every decision across all shards.

Shard Assignment

Decisions are assigned to shards by a deterministic routing function. The routing function must be:

1. Deterministic: the same decision always routes to the same shard
2. Auditable: the routing decision is itself logged
3. Collision-free for related decisions: decisions that must appear in the same evidentiary chain, for example all decisions for a single trade, must route to the same shard

Routing strategies:

Root Merkle Coordinator

The coordinator runs on a configurable interval (default: every 60 seconds) and:

1. Requests the current Merkle root from each live shard
2. Assembles the shard roots into a Root Merkle Tree
3. Signs the Global Merkle Root with the coordinator's Ed25519 key
4. Appends the Global Root to the permanent audit log
5. Publishes the Global Root (optionally to a public transparency log)

from agdr_aki.coordinator import RootMerkleCoordinator

coordinator = RootMerkleCoordinator(
    shards=["shard-0:50051", "shard-1:50052", "shard-2:50053"],
    aggregation_interval_s=60,
    signing_key_path="coordinator_key.pem",
    output_log="global_merkle_roots.log",
)
coordinator.start()

Global Root record structure:

{
  "global_root_id": "gr_20260315_094200",
  "timestamp_ns": 1742000000000000000,
  "interval_start_ns": 1741999940000000000,
  "interval_end_ns": 1742000000000000000,
  "shard_roots": [
    { "shard_id": "shard-0", "root": "a8f3c1...", "record_count": 15228441 },
    { "shard_id": "shard-1", "root": "b2e9d4...", "record_count": 15191033 },
    { "shard_id": "shard-2", "root": "c7f1a8...", "record_count": 15384526 }
  ],
  "global_merkle_root": "9e2b7f...",
  "total_records_in_interval": 45803000,
  "signature": "<Ed25519 signature of global_merkle_root>"
}

Throughput at Scale

Each shard runs at the single-process rate. With N shards:

Horizontal scaling is bounded only by network and storage, not by the AKI algorithm itself.

Evidentiary Properties at Scale

The horizontal architecture preserves all evidentiary guarantees of the single-process model:

Deployment Considerations

Shard key management

Each shard has its own Ed25519 signing key. The coordinator has its own key. All keys must be:

• Generated securely at deployment
• Backed up in a key management system, for example HashiCorp Vault or AWS KMS
• Rotated on a defined schedule
• Never shared between shards

FOI assignment in horizontal deployments

FOI designation is per decision class, not per shard. A single FOI may cover decisions across all shards in their designated scope. The routing function ensures escalated decisions from any shard reach the correct FOI through the same escalation path.

Network failure handling

If a shard is unreachable during root aggregation:

• The coordinator logs the missing shard
• The global root is computed from available shards
• The missing interval is flagged for reconstruction when the shard comes back online
• No decisions are dropped. The shard's local chain remains intact.

Storage

At 253,807 decisions per second per shard, with a typical record size of approximately 2 KB:

• Single shard: approximately 500 MB per second. Plan for approximately 43 TB per day.
• Use tiered storage: hot (NVMe) for recent records, cold (object storage) for archive
• The Merkle chain structure allows cold records to be verified without loading into memory

Part of the AgDR v1.8 foundational standard
Canonical source: github.com/aiccountability-source/AgDR