Running Crystal in Cluster Mode¶

Crystal is RAM-resident: one process holds its whole graph in memory, and disk provides durability rather than capacity (see ADR 0001). Scaling out therefore means running more processes, and there are two ways to do it that solve different problems.

They compose. A typical production layout uses both.

Sharding by graph¶

One process serves several named graphs, and graphs are completely isolated — no query, traversal or consistency check crosses between them. Route each graph to whichever node has room.

flowchart LR
    C[Clients] --> R[Router / ingress]
    R -->|case-a, case-b| N1[crystal-1]
    R -->|case-c| N2[crystal-2]
    R -->|case-d, case-e| N3[crystal-3]

This fits investigation and narrative workloads almost exactly, because those are naturally partitioned: one investigation, one case or one story is one graph. Capacity then scales with node count.

Creating a graph¶

curl -XPOST -H "Authorization: Bearer $CRYSTAL_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"name":"case-2024-07"}' \
     https://crystal-1:7540/api/v1/graphs

Every graph-scoped endpoint then exists in two forms:

# the default graph
curl -H "Authorization: Bearer $CRYSTAL_TOKEN" \
     https://crystal-1:7540/api/v1/vertices

# a named graph
curl -H "Authorization: Bearer $CRYSTAL_TOKEN" \
     https://crystal-1:7540/api/v1/graphs/case-2024-07/vertices

Names must match ^[a-z0-9][a-z0-9_-]{0,62}$.

Sizing a node¶

Every graph on a node is resident in full, so a node's memory cost is the sum of its graphs:

live heap ≈ Σ (745 B × |V| + 626 B × |E|)

and budget RAM at roughly twice that for garbage-collector headroom. Set storage.max_graphs per node — without it, the create-graph API is a way to walk into the memory limit one namespace at a time.

{
  "storage": {
    "data_dir": "/var/lib/crystal",
    "default_graph": "default",
    "max_graphs": 16
  }
}

Creating a graph past the limit returns 507 Insufficient Storage, not 400: the request is well formed, the server is out of room.

Routing¶

Crystal does not route between nodes; the graph-to-node mapping lives in your ingress. Because the graph name is in the path, ordinary path-prefix routing is enough:

location ~ ^/api/v1/graphs/(case-a|case-b)/ { proxy_pass http://crystal-1:7540; }
location ~ ^/api/v1/graphs/(case-c)/        { proxy_pass http://crystal-2:7540; }

Keep the mapping in one place and treat it as configuration. Moving a graph between nodes is an export, an import and a routing change — there is no automatic rebalancing.

On isolation¶

Graph isolation is enforced in the storage layer, so it holds for traversals and queries as well as direct reads. Authentication, however, is not yet per-graph: a read token reads every graph on the node it can reach. Until that lands, treat one Crystal process as one trust boundary and run separate processes for tenants that must not see each other.

Read replicas¶

A follower consumes the primary's transaction stream and serves reads.

flowchart LR
    W[Writers] --> P[(primary)]
    P -.->|WAL stream| R1[(replica 1)]
    P -.->|WAL stream| R2[(replica 2)]
    RD[Readers] --> R1
    RD --> R2

Starting a follower¶

export CRYSTAL_AUTH_TOKEN=$(openssl rand -hex 32)   # this node's own tokens
export CRYSTAL_REPLICATION_TOKEN=<read token on the primary>

./crystal --data-dir /var/lib/crystal-replica \
          --replica-of https://primary:7540

The replication token only needs read scope — replication reads the primary, it does not write to it.

To follow one named graph rather than the primary's default, set replication.graph in the config file.

What a follower does¶

On start it compares its own log position with the primary's:

• empty store — bootstraps from GET /replication/snapshot, a consistent full copy plus the LSN describing it
• existing store — resumes from its own persisted LSN, so a restart does not re-copy the graph
• too far behind — if the primary has already checkpointed away the transactions it needs, it re-bootstraps automatically

It then holds GET /replication/stream open and applies transactions as they commit.

Followers refuse writes¶

Every mutating request to a follower returns 403. This is not a safety rail that can be turned off: a follower's data is defined entirely by the primary's stream, so a local write would either be silently overwritten by the next transaction touching the same entity, or survive as divergence that nothing ever reconciles.

Monitoring replication¶

curl -H "Authorization: Bearer $TOKEN" https://primary:7540/api/v1/replication/status
{"lsn":4210,"snapshot_lsn":4096}

Compare lsn on the primary with lsn on each follower; the difference is the lag in transactions. A follower whose lag is growing is either under-resourced or losing its connection — check its logs for reconnection messages.

snapshot_lsn is the oldest transaction still in the primary's log. A follower whose position falls below it must re-bootstrap, which for a large graph is expensive. Widen that window on the primary by raising max_wal_bytes and flush_interval.

Consistency model¶

Replication is asynchronous.

• A follower lags the primary by however long it takes to ship and apply the last transaction.
• Read-your-writes holds on the primary only. A client that writes to the primary and immediately reads from a follower may not see its own write. If your application needs that guarantee, route those reads to the primary.
• Each follower applies transactions in the primary's order, so a follower is always a consistent past state of the graph — never a mixture.
• There is no consensus and no automatic failover.

Failover is manual, on purpose¶

Crystal will not promote a follower by itself. Automatic promotion without consensus is how split-brain happens: two nodes both believing they are primary, both accepting writes, with no way to reconcile afterwards. Refusing to offer failover is better than offering one that can lose acknowledged writes.

Promotion runbook

1. Stop writes to the old primary. If it is reachable, stop the process. If it is not, make certain it cannot come back and receive traffic — a fenced node that returns later and accepts writes is the failure this step exists to prevent.
2. Pick the follower with the highest LSN (/api/v1/replication/status). Any transactions the old primary committed above that LSN are lost; this is the cost of asynchronous replication and the number to record in your incident notes.
3. Restart it without --replica-of. It becomes writable.
4. Repoint clients and the remaining followers at the new primary.
5. Rebuild the old node as a follower when it returns. Do not restart it as a primary — its log has diverged.

Combining both¶

For a large multi-tenant deployment, shard by graph and replicate each shard:

flowchart TB
    subgraph S1["Shard 1 — case-a, case-b"]
        P1[(primary)] -.-> R1[(replica)]
    end
    subgraph S2["Shard 2 — case-c, case-d"]
        P2[(primary)] -.-> R2[(replica)]
    end
    I[Ingress] --> P1
    I --> R1
    I --> P2
    I --> R2

Route writes for a graph to its shard's primary and reads to either the primary or its replicas. Note that a follower replicates one graph, so a sharded node hosting several graphs needs one follower process per graph it replicates. That is a real limitation of the current implementation and worth weighing when choosing how many graphs to put on a node.

What is not solved¶

• Capacity for a single graph — bounded by one machine's RAM.
• Automatic failover — manual, see above.
• Rebalancing — moving a graph between nodes is manual.
• Per-graph authentication — tokens are per-process, not per-graph.
• Cross-graph queries — a query addresses exactly one graph.

See the service levels page for what is and is not guaranteed.