Architecture Overview¶

Crystal is a property graph database built as a single Go binary with an embedded-first design. It can run as a standalone server with a REST API or be imported directly as a Go library.

Design Goals¶

• Zero external dependencies for the core engine
• Embedded-first (like SQLite for graphs)
• Suitable for investigation mapping and narrative design workflows
• Simple persistence with crash-safe writes

Layer Diagram¶

graph TD
  A["Public API (pkg/)<br/><small>Go library interface</small>"]
  B["REST API (api/http)<br/><small>HTTP server, handlers, middleware</small>"]
  C["Embedded Client (api/)<br/><small>Programmatic access layer</small>"]
  D["Query Engine (query/)<br/><small>Fluent query builder, executor</small>"]
  E["Traversal (traversal/)<br/><small>BFS, DFS, shortest path, all paths</small>"]
  F["Storage Interface (core)<br/><small>Storage trait in core/types.go</small>"]
  G["Memory"]
  H["Disk"]
  I["future: LSM, mmap"]

  A --> B --> C --> D --> E --> F
  F --> G
  F --> H
  F -.-> I

Package Structure¶

internal/core/¶

Defines the fundamental types: Vertex, Edge, Properties, Storage interface, and StorageStats. All other packages depend on core; core depends on nothing.

internal/storage/memory/¶

In-memory storage backend using Go maps with sync.RWMutex for concurrent access. Data lives only in process memory. Useful for testing, short-lived sessions, or when persistence is not needed.

internal/storage/disk/¶

File-based persistent storage with Write-Ahead Logging (WAL) for crash safety.

On-disk layout in the data directory:

• graph.json -- full snapshot of all vertices and edges
• wal.bin -- binary WAL file with framed mutation records

The adjacency index (outEdges / inEdges maps) is rebuilt from the edge list on load, so the on-disk format only stores vertices and edges.

internal/query/¶

Fluent query builder with filtering, pagination, and sorting. The executor resolves queries against the storage backend. Supports three query types:

• Vertex queries: filter by type, labels, properties
• Edge queries: filter by type, properties
• Neighbor queries: multi-hop traversal from a starting vertex

internal/traversal/¶

Graph traversal algorithms:

• BFS: breadth-first search with configurable max depth, direction, and edge type filtering. Also provides shortest path computation.
• DFS: depth-first search with the same configuration. Also provides all-paths enumeration between two vertices.

internal/api/http/¶

HTTP server using only the Go standard library (net/http). Includes:

• A custom pattern-matching router (api/router/)
• Middleware for logging, CORS, and panic recovery
• RESTful handlers for vertices, edges, graph operations, queries, and traversals

internal/api/embedded/¶

Direct programmatic access to all graph operations without going through HTTP. Used by the public pkg/crystal library.

internal/config/¶

JSON-based configuration with environment variable overrides and CLI flag support.

pkg/crystal/¶

The public API surface. External Go projects import this package to use Crystal as an embedded database.

Data Model¶

Crystal uses a property graph model:

• Vertices have an ID, a type string, a set of labels, and a property map.
• Edges have an ID, source/target vertex IDs, a type string, a property map, a directed flag, and an optional weight.
• Properties are map[string]any allowing flexible schema.

Concurrency Model¶

Both storage backends use sync.RWMutex:

• Multiple concurrent reads are allowed
• Writes acquire an exclusive lock
• The HTTP server handles requests concurrently via goroutines

Persistence Strategy¶

The disk backend combines Write-Ahead Logging with periodic snapshots for crash-safe persistence.

Write Path¶

Every mutation (PutVertex, DeleteVertex, PutEdge, DeleteEdge) follows this sequence:

1. Acquire write lock
2. Append a binary WAL record to wal.bin and fsync
3. If the WAL append fails, return error without modifying in-memory state
4. Apply the change to in-memory maps
5. Release lock

This guarantees that any acknowledged write is durable on disk before the caller sees success.

WAL Record Format¶

Each WAL record uses a binary frame with integrity checking:

block-beta
  columns 6
  a["Magic<br/>2B"]
  b["OpType<br/>1B"]
  c["EntType<br/>1B"]
  d["DataLen<br/>4B"]
  e["Data<br/>N bytes"]
  f["CRC32<br/>4B"]

• Magic: 0xCF 0xA1 (corruption sentinel)
• OpType: Put (0x01) or Delete (0x02)
• EntType: Vertex (0x01) or Edge (0x02)
• Data: JSON-encoded entity for Put, raw ID bytes for Delete
• CRC32: IEEE checksum over all preceding bytes in the record

Checkpointing¶

A background goroutine runs on a configurable interval (flush_interval, default 30s):

1. Acquire write lock
2. Serialize the full graph state to JSON
3. Write to graph.json.tmp, atomically rename to graph.json
4. Truncate wal.bin
5. Release lock

Explicit Flush() calls (via API or shutdown) perform the same operation.

Recovery¶

On startup, the disk backend:

1. Loads the last graph.json snapshot (if it exists)
2. Opens the WAL file
3. Replays all valid WAL records on top of the snapshot state
4. Discards any torn trailing record (partial write from a crash)
5. Writes a fresh snapshot and truncates the WAL (so recovery cost is paid once)
6. Starts the checkpoint goroutine

Durability Guarantees¶

The sync_on_write option still exists for backward compatibility and triggers a full snapshot on every mutation in addition to the WAL append.

Future Work¶

The architecture is designed to support:

• LSM-tree and memory-mapped storage backends (storage/lsm/, storage/disk/mmap.go)
• B-tree based indexing (storage/btree/, index/)
• MVCC transactions (transaction/)
• A declarative query language with parser and optimizer (query/parser/, query/planner/)
• Graph algorithms (PageRank, community detection, etc.) (algorithm/)
• Transform/enrichment pipelines (transform/)
• Git-like graph versioning with branching and merging (versioning/)
• WebSocket subscriptions for real-time updates
• gRPC and GraphQL API layers