AI Ingestion
Database Setup
The Parallel Ledger requires a dedicated PostgreSQL 17+ instance. For the most predictable performance, we recommend colocating the database on the same host to minimize network-induced latency.
The Data Model
The protocol manages its own tables, namespacing, and indexing.
We recommend provisioning an empty database with an owner-level user for the initial boot.
Data is categorized into three distinct functional areas. This separation maintains high performance even as volumes scale.
Ledger Updates (History)
A sequential, append-only record of every state change. This table provides the audit trail and grows linearly over time.
Most Recent Updates (Current)
A compact table representing the "now." It uses UPSERT operations to overwrite old values with new ones, keeping lookups fast and the dataset small.
String Dictionaries (Reference)
To minimize row-width and storage bloat, the Ledger normalizes repetitive strings (Slugs, Hashes, and System Codes) into a global dictionary. This results in high index efficiency and significantly reduced I/O for repetitive transactional data.
Data Lifecycle & GDPR
The Ledger is mathematically append-only. To satisfy "Right to Erasure" requirements without breaking cryptographic continuity, the protocol supports Payload Tombstoning.
This process redacts or masks sensitive data within a ledger entry while preserving the Chain Hash. This ensures that sibling records and subsequent blocks can still verify their own integrity against the immutable history, even if specific payload values have been cleared for compliance.
Performance Baseline
The following settings are derived from our internal benchmarks on 16GB RAM environments. They are intended as a starting point for your own environment-specific tuning.
# Memory
shared_buffers = 4GB
effective_cache_size = 12GB
work_mem = 10MB
maintenance_work_mem = 1GB
# Write-Ahead Log (WAL)
# Prevents I/O spikes during high-volume periods
wal_buffers = 16MB
min_wal_size = 1GB
max_wal_size = 4GB
checkpoint_completion_target = 0.9
# Disk I/O
# Set for SSD/NVMe performance
random_page_cost = 1.1
effective_io_concurrency = 200Implementation Note: Because the Current State table is updated frequently, ensure that your standard Postgres Autovacuum process is active. This prevents the database from holding onto disk space from overwritten records.
Database Setup
The Parallel Ledger requires a dedicated PostgreSQL 17+ instance. For the most predictable performance, we recommend colocating the database on the same host to minimize network-induced latency.
The Data Model
The protocol manages its own tables, namespacing, and indexing.
We recommend provisioning an empty database with an owner-level user for the initial boot.
Data is categorized into three distinct functional areas. This separation maintains high performance even as volumes scale.
Ledger Updates (History)
A sequential, append-only record of every state change. This table provides the audit trail and grows linearly over time.
Most Recent Updates (Current)
A compact table representing the "now." It uses UPSERT operations to overwrite old values with new ones, keeping lookups fast and the dataset small.
String Dictionaries (Reference)
To minimize row-width and storage bloat, the Ledger normalizes repetitive strings (Slugs, Hashes, and System Codes) into a global dictionary. This results in high index efficiency and significantly reduced I/O for repetitive transactional data.
Data Lifecycle & GDPR
The Ledger is mathematically append-only. To satisfy "Right to Erasure" requirements without breaking cryptographic continuity, the protocol supports Payload Tombstoning.
This process redacts or masks sensitive data within a ledger entry while preserving the Chain Hash. This ensures that sibling records and subsequent blocks can still verify their own integrity against the immutable history, even if specific payload values have been cleared for compliance.
Performance Baseline
The following settings are derived from our internal benchmarks on 16GB RAM environments. They are intended as a starting point for your own environment-specific tuning.
# Memory
shared_buffers = 4GB
effective_cache_size = 12GB
work_mem = 10MB
maintenance_work_mem = 1GB
# Write-Ahead Log (WAL)
# Prevents I/O spikes during high-volume periods
wal_buffers = 16MB
min_wal_size = 1GB
max_wal_size = 4GB
checkpoint_completion_target = 0.9
# Disk I/O
# Set for SSD/NVMe performance
random_page_cost = 1.1
effective_io_concurrency = 200Implementation Note: Because the Current State table is updated frequently, ensure that your standard Postgres Autovacuum process is active. This prevents the database from holding onto disk space from overwritten records.