This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

CatalogManager API

Relevant source files

Purpose and Scope

The CatalogManager is responsible for managing table lifecycle operations (CREATE, ALTER, DROP) and coordinating metadata storage through the system catalog. It serves as the primary interface between DDL statements and the underlying storage layer, ensuring that schema changes are transactional, MVCC-compliant, and crash-consistent.

This document covers the CatalogManager's role in orchestrating table creation, schema validation, and metadata persistence. For details about the underlying storage mechanism, see System Catalog and SysCatalog. For information about custom type definitions, see Custom Types and Type Registry. For the table-level data operations API, see Table Abstraction.

Sources: llkv-table/README.md:1-57 llkv-runtime/README.md:1-63

Architectural Overview

CatalogManager in the Runtime Layer

CatalogManager Coordination Flow

The CatalogManager orchestrates DDL operations by validating schemas, injecting MVCC metadata, and coordinating with the dual-context transaction model.

Sources: llkv-runtime/README.md:1-63 README.md:43-73

Key Responsibilities

The CatalogManager handles the following responsibilities within the LLKV runtime:

Responsibility	Description
Schema Validation	Validates Arrow schema definitions, checks for duplicate names, ensures data type compatibility
MVCC Integration	Injects `row_id`, `created_by`, `deleted_by` columns into all table definitions
Metadata Persistence	Stores `TableMeta` and `ColMeta` entries in SysCatalog (table 0) using Arrow append operations
Transaction Coordination	Manages dual-context execution (persistent + staging) for transactional DDL
Conflict Detection	Checks for concurrent schema changes during commit and aborts on conflicts
Visibility Control	Ensures snapshot isolation for catalog queries based on transaction context

Sources: llkv-table/README.md:13-30 llkv-runtime/README.md:12-17

Table Lifecycle Management

CREATE TABLE Flow

Table Creation Sequence

The CREATE TABLE flow validates schemas, injects MVCC columns, and either commits immediately (auto-commit) or stages definitions for later replay (explicit transactions).

Sources: llkv-runtime/README.md:20-32 llkv-table/README.md:25-30

Dual-Context Catalog Management

The CatalogManager maintains two separate contexts during explicit transactions:

Persistent Context

Backing Storage : BoxedPager (typically SimdRDrivePager for persistent storage)
Contains : All committed table definitions from previous transactions
Visibility : Tables visible to all transactions with appropriate snapshot isolation
Lifetime : Survives process restarts and crashes

Staging Context

Backing Storage : MemPager (in-memory hash map)
Contains : Table definitions created within the current transaction
Visibility : Only visible within the creating transaction
Lifetime : Discarded on rollback, replayed to persistent context on commit

Dual-Context Transaction Model

CREATE TABLE operations during explicit transactions stage in MemPager and merge into persistent storage on commit.

Sources: llkv-runtime/README.md:26-32 README.md:64-71

ALTER and DROP Operations

ALTER TABLE

The CatalogManager handles schema alterations by:

Validating the requested change against existing data
Updating ColMeta entries in SysCatalog
Tagging the change with the current transaction ID
Maintaining snapshot isolation so concurrent readers see consistent schemas

DROP TABLE

Table deletion follows MVCC semantics:

Marks the table's TableMeta entry with deleted_by = current_txn_id
Table remains visible to transactions with earlier snapshots
New transactions cannot see the dropped table
Physical cleanup is implementation-dependent and may occur during compaction

Sources: llkv-table/README.md:25-30

Metadata Structure

TableMeta and ColMeta

The CatalogManager persists two types of metadata entries in SysCatalog:

Metadata Type	Fields	Purpose
TableMeta	`table_id`, `table_name`, `namespace`, `created_by`, `deleted_by`	Describes table existence and lifecycle
ColMeta	`table_id`, `col_id`, `col_name`, `data_type`, `is_mvcc`, `created_by`, `deleted_by`	Describes individual column definitions

graph LR
    subgraph "SysCatalog (Table 0)"
        TABLEMETA["TableMeta\nrow_id / table_id / name / created_by / deleted_by"]
COLMETA["ColMeta\nrow_id / table_id / col_id / name / type / created_by / deleted_by"]
end
    
    subgraph "User Tables"
        TABLE1["user_table_1\nSchema from ColMeta"]
TABLE2["user_table_2\nSchema from ColMeta"]
end
    
 
   TABLEMETA --> TABLE1
 
   TABLEMETA --> TABLE2
 
   COLMETA --> TABLE1
 
   COLMETA --> TABLE2

Both metadata types use MVCC columns (created_by, deleted_by) to enable snapshot isolation for catalog queries.

Metadata to Table Mapping

The CatalogManager queries SysCatalog to resolve table names and reconstruct Arrow schemas for query execution.

Sources: llkv-table/README.md:25-30 llkv-column-map/README.md:18-23

API Surface

Table Creation

The CatalogManager exposes table creation through the runtime layer:

Input : Table name (string), Arrow Schema definition, optional namespace
Validation Steps :
- Check for duplicate table names within namespace
- Validate column names are unique
- Ensure data types are supported
- Verify constraints (PRIMARY KEY uniqueness)
MVCC Injection : Automatically adds row_id (UInt64), created_by (UInt64), deleted_by (UInt64) columns
Output : TableId identifier for subsequent operations

Table Lookup

The CatalogManager provides catalog query operations:

By Name : Resolve table name to TableId within a namespace
By ID : Retrieve TableMeta and ColMeta for a given TableId
Visibility Filtering : Apply transaction snapshot to filter dropped tables
Schema Reconstruction : Build Arrow Schema from ColMeta entries

Schema Validation

Validation operations performed by CatalogManager:

Column Uniqueness : Ensure no duplicate column names within a table
Type Compatibility : Verify data types are supported by Arrow and the storage layer
Constraint Validation : Check PRIMARY KEY, FOREIGN KEY, NOT NULL constraints
Naming Conventions : Enforce reserved column name restrictions (e.g., row_id)

Sources: llkv-table/README.md:13-18 llkv-runtime/README.md:12-17

Transaction Coordination

Snapshot Isolation for DDL

DDL Snapshot Isolation

Transactions see a consistent catalog snapshot; tables created by T1 are not visible to T2 until T1 commits.

Sources: llkv-runtime/README.md:20-24 README.md:64-71

Conflict Detection

On commit, the CatalogManager checks for conflicting operations:

Conflict Type	Detection Method	Resolution
Duplicate CREATE	Query SysCatalog for tables created after snapshot timestamp	Abort transaction
Concurrent DROP	Check if table's `deleted_by` was set by another transaction	Abort transaction
Schema Mismatch	Compare staged schema against current persistent schema	Abort transaction

Conflict detection ensures serializable DDL semantics despite optimistic concurrency control.

Sources: llkv-runtime/README.md:20-32

Integration with Runtime Components

RuntimeContext Coordination

The CatalogManager coordinates with RuntimeContext for:

Transaction Snapshots : Obtains current snapshot from TransactionSnapshot for visibility filtering
Transaction ID Allocation : Requests new transaction IDs from TxnIdManager for MVCC tagging
Dual-Context Management : Coordinates between persistent and staging pagers
Commit Protocol : Invokes staged operation replay during commit

Table Layer Integration

Interactions with llkv-table:

Table Instantiation : Creates Table instances from TableMeta and ColMeta
Schema Validation : Validates incoming RecordBatch schemas during append operations
Field Mapping : Resolves logical field names to FieldId identifiers
MVCC Column Access : Provides metadata for row_id, created_by, deleted_by columns

Executor Integration

The CatalogManager supports llkv-executor by:

Table Resolution : Resolves table references during query planning
Schema Information : Supplies Arrow schemas for projection and filtering
Column Validation : Validates column references in expressions and predicates
Subquery Support : Provides catalog context for correlated subquery evaluation

Sources: llkv-runtime/README.md:42-46 llkv-table/README.md:36-40

Error Handling and Recovery

Validation Errors

The CatalogManager returns structured errors for:

Duplicate Table Names : Table already exists within the namespace
Invalid Column Definitions : Unsupported data type or constraint violation
Reserved Column Names : Attempt to use system-reserved names like row_id
Constraint Violations : PRIMARY KEY or FOREIGN KEY constraint failures

Transaction Errors

Transaction-related failures:

Commit Conflicts : Concurrent DDL operations detected during commit
Snapshot Violations : Attempt to query table created after snapshot timestamp
Pager Failures : Persistent storage write failures during commit
Staging Inconsistencies : Corrupted staging context state

Crash Recovery

After crash recovery:

Persistent Catalog Loaded : SysCatalog read from pager root key
Uncommitted Transactions Discarded : Staging contexts do not survive restarts
MVCC Visibility Applied : Only committed tables with valid created_by are visible
No Replay Required : Catalog state is consistent without separate recovery log

Sources: llkv-table/README.md:25-30 README.md:64-71

Performance Characteristics

Catalog Query Optimization

The CatalogManager optimizes metadata access through:

Schema Caching : Frequently accessed schemas cached in RuntimeContext
Batch Lookups : Multiple table lookups batched into single SysCatalog scan
Snapshot Reuse : Transaction snapshots reused across multiple catalog queries
Lazy Loading : Column metadata loaded only when required

Concurrent DDL Handling

Concurrency characteristics:

Optimistic Concurrency : No global catalog locks; conflicts detected at commit
Snapshot Isolation : Long-running transactions see stable schema
Minimal Blocking : DDL operations do not block concurrent queries
Serializable DDL : Conflict detection ensures serializable execution

Scalability Considerations

System behavior at scale:

Linear Growth : SysCatalog size grows linearly with table and column count
Efficient Lookups : Table name resolution uses indexed scans
Distributed Metadata : Column metadata distributed across ColMeta entries
No Centralized Bottleneck : No single global lock for catalog operations

Sources: llkv-column-map/README.md:30-35 README.md:35-42

Example Usage Patterns

Auto-Commit CREATE TABLE

Client: CREATE TABLE users (id INT, name TEXT);

Flow:
1. SqlEngine parses to CreateTablePlan
2. RuntimeContext invokes CatalogManager.create_table()
3. CatalogManager validates schema, injects MVCC columns
4. TableMeta and ColMeta appended to SysCatalog (table 0)
5. Persistent pager commits atomically
6. Table immediately visible to all transactions

Transactional CREATE TABLE

Client: BEGIN;
Client: CREATE TABLE temp_results (id INT, value DOUBLE);
Client: INSERT INTO temp_results SELECT ...;
Client: COMMIT;

Flow:
1. BEGIN captures snapshot = 500
2. CREATE TABLE stages TableMeta in MemPager
3. INSERT operations target staging context
4. COMMIT replays staged table to persistent context
5. Conflict detection checks for concurrent creates
6. Table committed with created_by = 501

Concurrent DDL with Conflict

Transaction T1: BEGIN; CREATE TABLE foo (...); [waits]
Transaction T2: BEGIN; CREATE TABLE foo (...); COMMIT;
Transaction T1: COMMIT; [aborts with conflict error]

Reason: T1 detects that foo was created by T2 after T1's snapshot

Sources: demos/llkv-sql-pong-demo/src/main.rs:44-81 llkv-runtime/README.md:20-32

Keyboard shortcuts

rust-llkv Documentation