Workspace and Crates

Loading…

Workspace and Crates

Relevant source files

• .github/workflows/build.docs.yml
• Cargo.lock
• Cargo.toml
• demos/llkv-sql-pong-demo/assets/llkv-sql-pong-screenshot.png
• dev-docs/doc-preview.md
• llkv-executor/Cargo.toml
• llkv-join/Cargo.toml
• llkv-plan/Cargo.toml
• llkv-sql/Cargo.toml
• llkv-table/Cargo.toml
• llkv-test-utils/Cargo.toml

Purpose and Scope

This document describes the modular workspace structure of the LLKV codebase, detailing the 22 crates that comprise the system, their responsibilities, and interdependencies. Each crate serves a specific architectural layer, enabling separation of concerns and independent development. For information about the overall system architecture and how these crates fit into the layered design, see Architecture. For details about specific subsystems like SQL processing or storage, refer to the respective sections in the table of contents.

Sources: Cargo.toml:1-109

Workspace Configuration

LLKV uses a Cargo workspace to manage multiple interdependent crates under a unified build system. The workspace is configured with resolver = "2", enabling the newer dependency resolver that provides more predictable builds across different platforms and feature combinations.

All workspace members share common metadata including version 0.8.5-alpha, Apache-2.0 license, and Rust edition 2024. Workspace-level lints enforce quality standards, prohibiting print statements to stdout/stderr and disallowing specific methods like Box::leak.

Sources: Cargo.toml:2-28 Cargo.toml:29-35 Cargo.toml:98-104

Crate Dependency Architecture

Sources: Cargo.toml:37-96 llkv-table/Cargo.toml:20-40 llkv-executor/Cargo.toml:20-41 llkv-sql/Cargo.toml:20-37

Foundation Crates

llkv-types

The llkv-types crate provides the shared type system used throughout LLKV. It defines fundamental data types, field identifiers, and type conversions that enable type-safe operations across all layers. This crate has no internal LLKV dependencies, making it the true foundation upon which other crates build.

Purpose: Shared type definitions and type system infrastructure
Key Exports: Type enums, field identifiers, type conversion utilities
Internal Dependencies: None
External Dependencies: Minimal (bitcode, serde)

Sources: Cargo.toml74 llkv-table/Cargo.toml33 llkv-executor/Cargo.toml35

llkv-result

The llkv-result crate defines error types and result handling patterns used throughout the system. It provides a unified error type hierarchy that enables precise error reporting and recovery across all subsystem boundaries.

Purpose: Centralized error handling and result types
Key Exports: LlkvResult type alias, error enums
Internal Dependencies: None
External Dependencies: thiserror (for error derivation)

Sources: Cargo.toml64 llkv-table/Cargo.toml30 llkv-executor/Cargo.toml30

llkv-storage

The llkv-storage crate abstracts the underlying key-value storage layer through the Pager trait. It provides batch operations (batch_get, batch_put, alloc, free) that enable efficient bulk access to the persistent storage backend. When the simd-r-drive-support feature is enabled, it provides an implementation using the SIMD-optimized memory-mapped key-value store.

Purpose: Storage abstraction layer and pager interface
Key Exports: Pager trait, storage implementations
Internal Dependencies: llkv-types, llkv-result
External Dependencies: simd-r-drive (conditional)

Sources: Cargo.toml69 llkv-table/Cargo.toml32 llkv-sql/Cargo.toml29

Data Management Crates

llkv-column-map

The llkv-column-map crate implements the Arrow-native columnar storage engine. It manages data organized in chunks with metadata (min/max values, null counts) and provides operations like append (with Last-Write-Wins semantics), gather (row ID to RecordBatch assembly), filtering, and compaction. The ColumnStore and ColumnCatalog types bridge Apache Arrow’s in-memory format with the persistent key-value backend.

Purpose: Columnar data storage with Arrow integration
Key Exports: ColumnStore, ColumnCatalog, ColumnDescriptor
Internal Dependencies: llkv-types, llkv-storage
External Dependencies: arrow, croaring (for bitmap indexes)

Sources: Cargo.toml57 llkv-table/Cargo.toml26 llkv-executor/Cargo.toml25 llkv-sql/Cargo.toml22

llkv-table

The llkv-table crate provides the Table abstraction that represents a logical table with a schema. It integrates the columnar storage with catalog management, MVCC column injection, and scan operations. The SysCatalog within this crate stores metadata about all tables (Table 0 stores information about Tables 1+). The CatalogManager handles table lifecycle operations including create, alter, and drop.

Purpose: Table abstraction, catalog management, and metadata storage
Key Exports: Table, SysCatalog, CatalogManager, TableMeta, ColMeta
Internal Dependencies: llkv-types, llkv-result, llkv-storage, llkv-column-map, llkv-expr, llkv-scan, llkv-compute, llkv-plan
External Dependencies: arrow, arrow-array, arrow-schema, bitcode

Sources: Cargo.toml70 llkv-table/Cargo.toml:1-72 llkv-executor/Cargo.toml33 llkv-sql/Cargo.toml30

llkv-scan

The llkv-scan crate implements table scanning operations with predicate evaluation. It provides efficient row filtering using vectorized operations and bitmap indexes, enabling predicate pushdown to minimize data movement during query execution.

Purpose: Table scan operations and predicate evaluation
Key Exports: Scan iterators, filter evaluation functions
Internal Dependencies: llkv-types, llkv-column-map, llkv-expr
External Dependencies: arrow, croaring

Sources: Cargo.toml66 llkv-table/Cargo.toml31 llkv-executor/Cargo.toml31 llkv-plan/Cargo.toml26

Expression and Computation Crates

llkv-expr

The llkv-expr crate defines the expression Abstract Syntax Tree (AST) used throughout query processing. It includes Expr for general expressions and ScalarExpr for scalar computations. The crate provides expression translation (string column names to FieldId identifiers), compilation into bytecode (EvalProgram, DomainProgram), and optimization passes.

Purpose: Expression AST, translation, and compilation
Key Exports: Expr, ScalarExpr, EvalProgram, expression translators
Internal Dependencies: llkv-types, llkv-result
External Dependencies: arrow, time (for date/time handling)

Sources: Cargo.toml61 llkv-table/Cargo.toml28 llkv-executor/Cargo.toml27 llkv-sql/Cargo.toml25 llkv-plan/Cargo.toml24

llkv-compute

The llkv-compute crate implements vectorized compute kernels for scalar expression evaluation. It provides the NumericKernels system for optimized arithmetic, comparison, and logical operations on Arrow arrays. These kernels leverage SIMD instructions where possible for high-performance data processing.

Purpose: Vectorized compute kernels for expression evaluation
Key Exports: NumericKernels, scalar evaluation functions
Internal Dependencies: llkv-types, llkv-expr
External Dependencies: arrow

Sources: Cargo.toml58 llkv-table/Cargo.toml27 llkv-executor/Cargo.toml26 llkv-sql/Cargo.toml23 llkv-plan/Cargo.toml23

Query Processing Crates

llkv-plan

The llkv-plan crate converts SQL Abstract Syntax Trees (from sqlparser-rs) into executable query plans. It defines plan structures including SelectPlan, InsertPlan, UpdatePlan, DeletePlan, and DDL plans. The planner handles subquery correlation tracking, expression binding, and plan optimization.

Purpose: SQL-to-plan conversion and query planning
Key Exports: SelectPlan, InsertPlan, plan builder types
Internal Dependencies: llkv-types, llkv-result, llkv-expr, llkv-compute, llkv-scan, llkv-storage, llkv-column-map
External Dependencies: arrow, sqlparser, regex, time

Sources: Cargo.toml63 llkv-plan/Cargo.toml:1-42 llkv-executor/Cargo.toml29 llkv-sql/Cargo.toml26

llkv-executor

The llkv-executor crate executes query plans to produce results. It implements the TablePlanner and TableExecutor that optimize and execute table-level operations, including full table scans, filtered scans, aggregations, joins, and sorting. The executor uses parallel processing (via rayon) where beneficial and provides streaming result iteration.

Purpose: Query plan execution engine
Key Exports: TableExecutor, TablePlanner, execution functions
Internal Dependencies: llkv-types, llkv-result, llkv-expr, llkv-compute, llkv-plan, llkv-scan, llkv-table, llkv-storage, llkv-column-map, llkv-aggregate, llkv-join, llkv-threading
External Dependencies: arrow, rayon, croaring

Sources: Cargo.toml60 llkv-executor/Cargo.toml:1-47 llkv-sql/Cargo.toml24

llkv-aggregate

The llkv-aggregate crate implements aggregate function evaluation for GROUP BY queries. It provides accumulators for functions like SUM, AVG, COUNT, MIN, MAX, and handles DISTINCT aggregation using bitmap sets. The aggregation engine supports both hash-based and sort-based strategies.

Purpose: Aggregate function evaluation and accumulation
Key Exports: Accumulator types, aggregation operators
Internal Dependencies: llkv-types, llkv-expr, llkv-compute
External Dependencies: arrow, croaring

Sources: Cargo.toml56 llkv-executor/Cargo.toml24

llkv-join

The llkv-join crate implements table join operations using hash join algorithms. It supports INNER, LEFT, RIGHT, and FULL OUTER joins with optimizations for build/probe side selection and parallel hash table construction. The join implementation integrates with the table scan and filter systems for efficient multi-table queries.

Purpose: Table join operations and algorithms
Key Exports: Join operators, hash join implementation
Internal Dependencies: llkv-types, llkv-result, llkv-expr, llkv-table, llkv-storage, llkv-column-map, llkv-threading
External Dependencies: arrow, rayon, rustc-hash

Sources: Cargo.toml62 llkv-join/Cargo.toml:1-44 llkv-executor/Cargo.toml28

Runtime and Transaction Crates

llkv-runtime

The llkv-runtime crate provides the runtime engine that orchestrates query execution with transaction management. It integrates the executor, table management, and transaction systems, providing the high-level API for executing SQL statements within transactional contexts. The runtime manages catalog operations, schema validation, and result formatting.

Purpose: Runtime orchestration and transaction coordination
Key Exports: Runtime engine, execution context
Internal Dependencies: llkv-types, llkv-result, llkv-table, llkv-executor, llkv-transaction, llkv-storage
External Dependencies: arrow

Sources: Cargo.toml65 llkv-sql/Cargo.toml28

llkv-transaction

The llkv-transaction crate implements Multi-Version Concurrency Control (MVCC) for transactional semantics. It manages transaction identifiers, version visibility, and isolation. The system injects created_by and deleted_by columns into tables to track row versions, enabling snapshot isolation for concurrent queries.

Purpose: MVCC transaction management
Key Exports: Transaction manager, transaction ID generation
Internal Dependencies: llkv-types
External Dependencies: None (minimal)

Sources: Cargo.toml73 llkv-sql/Cargo.toml31

SQL Interface Crates

llkv-sql

The llkv-sql crate provides the primary SQL interface through the SqlEngine type. It handles SQL parsing (using sqlparser-rs), preprocessing (dialect normalization), plan building, and execution. The engine includes an INSERT buffering system that batches multiple INSERT statements for improved bulk insert performance. This is the main entry point for executing SQL statements against LLKV.

Purpose: SQL parsing, preprocessing, and execution interface
Key Exports: SqlEngine, SQL execution methods
Internal Dependencies: llkv-types, llkv-result, llkv-expr, llkv-plan, llkv-executor, llkv-runtime, llkv-table, llkv-storage, llkv-column-map, llkv-compute, llkv-transaction
External Dependencies: arrow, sqlparser, regex

Sources: Cargo.toml68 llkv-sql/Cargo.toml:1-44

Utility and Support Crates

llkv-csv

The llkv-csv crate provides utilities for importing and exporting CSV data to/from LLKV tables. It handles schema inference, data type conversion, and batch processing for efficient bulk data operations.

Purpose: CSV import/export functionality
Key Exports: CSV reader/writer utilities
Internal Dependencies: llkv-table, llkv-types
External Dependencies: arrow, csv

Sources: Cargo.toml59

llkv-test-utils

The llkv-test-utils crate provides testing utilities used across the workspace. When the auto-init feature is enabled, it automatically initializes tracing at test binary startup via the ctor crate, simplifying test debugging. This crate is marked as a development dependency in most other crates.

Purpose: Shared test utilities and test infrastructure
Key Exports: Test helpers, tracing initialization
Internal Dependencies: None
External Dependencies: tracing, tracing-subscriber, ctor (optional)

Sources: Cargo.toml71 llkv-test-utils/Cargo.toml:1-34 llkv-table/Cargo.toml45

llkv-threading

The llkv-threading crate provides threading utilities and abstractions used by parallelized operations in the executor and join crates. It wraps rayon patterns and provides consistent threading primitives across the codebase.

Purpose: Threading utilities and parallel processing abstractions
Key Exports: Thread pool management, parallel iterators
Internal Dependencies: llkv-types
External Dependencies: rayon, crossbeam-channel

Sources: Cargo.toml72 llkv-executor/Cargo.toml34 llkv-join/Cargo.toml27

Application and Demo Crates

llkv (Main Library)

The root llkv crate serves as the main library crate, aggregating and re-exporting key types and functions from the specialized crates. It provides a unified API surface for external consumers of the LLKV database system.

Purpose: Main library aggregation and public API
Key Exports: Re-exports from llkv-sql and other core crates
Internal Dependencies: llkv-sql and other workspace crates
External Dependencies: Inherited from dependencies

Sources: Cargo.toml5 Cargo.toml55

llkv-sql-pong-demo

An interactive terminal-based demonstration application that showcases LLKV’s SQL capabilities through a ping-pong game scenario. The demo creates tables, inserts data, and executes queries to demonstrate SQL functionality in an engaging way.

Purpose: Interactive SQL demonstration
Key Exports: Demo application binary
Internal Dependencies: llkv-sql
External Dependencies: crossterm (for terminal UI)

Sources: Cargo.toml4

llkv-slt-tester

The llkv-slt-tester crate implements a SQLLogicTest runner for LLKV. It executes standardized SQL test suites to verify correctness against established database behavior expectations, enabling regression testing and compatibility validation.

Purpose: SQLLogicTest execution and validation
Key Exports: Test runner binary
Internal Dependencies: llkv-sql
External Dependencies: sqllogictest, libtest-mimic

Sources: Cargo.toml67

llkv-tpch

The llkv-tpch crate implements the TPC-H benchmark suite for LLKV. It generates TPC-H schema and data, executes the 22 standard TPC-H queries, and measures performance metrics. This crate is used for performance evaluation and regression testing.

Purpose: TPC-H benchmark execution
Key Exports: Benchmark runner binary
Internal Dependencies: llkv-sql, llkv-csv
External Dependencies: tpchgen

Sources: Cargo.toml23

Workspace Dependencies Overview

The following table summarizes key external dependencies used across the workspace:

Sources: Cargo.toml:37-96

Crate Interdependency Matrix

Sources: Cargo.toml:37-96 llkv-table/Cargo.toml:20-40 llkv-executor/Cargo.toml:20-41 llkv-sql/Cargo.toml:20-37 llkv-join/Cargo.toml:20-32 llkv-plan/Cargo.toml:20-36

Build Configuration and Features

The workspace defines several build profiles and configuration options:

• resolver = “2” : Uses Cargo’s newer dependency resolver for more consistent builds
• edition = “2024” : Uses the latest Rust edition (2024)
• samply profile : Inherits from release with debug symbols enabled for profiling

Workspace lints enforce code quality by denying print statements and specific unsafe operations. Individual crates may enable conditional features like simd-r-drive-support for the storage backend.

Sources: Cargo.toml27 Cargo.toml32 Cargo.toml:98-109 llkv-sql/Cargo.toml40

Dismiss

Refresh this wiki

Enter email to refresh