This documentation is part of the "Projects with Books" initiative at zenOSmosis.

The source code for this project is available on GitHub.

SqlEngine API

Loading…

SqlEngine API

Relevant source files

This document describes the SqlEngine struct, which is the primary user-facing interface for executing SQL statements in LLKV. The SqlEngine parses SQL text using sqlparser-rs, converts it into execution plans, and delegates to the RuntimeEngine for execution. For details on how SQL syntax is normalized before parsing, see SQL Preprocessing and Dialect Handling. For information about the INSERT batching optimization, see INSERT Buffering System.

Sources: llkv-sql/src/sql_engine.rs:441-486 llkv-sql/src/lib.rs:1-52

SqlEngine Struct and Core Components

The SqlEngine struct wraps a RuntimeEngine and adds SQL parsing, statement caching, INSERT buffering, and dialect preprocessing capabilities.

Diagram: SqlEngine component structure and user interaction

graph TB
    SqlEngine["SqlEngine"]
RuntimeEngine["RuntimeEngine"]
RuntimeContext["RuntimeContext&lt;Pager&gt;"]
InsertBuffer["InsertBuffer"]
StatementCache["Statement Cache"]
InfoSchema["Information Schema"]
SqlEngine -->|contains| RuntimeEngine
 
   SqlEngine -->|maintains| InsertBuffer
 
   SqlEngine -->|maintains| StatementCache
 
   SqlEngine -->|tracks| InfoSchema
 
   RuntimeEngine -->|contains| RuntimeContext
 
   RuntimeContext -->|manages| Tables["Tables"]
RuntimeContext -->|manages| Catalog["System Catalog"]
User["User Code"] -->|execute sql| SqlEngine
 
   User -->|prepare sql| SqlEngine
 
   User -->|sql query| SqlEngine
 
   SqlEngine -->|execute_statement| RuntimeEngine
 
   RuntimeEngine -->|Result| SqlEngine
 
   SqlEngine -->|RecordBatch[]| User

Sources: llkv-sql/src/sql_engine.rs:572-620

SqlEngine Fields

Field	Type	Purpose
`engine`	`RuntimeEngine`	Core execution engine that manages tables, catalog, and transactions
`default_nulls_first`	`AtomicBool`	Controls default sort order for NULL values in ORDER BY clauses
`insert_buffer`	`RefCell<Option<InsertBuffer>>`	Accumulates literal INSERT statements for batch execution
`insert_buffering_enabled`	`AtomicBool`	Feature flag controlling cross-statement INSERT batching
`information_schema_ready`	`AtomicBool`	Tracks whether the information_schema has been initialized
`statement_cache`	`RwLock<FxHashMap<String, Arc<PreparedPlan>>>`	Cache for prepared statement plans

Sources: llkv-sql/src/sql_engine.rs:572-586

Construction and Configuration

Creating a SqlEngine

Diagram: SqlEngine construction paths

Sources: llkv-sql/src/sql_engine.rs:751-757 llkv-sql/src/sql_engine.rs:627-633

Constructor Methods

Method	Signature	Description
`new<Pg>(pager: Arc<Pg>)`	Creates engine with new `RuntimeContext`	Standard constructor for fresh database instances
`with_context(context, default_nulls_first)`	Creates engine from existing context	Used when sharing storage across engine instances

Configuration Options:

default_nulls_first : Controls whether NULL values sort first (true) or last (false) in ORDER BY clauses when NULLS FIRST/LAST is not explicitly specified
INSERT buffering : Disabled by default; enable via set_insert_buffering(true) for bulk loading workloads

Sources: llkv-sql/src/sql_engine.rs:751-757 llkv-sql/src/sql_engine.rs:627-633 llkv-sql/src/sql_engine.rs:1431-1449

Statement Execution

Primary Execution Methods

Diagram: Statement execution flow through SqlEngine

Sources: llkv-sql/src/sql_engine.rs:1109-1194 llkv-sql/src/sql_engine.rs:1196-1236 llkv-sql/src/sql_engine.rs:1715-1765

Execution Method Details

Method	Return Type	Use Case
`execute(&self, sql: &str)`	`Vec<RuntimeStatementResult>`	Execute one or more SQL statements (DDL, DML, SELECT)
`sql(&self, query: &str)`	`Vec<RecordBatch>`	Execute a single SELECT query and return Arrow results
`execute_single(&self, sql: &str)`	`RuntimeStatementResult`	Execute exactly one statement, error if multiple
`prepare(&self, sql: &str)`	`PreparedStatement`	Parse and cache a parameterized statement
`execute_prepared(&self, stmt, params)`	`Vec<RuntimeStatementResult>`	Execute a prepared statement with bound parameters

Sources: llkv-sql/src/sql_engine.rs:1109-1194 llkv-sql/src/sql_engine.rs:1196-1236 llkv-sql/src/sql_engine.rs:1238-1272 llkv-sql/src/sql_engine.rs:1715-1765 llkv-sql/src/sql_engine.rs:1767-1816

RuntimeStatementResult Variants

The RuntimeStatementResult enum represents the outcome of executing different statement types:

Variant	Fields	Produced By
`CreateTable`	`table_name: String`	`CREATE TABLE`
`CreateView`	`view_name: String`	`CREATE VIEW`
`CreateIndex`	`index_name: String, table_name: String`	`CREATE INDEX`
`DropTable`	`table_name: String`	`DROP TABLE`
`DropView`	`view_name: String`	`DROP VIEW`
`DropIndex`	`index_name: String`	`DROP INDEX`
`Insert`	`table_name: String, rows_inserted: usize`	`INSERT`
`Update`	`table_name: String, rows_updated: usize`	`UPDATE`
`Delete`	`table_name: String, rows_deleted: usize`	`DELETE`
`Select`	`execution: SelectExecution<P>`	`SELECT` (when using `execute()`)
`BeginTransaction`	`kind: TransactionKind`	`BEGIN TRANSACTION`
`CommitTransaction`		`COMMIT`
`RollbackTransaction`		`ROLLBACK`
`AlterTable`	`table_name: String`	`ALTER TABLE`
`Truncate`	`table_name: String`	`TRUNCATE`
`Reindex`	`index_name: Option<String>`	`REINDEX`

Sources: llkv-runtime/src/lib.rs (inferred from context), llkv-sql/src/lib.rs50

Prepared Statements and Parameter Binding

Parameter Placeholder Syntax

LLKV supports multiple parameter placeholder styles for prepared statements:

Style	Example	Description
Positional (?)	`SELECT * FROM t WHERE id = ?`	Auto-numbered starting from 1
Numbered (?)	`SELECT * FROM t WHERE id = ?2`	Explicit position
Numbered ($)	`SELECT * FROM t WHERE id = $1`	PostgreSQL style
Named (:)	`SELECT * FROM t WHERE name = :username`	Named parameters

Sources: llkv-sql/src/sql_engine.rs:94-133 llkv-sql/src/sql_engine.rs:258-268

Parameter Binding Flow

Diagram: Prepared statement lifecycle with parameter binding

Sources: llkv-sql/src/sql_engine.rs:1715-1765 llkv-sql/src/sql_engine.rs:1767-1816 llkv-sql/src/sql_engine.rs:223-256

SqlParamValue Types

The SqlParamValue enum represents typed parameter values:

Conversion methods:

as_literal(&self) -> Literal - Converts to internal Literal type for expression evaluation
as_plan_value(&self) -> PlanValue - Converts to PlanValue for INSERT operations

Sources: llkv-sql/src/sql_engine.rs:285-316

graph TB
    subgraph "Statement Processing"
        Parse["parse_statement()"]
Canonical["canonicalize_insert()"]
PreparedInsert["PreparedInsert"]
end
    
    subgraph "Buffer Management"
        BufferDecision{"Can Buffer?"}
BufferAdd["buffer.push_statement()"]
BufferFlush["flush_buffer()"]
BufferCheck{"buffer.should_flush()?"}
end
    
    subgraph "Execution"
        BuildPlan["InsertPlan::from_buffer"]
Execute["execute_insert_plan()"]
end
    
 
   Parse -->|INSERT statement| Canonical
 
   Canonical -->|VALUES only| PreparedInsert
 
   PreparedInsert --> BufferDecision
    
 
   BufferDecision -->|Yes: same table, same columns, same on_conflict| BufferAdd
 
   BufferDecision -->|No: different params| BufferFlush
    
 
   BufferAdd --> BufferCheck
 
   BufferCheck -->|Yes: >= 8192 rows| BufferFlush
 
   BufferCheck -->|No: continue| Continue["Continue processing"]
BufferFlush --> BuildPlan
 
   BuildPlan --> Execute
 
   Execute -->|Multiple Insert results| Results["Vec<RuntimeStatementResult>"]

INSERT Buffering System

The INSERT buffering system batches multiple literal INSERT ... VALUES statements together to reduce planning overhead during bulk loading operations.

Buffering Architecture

Diagram: INSERT buffering decision tree and execution flow

Sources: llkv-sql/src/sql_engine.rs:486-546 llkv-sql/src/sql_engine.rs:1869-2008

InsertBuffer Configuration

Constant/Field	Value/Type	Purpose
`MAX_BUFFERED_INSERT_ROWS`	`8192`	Maximum rows to accumulate before forcing a flush
`insert_buffering_enabled`	`AtomicBool`	Global enable/disable flag (default: false)
`table_name`	`String`	Target table for buffered inserts
`columns`	`Vec<String>`	Column list (must match across statements)
`on_conflict`	`InsertConflictAction`	Conflict resolution strategy (must match)
`statement_row_counts`	`Vec<usize>`	Tracks row counts per original statement
`rows`	`Vec<Vec<PlanValue>>`	Accumulated literal rows

Buffer acceptance rules:

Table name must match exactly
Column list must match exactly
on_conflict action must match
Only literal VALUES are buffered (subqueries flush immediately)

Flush triggers:

Row count reaches MAX_BUFFERED_INSERT_ROWS
Next INSERT targets different table/columns/conflict action
Non-INSERT statement encountered
execute() completes (flush remaining)
SqlEngine is dropped

Sources: llkv-sql/src/sql_engine.rs:486-546 llkv-sql/src/sql_engine.rs:1431-1449

graph LR
    Input["Raw SQL String"]
TPCH["preprocess_tpch_connect_syntax"]
CreateType["preprocess_create_type_syntax"]
Exclude["preprocess_exclude_syntax"]
TrailingComma["preprocess_trailing_commas_in_values"]
EmptyIn["preprocess_empty_in_lists"]
IndexHints["preprocess_index_hints"]
Reindex["preprocess_reindex_syntax"]
TriggerShorthand["preprocess_sqlite_trigger_shorthand"]
BareTable["preprocess_bare_table_in_clauses"]
Parser["sqlparser::Parser"]
AST["Vec<Statement>"]
Input --> TPCH
 
   TPCH --> CreateType
 
   CreateType --> Exclude
 
   Exclude --> TrailingComma
 
   TrailingComma --> EmptyIn
 
   EmptyIn --> IndexHints
 
   IndexHints --> Reindex
 
   Reindex --> TriggerShorthand
 
   TriggerShorthand --> BareTable
 
   BareTable --> Parser
 
   Parser --> AST

SQL Preprocessing

Before parsing with sqlparser, the SQL text undergoes multiple normalization passes to handle dialect-specific syntax variations. This allows LLKV to accept SQL from SQLite, DuckDB, PostgreSQL, and other sources.

Preprocessing Pipeline

Diagram: SQL preprocessing transformation pipeline

Sources: llkv-sql/src/sql_engine.rs:1024-1103

Preprocessing Transformations

Method	Transformation	Example
`preprocess_tpch_connect_syntax`	Strip `CONNECT TO <db>;`	`CONNECT TO tpch;` → (removed)
`preprocess_create_type_syntax`	`CREATE TYPE` → `CREATE DOMAIN`	`CREATE TYPE myint AS INTEGER` → `CREATE DOMAIN myint AS INTEGER`
`preprocess_exclude_syntax`	Quote qualified EXCLUDE identifiers	`EXCLUDE (schema.table.col)` → `EXCLUDE ("schema.table.col")`
`preprocess_trailing_commas_in_values`	Remove trailing commas	`VALUES (1, 2,)` → `VALUES (1, 2)`
`preprocess_empty_in_lists`	Convert empty IN to boolean expr	`col IN ()` → `(col = NULL AND 0 = 1)`
`preprocess_index_hints`	Strip SQLite index hints	`FROM t INDEXED BY idx` → `FROM t`
`preprocess_reindex_syntax`	Convert to VACUUM REINDEX	`REINDEX idx` → `VACUUM REINDEX idx`
`preprocess_sqlite_trigger_shorthand`	Add explicit timing/FOR EACH ROW	`CREATE TRIGGER t INSERT ON t` → `CREATE TRIGGER t AFTER INSERT ON t FOR EACH ROW`
`preprocess_bare_table_in_clauses`	Wrap table in subquery	`col IN tablename` → `col IN (SELECT * FROM tablename)`

Sources: llkv-sql/src/sql_engine.rs:759-1006

Session and Transaction Management

Accessing the RuntimeSession

The SqlEngine::session() method provides access to the underlying RuntimeSession, which manages transaction state and constraint enforcement:

Key session methods:

begin_transaction(kind) - Start a new transaction
commit_transaction() - Commit the current transaction
rollback_transaction() - Abort the current transaction
set_constraint_enforcement_mode(mode) - Control when constraints are checked
execute_insert_plan(plan) - Direct plan execution (bypasses SQL parsing)
execute_select_plan(plan) - Direct SELECT execution

Sources: llkv-sql/src/sql_engine.rs:1412-1424

Constraint Enforcement Modes

The session supports two constraint enforcement modes:

Mode	Behavior
`ConstraintEnforcementMode::Immediate`	Constraints checked after each statement
`ConstraintEnforcementMode::Deferred`	Constraints checked only at transaction commit

Deferred mode is useful for bulk loading when referential integrity violations may occur temporarily during the load process.

Sources: llkv-table/src/catalog/manager.rs (inferred from TpchToolkit usage), llkv-tpch/src/lib.rs:274-278

stateDiagram-v2
    [*] --> Uninitialized: SqlEngine::new()
    Uninitialized --> Checking : Query references information_schema
    Checking --> Initializing : ensure_information_schema_ready()
    Initializing --> Ready : refresh_information_schema()
    Ready --> Ready : Subsequent queries
    Ready --> Invalidated : DDL statement (CREATE/DROP TABLE, etc.)
    Invalidated --> Checking : Next information_schema query
    Checking --> Ready : Already initialized

Information Schema

The SqlEngine lazily initializes the information_schema tables on first access to avoid startup overhead. Querying information_schema.tables, information_schema.columns, or other information schema views triggers initialization.

Information Schema Lifecycle

Diagram: Information schema initialization and invalidation lifecycle

Key methods:

ensure_information_schema_ready() - Initialize if not already ready
invalidate_information_schema() - Mark as stale after schema changes
refresh_information_schema() - Delegate to RuntimeEngine for rebuild

Invalidation triggers:

CREATE TABLE, DROP TABLE
ALTER TABLE
CREATE VIEW, DROP VIEW

Sources: llkv-sql/src/sql_engine.rs:648-660 llkv-sql/src/sql_engine.rs:706-745

graph LR
    Execute["execute_plan_statement"]
RuntimeEngine["RuntimeEngine::execute_statement"]
Error["Error::NotFound or\nInvalidArgumentError"]
MapError["map_table_error"]
UserError["Error::CatalogError:\n'Table does not exist'"]
Execute -->|delegates| RuntimeEngine
 
   RuntimeEngine -->|returns| Error
 
   Execute -->|table name available| MapError
 
   MapError -->|transforms| UserError
 
   UserError -->|propagates| User["User Code"]

Error Handling and Diagnostics

Error Mapping

The SqlEngine transforms generic storage errors into user-friendly SQL errors, particularly for table lookup failures:

Diagram: Error mapping for table lookup failures

Error transformation rules:

Error::NotFound → Error::CatalogError("Table '...' does not exist")
Generic “unknown table” messages → Catalog error with table name
View-related operations (CREATE VIEW, DROP VIEW) skip mapping

Sources: llkv-sql/src/sql_engine.rs:677-745

Statement Expectations (Test Harness)

For test infrastructure, the SqlEngine supports registering expected outcomes for statements:

Expectation types:

StatementExpectation::Ok - Statement should succeed
StatementExpectation::Error - Statement should fail
StatementExpectation::Count(n) - Statement should affect n rows

Sources: llkv-sql/src/sql_engine.rs:65-391

Constant	Value	Purpose
`PARSER_RECURSION_LIMIT`	`200`	Maximum recursion depth for sqlparser (prevents stack overflow)
`MAX_BUFFERED_INSERT_ROWS`	`8192`	Flush threshold for INSERT buffering
`PARAM_SENTINEL_PREFIX`	`"__llkv_param__"`	Internal marker for parameter placeholders
`PARAM_SENTINEL_SUFFIX`	`"__"`	Internal marker suffix
`DROPPED_TABLE_TRANSACTION_ERR`	`"another transaction has dropped this table"`	Error message for concurrent table drops

Sources: llkv-sql/src/sql_engine.rs:78-588

Dismiss

Refresh this wiki

Enter email to refresh

rust-llkv Documentation

SqlEngine API

SqlEngine API

SqlEngine Struct and Core Components

SqlEngine Fields

Construction and Configuration

Creating a SqlEngine

Constructor Methods

Statement Execution

Primary Execution Methods

Execution Method Details

RuntimeStatementResult Variants

Prepared Statements and Parameter Binding

Parameter Placeholder Syntax

Parameter Binding Flow

SqlParamValue Types

INSERT Buffering System

Buffering Architecture

InsertBuffer Configuration

SQL Preprocessing

Preprocessing Pipeline

Preprocessing Transformations

Session and Transaction Management

Accessing the RuntimeSession

Constraint Enforcement Modes

Information Schema

Information Schema Lifecycle

Error Handling and Diagnostics

Error Mapping

Statement Expectations (Test Harness)

Advanced Usage Examples

Bulk Loading with INSERT Buffering

Prepared Statements with Named Parameters

Direct Plan Execution (Advanced)

Key Constants

Keyboard shortcuts

rust-llkv Documentation