SqlEngine API

Relevant source files

• llkv-aggregate/src/lib.rs
• llkv-executor/README.md
• llkv-plan/README.md
• llkv-sql/README.md
• llkv-sql/src/lib.rs
• llkv-sql/src/sql_engine.rs
• llkv-sql/src/sql_value.rs

Purpose and Scope

The SqlEngine provides the primary user-facing API for executing SQL statements against LLKV databases. It accepts SQL text, parses it, translates it to typed execution plans, and delegates to the runtime layer for evaluation. This page documents the SqlEngine struct's construction, methods, prepared statement handling, and configuration options.

For information about SQL preprocessing and dialect handling, see SQL Preprocessing and Dialect Handling. For details on INSERT buffering behavior, see INSERT Buffering System. For query planning internals, see Plan Structures.

Sources: llkv-sql/src/lib.rs:1-51 llkv-sql/README.md:1-68

SqlEngine Architecture Overview

The SqlEngine sits at the top of the LLKV SQL processing stack, coordinating parsing, planning, and execution:

Diagram: SqlEngine Position in SQL Processing Stack

graph TB
    User["User Code"]
SqlEngine["SqlEngine\n(llkv-sql)"]
Parser["sqlparser\nAST Generation"]
Preprocessor["SQL Preprocessor\nDialect Normalization"]
Planner["Plan Translation\nAST → PlanStatement"]
Runtime["RuntimeEngine\n(llkv-runtime)"]
Executor["llkv-executor\nQuery Execution"]
Table["llkv-table\nTable Layer"]
User -->|execute sql| SqlEngine
 
   User -->|sql select| SqlEngine
 
   User -->|prepare sql| SqlEngine
    
 
   SqlEngine --> Preprocessor
 
   Preprocessor --> Parser
 
   Parser --> Planner
 
   Planner --> Runtime
 
   Runtime --> Executor
 
   Executor --> Table
    
 
   SqlEngine -.->|owns| Runtime
 
   SqlEngine -.->|manages| InsertBuffer["InsertBuffer\nBatching State"]
SqlEngine -.->|caches| StmtCache["statement_cache\nPreparedPlan Cache"]

The SqlEngine wraps a RuntimeEngine, manages statement caching and INSERT buffering, and provides convenience methods for single-statement queries (sql()) and multi-statement execution (execute()).

Sources: llkv-sql/src/sql_engine.rs:496-509 llkv-sql/src/lib.rs:1-51

Constructing a SqlEngine

Basic Construction

Diagram: SqlEngine Construction Flow

The most common constructor is SqlEngine::new(), which accepts a pager and creates a new runtime context:

Sources: llkv-sql/src/sql_engine.rs:615-621

Table: SqlEngine Constructor Methods

Sources: llkv-sql/src/sql_engine.rs:543-556 llkv-sql/src/sql_engine.rs:615-621 llkv-sql/src/sql_engine.rs:879-885

Core Query Execution Methods

execute() - Multi-Statement Execution

The execute() method processes one or more SQL statements from a string and returns a vector of results:

Diagram: execute() Method Execution Flow

The execution pipeline:

1. Preprocessing : SQL text undergoes dialect normalization via preprocess_sql_input() llkv-sql/src/sql_engine.rs:1556-1564
2. Parsing : sqlparser converts normalized text to AST with recursion limit of 200 llkv-sql/src/sql_engine.rs324
3. Statement Loop : Each statement is translated to a PlanStatement and either buffered (for INSERTs) or executed immediately
4. Result Collection : Results are accumulated and returned as Vec<SqlStatementResult>

Sources: llkv-sql/src/sql_engine.rs:933-1044

sql() - Single SELECT Execution

The sql() method enforces single-statement SELECT semantics and returns Arrow RecordBatch results:

Key differences from execute():

• Accepts only a single statement
• Statement must be a SELECT query
• Returns Vec<RecordBatch> directly rather than RuntimeStatementResult
• Automatically collects streaming results

Sources: llkv-sql/src/sql_engine.rs:1046-1085

Prepared Statements

Prepared Statement Flow

Diagram: Prepared Statement Creation and Caching

prepare() Method

The prepare() method parses SQL with placeholders and caches the resulting plan:

Placeholder syntax supported:

• ? - Positional parameter (auto-increments)
• ?N - Numbered parameter (1-indexed)
• $N - PostgreSQL-style numbered parameter
• :name - Named parameter

Sources: llkv-sql/src/sql_engine.rs:1296-1376 llkv-sql/src/sql_engine.rs:86-132

Parameter Binding Mechanism

Parameter registration occurs via thread-local ParameterScope:

Diagram: Parameter Registration and Sentinel Generation

The parameter translation process:

1. During parsing, placeholders are intercepted and converted to sentinel strings: __llkv_param__N__
2. ParameterState tracks placeholder-to-index mappings in thread-local storage
3. At execution time, sentinel strings are replaced with actual parameter values

Sources: llkv-sql/src/sql_engine.rs:71-206

SqlParamValue Type

Parameter values are represented by the SqlParamValue enum:

Table: SqlParamValue Variants and Conversions

Sources: llkv-sql/src/sql_engine.rs:208-276

execute_prepared() Method

Execute a prepared statement with bound parameters:

Parameter substitution occurs in two phases:

1. Literal Substitution : Sentinels in Expr<String> are replaced via substitute_parameter_literals() llkv-sql/src/sql_engine.rs:1453-1497
2. Plan Value Substitution : Sentinels in Vec<PlanValue> are replaced via substitute_parameter_plan_values() llkv-sql/src/sql_engine.rs:1499-1517

Sources: llkv-sql/src/sql_engine.rs:1378-1451

Transaction Control

The SqlEngine supports explicit transaction boundaries via SQL statements:

Diagram: Transaction State Machine

Transaction management methods:

Transaction boundaries automatically flush the INSERT buffer to ensure consistent visibility semantics.

Sources: llkv-sql/src/sql_engine.rs:970-990 llkv-sql/src/sql_engine.rs:912-914

INSERT Buffering System

Buffer Architecture

Diagram: INSERT Buffer Accumulation and Flush

InsertBuffer Structure

The InsertBuffer struct accumulates literal INSERT payloads:

Sources: llkv-sql/src/sql_engine.rs:421-471

Buffer Compatibility

An INSERT can join the buffer if it matches:

1. Table Name : Target table must match buffer.table_name
2. Column List : Columns must match buffer.columns exactly
3. Conflict Action : on_conflict strategy must match

Sources: llkv-sql/src/sql_engine.rs:452-459

Flush Conditions

The buffer flushes when:

Table: INSERT Buffer Flush Triggers

Enabling/Disabling Buffering

INSERT buffering is controlled by the set_insert_buffering() method:

• Disabled by default to maintain statement-level transaction semantics
• Enable for bulk loading to reduce planning overhead
• Disabling flushes buffer to ensure pending rows are persisted

Sources: llkv-sql/src/sql_engine.rs:898-905

classDiagram
    class RuntimeStatementResult {<<enum>>\nSelect\nInsert\nUpdate\nDelete\nCreateTable\nDropTable\nCreateIndex\nDropIndex\nAlterTable\nCreateView\nDropView\nVacuum\nTransaction}
    
    class SelectVariant {+SelectExecution execution}
    
    class InsertVariant {+table_name: String\n+rows_inserted: usize}
    
    class UpdateVariant {+table_name: String\n+rows_updated: usize}
    
    RuntimeStatementResult --> SelectVariant
    RuntimeStatementResult --> InsertVariant
    RuntimeStatementResult --> UpdateVariant

Result Types

RuntimeStatementResult

The execute() and execute_prepared() methods return Vec<RuntimeStatementResult>:

Diagram: RuntimeStatementResult Variants

Key result variants:

Table: Common RuntimeStatementResult Variants

Sources: llkv-sql/src/lib.rs49

SelectExecution

SELECT queries return a SelectExecution handle for streaming results:

The sql() method automatically collects batches via execution.collect():

Sources: llkv-sql/src/sql_engine.rs:1065-1080

Configuration Methods

session() - Access Runtime Session

Provides access to the underlying RuntimeSession for transaction introspection or advanced error handling.

Sources: llkv-sql/src/sql_engine.rs:917-919

context_arc() - Access Runtime Context

Internal method to retrieve the shared RuntimeContext for engine composition.

Sources: llkv-sql/src/sql_engine.rs:875-877

Testing Utilities

StatementExpectation

Test harnesses can register expectations to control buffer flushing:

When a statement expectation is registered, the INSERT buffer flushes before executing that statement to ensure test assertions observe correct row counts.

Sources: llkv-sql/src/sql_engine.rs:64-315

Example Usage

Sources: llkv-sql/src/sql_engine.rs:299-309

Thread Safety and Cloning

The SqlEngine implements Clone with special semantics:

Warning : Cloning a SqlEngine creates a new RuntimeSession, not a shared reference. Each clone has independent transaction state and INSERT buffers.

Sources: llkv-sql/src/sql_engine.rs:522-540

Error Handling

Table Not Found Errors

The SqlEngine remaps generic errors to user-friendly catalog errors:

This converts low-level NotFound errors into: Catalog Error: Table 'tablename' does not exist

Sources: llkv-sql/src/sql_engine.rs:558-585