--- id: 134c150de56c188c925476664026a1ff author: Lemon Mint title: 'gosuda.org/sqlite3: CGO_ENABLED=0 database/sql Driver Over Shared-Memory IPC' description: Drop-in database/sql SQLite 3.54.1 driver for Go with CGO_ENABLED=0, HQQ shared-memory worker isolation, 64 KiB batch row prefetching, and SLSA-attested binaries. language: en date: 2026-10-18T14:02:16.803040007Z path: /sqlite3 go_package: gosuda.org/sqlite3 go_repourl: https://github.com/gosuda/hqq-sqlite3.git category: database --- `gosuda.org/sqlite3` is a `CGO_ENABLED=0` `database/sql` driver for Go that runs the upstream C SQLite `3.54.1` amalgamation inside an isolated worker process over [HQQ](https://gosuda.org/hqq) shared-memory rings. It registers under the standard `"sqlite3"` driver name, supports `github.com/mattn/go-sqlite3` DSN query parameters, embeds SLSA-attested prebuilt server binaries for six OS and architecture targets, and uses request pipelining with 64 KiB batch prefetching to outperform in-process `cgo` on multi-row queries. ```bash go get gosuda.org/sqlite3@v0.3.3540100 ``` ## Why Another Go SQLite Driver? SQLite is one of the most deployed pieces of software in the world, yet using it from Go has historically forced engineers into an uncomfortable choice between build simplicity and engine fidelity. On one side stands `github.com/mattn/go-sqlite3`, the longtime community standard. It wraps the official C SQLite amalgamation directly via `cgo`. You get the real C query planner, full WAL concurrency semantics, and predictable C memory management. The cost is `CGO_ENABLED=1`: local builds and CI pipelines slow down, minimal scratch or distroless container images require matching C runtime libraries, cross-compiling to another OS or CPU architecture requires a full C cross-toolchain, and any memory corruption in C shares the Go process address space. On the other side are pure-Go SQLite ports generated by mechanically translating the C amalgamation into Go source code over an emulated C runtime (`libc`). They restore `CGO_ENABLED=0`, which makes builds trivial. Under sustained workloads, however, the translated C heap and per-row Go allocations increase garbage-collection pressure, and subtle edge cases in locking, busy timeouts, or statement interruption can diverge from native C builds. We built `gosuda.org/sqlite3` so Go applications do not have to choose between `CGO_ENABLED=0` and the real C SQLite engine. By compiling the official SQLite `3.54.1` C amalgamation into a dedicated shared-memory server (`hqq-sqlite-server`) and communicating over `gosuda.org/hqq`, your Go binary compiles with `CGO_ENABLED=0` while executing unmodified, `-O3`-optimized C SQLite across a process boundary. ## Zero-Config Execution: From `go get` to `sql.Open` Replacing `github.com/mattn/go-sqlite3` in an existing Go codebase requires changing only the blank import path. The driver registers as `"sqlite3"` (and `"hqq-sqlite"` for explicit selection) and accepts the same DSN query strings: ```go package main import ( "context" "database/sql" "fmt" "log" _ "gosuda.org/sqlite3" ) func main() { dsn := "file:app.db?_journal_mode=WAL&_busy_timeout=5000&_txlock=immediate&_foreign_keys=on" db, err := sql.Open("sqlite3", dsn) if err != nil { log.Fatal(err) } defer db.Close() ctx := context.Background() if _, err := db.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)`); err != nil { log.Fatal(err) } if _, err := db.ExecContext(ctx, `INSERT INTO users (name) VALUES (?)`, "Ada Lovelace"); err != nil { log.Fatal(err) } var id int64 var name string if err := db.QueryRowContext(ctx, `SELECT id, name FROM users WHERE name = ?`, "Ada Lovelace").Scan(&id, &name); err != nil { log.Fatal(err) } fmt.Printf("user #%d: %s\n", id, name) } ``` ### Embedded SLSA-Attested Worker Binaries A natural question when hearing that a library uses a C worker process is: _"Do I have to install or ship a separate C binary alongside my Go program?"_ No. `gosuda.org/sqlite3` embeds prebuilt `hqq-sqlite-server` binaries for all six major targets directly inside platform-tagged Go files (`internal/embedded/embed__.go`). Because Go's build constraints compile only the file matching your target `GOOS` and `GOARCH`, cross-compiling `GOOS=linux GOARCH=arm64 go build` automatically embeds only the ~1.4 MiB `linux/arm64` server binary. | Target OS | Target Arch (`GOARCH`) | Embedded Binary | Build Toolchain & Provenance | | :---------- | :--------------------- | :------------------------------------ | :------------------------------------------------ | | **Linux** | `amd64`, `arm64` | `hqq-sqlite-server_linux_` | GCC `-O3`, SLSA Build L3 Provenance | | **macOS** | `arm64`, `amd64` | `hqq-sqlite-server_darwin_` | AppleClang `-O3`, SLSA Build L3 Provenance | | **Windows** | `amd64` | `hqq-sqlite-server_windows_amd64.exe` | MSVC `-O2`, SLSA Build L3 Provenance | | **Windows** | `arm64` | `hqq-sqlite-server_windows_arm64.exe` | `zig cc -target aarch64-windows-gnu -O3`, SLSA L3 | When `sql.Open` establishes its first connection, `autolaunch.go` materializes the embedded binary into the user cache directory (`$XDG_CACHE_HOME/hqq/bin//`), verifies its cryptographic SHA-256 digest against the pinned `ReleaseHashes` map in `hashes.go` via `gosuda.org/hqq/secureexec`, and spawns the worker with `--parent-stdin` so its lifetime is strictly bound to your Go process. ### Clean-Room `mattn/go-sqlite3` DSN Compatibility To work as a true drop-in replacement, the DSN parser (`dsn.go`) implements a clean-room replica of the connection and pragma knobs used across `mattn/go-sqlite3` codebases: | DSN Query Parameter | Accepted Values | SQLite Effect | | :-------------------------------------------------------------------------------------------- | :------------------------------------------------------ | :--------------------------------------------------------------------------------------------------- | | `_busy_timeout` / `_timeout` | Integer milliseconds (e.g. `5000`) | Calls `sqlite3_busy_timeout(db, ms)` on session open. | | `_journal_mode` / `_journal` | `DELETE`, `TRUNCATE`, `PERSIST`, `MEMORY`, `WAL`, `OFF` | Executes `PRAGMA journal_mode = `. | | `_txlock` | `immediate`, `exclusive`, `deferred` | Selects `BEGIN IMMEDIATE`, `BEGIN EXCLUSIVE`, or `BEGIN DEFERRED` for `db.BeginTx`. | | `_foreign_keys` / `_fk` | `on`, `true`, `1`, `off`, `false`, `0` | Executes `PRAGMA foreign_keys = ON/OFF`. | | `_synchronous` / `_sync` | `0`/`OFF`, `1`/`NORMAL`, `2`/`FULL`, `3`/`EXTRA` | Executes `PRAGMA synchronous = `. | | `_cache_size` | Integer pages or `-KiB` (e.g. `-20000`) | Executes `PRAGMA cache_size = `. | | `_loc` | `auto` or IANA timezone (e.g. `UTC`, `Asia/Seoul`) | Parses `DATETIME`/`TIMESTAMP` text columns into `*time.Time` in the target location. | | `mode` | `ro`, `rw`, `rwc`, `memory` | Sets `SQLITE_OPEN_READONLY`, `SQLITE_OPEN_READWRITE`, `SQLITE_OPEN_CREATE`, or `SQLITE_OPEN_MEMORY`. | | `_case_sensitive_like`, `_query_only`, `_recursive_triggers`, `_secure_delete`, `_AutoVacuum` | Boolean or mode integers | Applies the corresponding session `PRAGMA` on connection initialization. | ## Turning Process Isolation into a Throughput Advantage At first glance, running SQLite in a separate process sounds like it must be slower than calling `sqlite3_step` in-process via `cgo`. In practice, the opposite happens as soon as a query returns more than a single row. ### The Hidden Cost of Row-by-Row `cgo` Calls In a conventional `cgo` SQLite driver, `database/sql` drives iteration one row and one column at a time. Reading 1,000 rows with 4 columns triggers: - 1,000 `cgo` calls to `sqlite3_step` - 4,000 `cgo` calls to `sqlite3_column_type` and `sqlite3_column_int64` / `sqlite3_column_text` - Thousands of individual boundary crossings between the Go scheduler and the C stack ### Request Pipelining and 64 KiB Batch Prefetching `gosuda.org/sqlite3` eliminates per-row boundary crossings by redesigning the wire protocol around two mechanisms in `stmt.go`, `rows.go`, and `sqlite3/server/sqlite3_hooks.c`: ```text Go Client (stmt.go / rows.go) C11 Worker (sqlite3_hooks.c) ----------------------------- ---------------------------- 1. Pipeline in 1 flight: ResetAsync + BindValuesAsync + StmtInfoAsync + FetchRowsAsync(128) =======> reset -> bind -> stmt_info -> loop sqlite3_step() up to 128 rows <======= Batch 1 (rows 1..128 packed in 64 KiB slot) 2. Immediately fire prefetch: FetchRowsAsync(128) (Batch 2) =======> loop sqlite3_step() for rows 129..256 While C steps Batch 2, Go caller scans rows 1..128 from Batch 1! <======= Batch 2 ready in shared memory ``` 1. **Single-Flight Query Pipelining (`stmt.go`)**: When `QueryContext` or `Stmt.QueryContext` runs a read-only statement, `s.query()` submits `reset`, `bind_values`, `stmt_info` (on first execution), and `fetch_rows` asynchronously in a single shared-memory flight. All four operations execute back-to-back on the C worker and return in one round trip. 2. **Tight C Batch Loops and Asynchronous Prefetch (`rows.go`)**: Inside `fetch_rows` (`sqlite3_hooks.c`), the C worker steps `sqlite3_step(stmt)` in a native C loop, encoding up to `FetchSize` rows (default 128) directly into the 64 KiB shared-memory response slot until the batch limit is reached, the slot is full, or `SQLITE_DONE` occurs. As soon as `rows.ingest()` decodes Batch 1, it fires `FetchRowsAsync` for Batch 2 before returning the first row to the caller. While your Go code scans Batch 1, the C worker is already stepping Batch 2 on another CPU core. On Apple M4 (`darwin/arm64`), comparing `gosuda.org/sqlite3` (`CGO_ENABLED=0`) against `github.com/mattn/go-sqlite3` (`CGO_ENABLED=1`) on the same SQLite database shows the effect of batching and pipelining: | Workload | `gosuda.org/sqlite3` (`CGO_ENABLED=0`) | `mattn/go-sqlite3` (`CGO_ENABLED=1`) | Relative Speed | | :---------------------------------- | -------------------------------------: | ------------------------------------: | :----------------------------------- | | **Point Select (`QueryRow` by PK)** | `3.52 µs/op` (9 B, 1 alloc) | `3.56 µs/op` (232 B, 11 allocs) | **1.01x (parity, 11x fewer allocs)** | | **100-Row Scan (`rows.Next`)** | `20.41 µs/op` (30.5 KB, 211 allocs) | `36.34 µs/op` (21.8 KB, 309 allocs) | **1.78x faster** | | **1,000-Row Scan (`rows.Next`)** | `125.92 µs/op` (296 KB, 2,030 allocs) | `349.10 µs/op` (216 KB, 3,009 allocs) | **2.77x faster** | | **Prepared Insert (Autocommit)** | `5.33 µs/op` (0 B, 0 allocs) | `5.88 µs/op` (120 B, 5 allocs) | **1.10x faster (0 allocs)** | | **100-Row Transaction Batch** | `189.99 µs/op` (16 B, 1 alloc) | `191.50 µs/op` (12.1 KB, 503 allocs) | **1.01x (500x fewer allocs)** | ## Cancellation, Path Sandboxing, and Connection Pool Recovery Running SQLite in an isolated worker also strengthens how the driver handles timeouts, untrusted SQL, and filesystem boundaries: - **Deterministic Context Cancellation (`conn.go` and `errors.go`)**: Every driver call arms a lightweight `context.AfterFunc` guard (`cn.guard`). If the caller's `context.Context` is canceled or its deadline expires while a slow recursive query or lock wait is running on the server, the guard marks the connection bad, kills the local session link, and sends an out-of-band `Cancel()` frame. The C worker invokes `sqlite3_interrupt(db)` immediately, unblocking `sqlite3_step`, while the Go driver maps the resulting `SQLITE_INTERRUPT` status to `context.Canceled` or `context.DeadlineExceeded` and instructs `database/sql` to replace the connection in the pool. - **Directory Sandboxing (`sqlsrv_pathrules.c`)**: The C worker enforces `--allow-dir` path rules on every `open_v2` call, resolving symlinks and rejecting relative `..` traversal outside the allowed directory. `sqlite3_load_extension` and `SQLITE_LIMIT_ATTACHED` are disabled by default so SQL injection cannot escape the database directory or load arbitrary shared objects. ## Repository Layout The repository is structured so that the top-level package provides the drop-in `database/sql` driver while `launcher/` and `sqlite3/` remain accessible for advanced custom-server deployments: ```text gosuda.org/sqlite3 ├── driver.go / api.go # database/sql driver registration ("sqlite3") and Connector API ├── dsn.go # Clean-room mattn/go-sqlite3 DSN parser and pragma configuration ├── conn.go / tx.go # Session lifecycle, context guard, and transaction state machine ├── stmt.go / rows.go # Single-flight request pipelining and 64 KiB batch prefetching ├── errors.go # Typed SQLite primary/extended error codes and sentinel matching ├── autolaunch.go / hashes.go # Zero-config embedded server materialization and SHA-256 pinning ├── internal/embedded/ # Platform-tagged //go:embed binaries (linux, darwin, windows) ├── launcher/ # Configurable process supervisor for custom server deployments └── sqlite3/ # Generated hqq-bindgen Go client and C11 SQLite server source └── server/ # C11 server implementation (sqlite3_hooks.c, sqlsrv_policy.c) ``` ## Getting Started Install the module with `go get` and open your database with `sql.Open("sqlite3", dsn)`: ```bash go get gosuda.org/sqlite3@v0.3.3540100 ``` - **GitHub Repository**: [https://github.com/gosuda/hqq-sqlite3](https://github.com/gosuda/hqq-sqlite3) - **DSN & Pragma Reference**: [docs/DSN.md](https://github.com/gosuda/hqq-sqlite3/blob/main/docs/DSN.md) - **Architecture & Wire Protocol**: [docs/ARCHITECTURE.md](https://github.com/gosuda/hqq-sqlite3/blob/main/docs/ARCHITECTURE.md)