gosuda.org/sqlite3: A Drop-In database/sql Driver Running Native C SQLite Over Shared Memory

gosuda.org/sqlite3 is a CGO_ENABLED=0 database/sql driver for Go that runs the upstream C SQLite 3.54.1 amalgamation inside an isolated worker process over HQQ shared-memory rings. It registers under the standard "sqlite3" driver name, supports github.com/mattn/go-sqlite3 DSN query parameters, embeds SLSA-attested prebuilt server binaries for six OS and architecture targets, and uses request pipelining with 64 KiB batch prefetching to outperform in-process cgo on multi-row queries.

1go get gosuda.org/sqlite3@v0.3.3540100

Why Another Go SQLite Driver?

SQLite is one of the most deployed pieces of software in the world, yet using it from Go has historically forced engineers into an uncomfortable choice between build simplicity and engine fidelity.

On one side stands github.com/mattn/go-sqlite3, the longtime community standard. It wraps the official C SQLite amalgamation directly via cgo. You get the real C query planner, full WAL concurrency semantics, and predictable C memory management. The cost is CGO_ENABLED=1: local builds and CI pipelines slow down, minimal scratch or distroless container images require matching C runtime libraries, cross-compiling to another OS or CPU architecture requires a full C cross-toolchain, and any memory corruption in C shares the Go process address space.

On the other side are pure-Go SQLite ports generated by mechanically translating the C amalgamation into Go source code over an emulated C runtime (libc). They restore CGO_ENABLED=0, which makes builds trivial. Under sustained workloads, however, the translated C heap and per-row Go allocations increase garbage-collection pressure, and subtle edge cases in locking, busy timeouts, or statement interruption can diverge from native C builds.

We built gosuda.org/sqlite3 so Go applications do not have to choose between CGO_ENABLED=0 and the real C SQLite engine. By compiling the official SQLite 3.54.1 C amalgamation into a dedicated shared-memory server (hqq-sqlite-server) and communicating over gosuda.org/hqq, your Go binary compiles with CGO_ENABLED=0 while executing unmodified, -O3-optimized C SQLite across a process boundary.

Zero-Config Execution: From go get to sql.Open

Replacing github.com/mattn/go-sqlite3 in an existing Go codebase requires changing only the blank import path. The driver registers as "sqlite3" (and "hqq-sqlite" for explicit selection) and accepts the same DSN query strings:

 1package main
 2
 3import (
 4	"context"
 5	"database/sql"
 6	"fmt"
 7	"log"
 8
 9	_ "gosuda.org/sqlite3"
10)
11
12func main() {
13	dsn := "file:app.db?_journal_mode=WAL&_busy_timeout=5000&_txlock=immediate&_foreign_keys=on"
14	db, err := sql.Open("sqlite3", dsn)
15	if err != nil {
16		log.Fatal(err)
17	}
18	defer db.Close()
19
20	ctx := context.Background()
21	if _, err := db.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT NOT NULL)`); err != nil {
22		log.Fatal(err)
23	}
24	if _, err := db.ExecContext(ctx, `INSERT INTO users (name) VALUES (?)`, "Ada Lovelace"); err != nil {
25		log.Fatal(err)
26	}
27
28	var id int64
29	var name string
30	if err := db.QueryRowContext(ctx, `SELECT id, name FROM users WHERE name = ?`, "Ada Lovelace").Scan(&id, &name); err != nil {
31		log.Fatal(err)
32	}
33	fmt.Printf("user #%d: %s\n", id, name)
34}

Embedded SLSA-Attested Worker Binaries

A natural question when hearing that a library uses a C worker process is: "Do I have to install or ship a separate C binary alongside my Go program?"

No. gosuda.org/sqlite3 embeds prebuilt hqq-sqlite-server binaries for all six major targets directly inside platform-tagged Go files (internal/embedded/embed_<goos>_<goarch>.go). Because Go's build constraints compile only the file matching your target GOOS and GOARCH, cross-compiling GOOS=linux GOARCH=arm64 go build automatically embeds only the ~1.4 MiB linux/arm64 server binary.

When sql.Open establishes its first connection, autolaunch.go materializes the embedded binary into the user cache directory ($XDG_CACHE_HOME/hqq/bin/<sha256>/), verifies its cryptographic SHA-256 digest against the pinned ReleaseHashes map in hashes.go via gosuda.org/hqq/secureexec, and spawns the worker with --parent-stdin so its lifetime is strictly bound to your Go process.

Clean-Room mattn/go-sqlite3 DSN Compatibility

To work as a true drop-in replacement, the DSN parser (dsn.go) implements a clean-room replica of the connection and pragma knobs used across mattn/go-sqlite3 codebases:

Turning Process Isolation into a Throughput Advantage

At first glance, running SQLite in a separate process sounds like it must be slower than calling sqlite3_step in-process via cgo. In practice, the opposite happens as soon as a query returns more than a single row.

The Hidden Cost of Row-by-Row cgo Calls

In a conventional cgo SQLite driver, database/sql drives iteration one row and one column at a time. Reading 1,000 rows with 4 columns triggers:

• 1,000 cgo calls to sqlite3_step
• 4,000 cgo calls to sqlite3_column_type and sqlite3_column_int64 / sqlite3_column_text
• Thousands of individual boundary crossings between the Go scheduler and the C stack

Request Pipelining and 64 KiB Batch Prefetching

gosuda.org/sqlite3 eliminates per-row boundary crossings by redesigning the wire protocol around two mechanisms in stmt.go, rows.go, and sqlite3/server/sqlite3_hooks.c:

 1Go Client (stmt.go / rows.go)                       C11 Worker (sqlite3_hooks.c)
 2-----------------------------                       ----------------------------
 31. Pipeline in 1 flight:
 4   ResetAsync + BindValuesAsync
 5   + StmtInfoAsync + FetchRowsAsync(128)  =======>  reset -> bind -> stmt_info ->
 6                                                    loop sqlite3_step() up to 128 rows
 7                                          <=======  Batch 1 (rows 1..128 packed in 64 KiB slot)
 82. Immediately fire prefetch:
 9   FetchRowsAsync(128) (Batch 2)          =======>  loop sqlite3_step() for rows 129..256
10   While C steps Batch 2, Go caller
11   scans rows 1..128 from Batch 1!        <=======  Batch 2 ready in shared memory

1. Single-Flight Query Pipelining (stmt.go): When QueryContext or Stmt.QueryContext runs a read-only statement, s.query() submits reset, bind_values, stmt_info (on first execution), and fetch_rows asynchronously in a single shared-memory flight. All four operations execute back-to-back on the C worker and return in one round trip.
2. Tight C Batch Loops and Asynchronous Prefetch (rows.go): Inside fetch_rows (sqlite3_hooks.c), the C worker steps sqlite3_step(stmt) in a native C loop, encoding up to FetchSize rows (default 128) directly into the 64 KiB shared-memory response slot until the batch limit is reached, the slot is full, or SQLITE_DONE occurs. As soon as rows.ingest() decodes Batch 1, it fires FetchRowsAsync for Batch 2 before returning the first row to the caller. While your Go code scans Batch 1, the C worker is already stepping Batch 2 on another CPU core.

On Apple M4 (darwin/arm64), comparing gosuda.org/sqlite3 (CGO_ENABLED=0) against github.com/mattn/go-sqlite3 (CGO_ENABLED=1) on the same SQLite database shows the effect of batching and pipelining:

Cancellation, Path Sandboxing, and Connection Pool Recovery

Running SQLite in an isolated worker also strengthens how the driver handles timeouts, untrusted SQL, and filesystem boundaries:

• Deterministic Context Cancellation (conn.go and errors.go): Every driver call arms a lightweight context.AfterFunc guard (cn.guard). If the caller's context.Context is canceled or its deadline expires while a slow recursive query or lock wait is running on the server, the guard marks the connection bad, kills the local session link, and sends an out-of-band Cancel() frame. The C worker invokes sqlite3_interrupt(db) immediately, unblocking sqlite3_step, while the Go driver maps the resulting SQLITE_INTERRUPT status to context.Canceled or context.DeadlineExceeded and instructs database/sql to replace the connection in the pool.
• Directory Sandboxing (sqlsrv_pathrules.c): The C worker enforces --allow-dir path rules on every open_v2 call, resolving symlinks and rejecting relative .. traversal outside the allowed directory. sqlite3_load_extension and SQLITE_LIMIT_ATTACHED are disabled by default so SQL injection cannot escape the database directory or load arbitrary shared objects.

Repository Layout

The repository is structured so that the top-level package provides the drop-in database/sql driver while launcher/ and sqlite3/ remain accessible for advanced custom-server deployments:

 1gosuda.org/sqlite3
 2├── driver.go / api.go         # database/sql driver registration ("sqlite3") and Connector API
 3├── dsn.go                     # Clean-room mattn/go-sqlite3 DSN parser and pragma configuration
 4├── conn.go / tx.go            # Session lifecycle, context guard, and transaction state machine
 5├── stmt.go / rows.go          # Single-flight request pipelining and 64 KiB batch prefetching
 6├── errors.go                  # Typed SQLite primary/extended error codes and sentinel matching
 7├── autolaunch.go / hashes.go  # Zero-config embedded server materialization and SHA-256 pinning
 8├── internal/embedded/         # Platform-tagged //go:embed binaries (linux, darwin, windows)
 9├── launcher/                  # Configurable process supervisor for custom server deployments
10└── sqlite3/                   # Generated hqq-bindgen Go client and C11 SQLite server source
11    └── server/                # C11 server implementation (sqlite3_hooks.c, sqlsrv_policy.c)

Getting Started

Install the module with go get and open your database with sql.Open("sqlite3", dsn):

1go get gosuda.org/sqlite3@v0.3.3540100

• GitHub Repository: https://github.com/gosuda/hqq-sqlite3
• DSN & Pragma Reference: docs/DSN.md
• Architecture & Wire Protocol: docs/ARCHITECTURE.md