GoSuda

Running Docker-in-Docker Inside gVisor and Nested Sandboxes with gvx

By Lemon Mint views ...

Quick Solution: Securely Running docker:dind Inside an Unprivileged gVisor (runsc) Sandbox

Cloud AI agent platforms, Kubernetes agent-sandbox deployments (RuntimeClass: gvisor), and reinforcement learning evaluation harnesses isolate untrusted workloads inside gVisor (runsc) containers specifically to protect the host kernel. When a workload inside a default unprivileged runsc container tries to install docker.io or docker-ce via apt-get and start dockerd, it fails even if it has root (UID 0) inside the container.

With gvx (rungv), you can securely run docker:dind, docker build, and docker compose directly inside an unprivileged gVisor (runsc) container (or nest gvx inside gvx) without weakening the outer RuntimeClass, exposing host sockets, or granting --privileged.

Preserving and Strengthening Outer Sandbox Isolation: Running rungv inside runsc does not bypass a single restriction enforced by the outer gVisor sandbox. Instead, gvx creates a second, nested user-space gVisor Sentry kernel with its own hostlockdown (PR_SET_NO_NEW_PRIVS + seccomp), its own in-memory netstack, and a single-file ext4lite volume. Inner dockerd and runc containers are double-sandboxed: their system calls, mounts, and raw packets never leave the inner gvx user-space kernel.

 1# Inside an existing gVisor (runsc) container or Kubernetes Pod:
 2# Install rungv from a SLSA-attested GitHub release archive or via go install:
 3CGO_ENABLED=0 go install z.gfire.dev/gvx/cmd/rungv@latest
 4
 5mkdir -p ~/.gvx
 6rungv run \
 7  -p unix:~/.gvx/docker.sock:2375 \
 8  -v gvx-docker:/var/lib/docker \
 9  -e DOCKER_TLS_CERTDIR= \
10  docker:dind --tls=false &
11
12# Run standard Docker builds and containers against the inner gvx socket
13export DOCKER_HOST="unix://$HOME/.gvx/docker.sock"
14until docker info >/dev/null 2>&1; do sleep 0.5; done
15docker info --format '{{.Driver}} {{.OperatingSystem}}'
16# Output: overlayfs Alpine Linux ... (containerized)
17
18docker build -t nested-app:test .
19docker run --rm nested-app:test

Five Reasons Why Direct Docker Installation Fails Inside gVisor (runsc)

When a user or AI coding agent runs apt-get install -y docker.io (or docker-ce) inside a standard runsc container and attempts to start dockerd, it hits five sequential technical barriers.

1. Kernel Module Loading (modprobe) Against Virtual Kernel 4.4.0

During package installation and dockerd startup, initialization scripts run modprobe to load overlay, br_netfilter, ip_tables, nf_nat, and xt_conntrack. The outer gVisor Sentry blocks init_module(2) and finit_module(2) and reports a virtual kernel version (4.4.0) that has no /lib/modules/4.4.0 directory:

1modprobe: FATAL: Module overlay not found in directory /lib/modules/4.4.0
2modprobe: FATAL: Module br_netfilter not found in directory /lib/modules/4.4.0
3modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0
4modprobe: FATAL: Module nf_nat not found in directory /lib/modules/4.4.0

2. iptables-nft (nf_tables Netlink) and docker0 Bridge Failures

On Ubuntu 22.04, Ubuntu 24.04, and Debian 11/12, apt-get install docker.io selects iptables-nft (backed by the nf_tables netlink subsystem) as the default /usr/sbin/iptables alternative. In a standard gVisor container without network admin privileges or nf_tables netlink support, dockerd crashes while initializing the DOCKER NAT chain and the docker0 bridge:

1failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER
2iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
3iptables v1.8.10 (legacy): can't initialize iptables table 'nat': Table does not exist (do you need to insmod?)
4iptables: No chain/target/match by that name.
5Unable to enable SKIP DNAT rule: (iptables failed: iptables --wait -t nat -I DOCKER -i docker0 -j RETURN: iptables: No chain/target/match by that name.)
6Error initializing network controller: error obtaining controller instance: failed to create bridge "docker0": operation not permitted

3. overlay2 Failure on gofer / 9p Rootfs and vfs Disk Exhaustion

Unless the outer runsc administrator explicitly pre-configured a dedicated tmpfs or filestore mount at /var/lib/docker, the container rootfs is backed by gVisor's gofer (9p/lisafs) or a nested overlay mount. Mounting overlay2 on top of /var/lib/docker fails:

1level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
2failed to mount overlay: invalid argument
3Error starting daemon: error initializing graphdriver: driver not supported: overlay2

When users or agents attempt to bypass this error by starting dockerd --storage-driver=vfs, Docker disables Copy-on-Write layer sharing completely. Every RUN instruction in a Dockerfile performs a full recursive copy (dirCopy) of the entire root filesystem, multiplying a 500 MB image across 10 build steps into 5 GB of writes and terminating the sandbox with no space left on device or an I/O timeout.

4. Nested runc Cgroup and /proc Mount Rejections

Even if dockerd is forced to start with dockerd --iptables=false --bridge=none --storage-driver=vfs, running docker run or docker build invokes runc to create the inner container. In a default unprivileged gVisor container, runc fails when mounting cgroups, remounting /proc, or loading inner seccomp filters:

1docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process:
2error during container init: error mounting "cgroup" to rootfs at "/sys/fs/cgroup": mount cgroup:/sys/fs/cgroup/cpuset (via /proc/self/fd/6), flags: 0xf, data: cpuset: operation not permitted
3cgroups: cgroup mountpoint does not exist: unknown
4runc create failed: systemd not running on this host: cannot use systemd cgroup manager
5error during container init: error mounting "proc" to rootfs at "/proc": mount proc:/proc (via /proc/self/fd/6), flags: 0xe: operation not permitted
6error during container init: error loading seccomp filter into kernel: operation not permitted

5. Nested runsc Fails with SECCOMP_FILTER_FLAG_NEW_LISTENER (EBUSY)

If a user tries to run another runsc instance inside runsc using the systrap platform, initialization aborts immediately:

1creating subprocess: sysmsg: failed to initialize thread: failed to install seccomp notify rules: device or resource busy

Both the Linux kernel and the gVisor Sentry allow at most one SECCOMP_FILTER_FLAG_NEW_LISTENER (SECCOMP_RET_USER_NOTIF) listener per task hierarchy. Because the outer sandbox already holds that listener slot, the inner seccomp(2) call returns EBUSY (device or resource busy).

How gvx Solves Nested Execution Inside gVisor

gvx addresses all five barriers inside its own user-space kernel and platform layer, enabling rungv to run inside an outer gVisor (runsc) container or even recursively inside another gvx container (gvx in gvx).

Nested gvx Execution Inside Outer gVisor

1. Automatic systrap Fallback from EBUSY to the _RUN_SYSCALL_LOOP Futex Loop

In Linux systrap, each stub subprocess runs two kinds of threads:

  • Guest execution threads (sysmsg): Trap guest system calls via SECCOMP_RET_TRAP (SIGSYS) and binary-patched jmp rel32 trampolines (syshandler), exchanging register state with the Sentry over shared memory and SYS_FUTEX.
  • A stub syscallThread (syscall_thread.go): Executes address-space system calls (mmap, munmap, clone) inside the stub subprocess on behalf of the Sentry.

In upstream gVisor, syscallThread supports two wake-up modes in stub_amd64.s and stub_arm64.s:

  1. Seccomp User-Notify Loop (_RUN_SECCOMP_LOOP = 6, when t.seccompNotify != nil): During syscallThread.init(), t.installSeccompNotify() installs a SECCOMP_RET_USER_NOTIF listener (seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, stubSyscallRules)) so the Sentry can dispatch requests via ioctl(SECCOMP_IOCTL_NOTIF_SEND) and ioctl(SECCOMP_IOCTL_NOTIF_RECV).
  2. Futex Syscall Loop (_RUN_SYSCALL_LOOP = 5, when t.seccompNotify == nil): The syscallThread runs syscall_loop (stub_amd64.s / stub_arm64.s), sleeping on SYS_FUTEX (FUTEX_WAIT) over the shared 2-page syscallSentryMessage (PROT_READ in the stub) and syscallStubMessage (PROT_READ | PROT_WRITE in the stub) region.

Upstream gVisor unconditionally calls t.installSeccompNotify() whenever seccompNotifyIsSupported is true, and aborts stub creation if seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_NEW_LISTENER, ...) returns EBUSY.

In gvx (pkg/sentry/platform/systrap/seccomp_notify_xplat.go and syscall_thread.go), syscallThread.init() calls t.installSeccompNotifyIfFree():

 1// pkg/sentry/platform/systrap/seccomp_notify_xplat.go
 2var seccompNotifyBusy atomic.Bool
 3
 4func (t *syscallThread) installSeccompNotifyIfFree() (*os.File, error) {
 5	if seccompNotifyBusy.Load() {
 6		return nil, nil
 7	}
 8	f, err := t.installSeccompNotify()
 9	if errors.Is(err, unix.EBUSY) {
10		if seccompNotifyBusy.CompareAndSwap(false, true) {
11			log.Warningf("Another seccomp listener exists in the filter chain of the host process. " +
12				"The stub threads use the slower futex loop instead of seccomp-unotify.")
13		}
14		return nil, nil
15	}
16	return f, err
17}

When installSeccompNotify() returns unix.EBUSY because an outer gVisor (runsc) or outer gvx sandbox already holds the process filter chain's single listener slot:

  • installSeccompNotifyIfFree() sets seccompNotifyBusy (atomic.Bool), logs one warning for the process, and leaves t.seccompNotify == nil for that stub and all subsequent stubs.
  • In syscallThread.detach() (syscall_thread_amd64.go and syscall_thread_arm64.go), t.seccompNotify == nil sets regs.Rbx = _RUN_SYSCALL_LOOP (5) on amd64 (or regs.Regs[9] = _RUN_SYSCALL_LOOP on arm64) instead of _RUN_SECCOMP_LOOP (6).
  • In syscallThread.syscall() (syscall_thread.go), the inner Sentry dispatches stub syscalls purely through shared memory and SYS_FUTEX:
1// pkg/sentry/platform/systrap/syscall_thread.go
2// Notify the syscall thread about a new syscall request.
3atomic.AddUint32(&sentryMsg.state, 1)
4futexWakeUint32(&sentryMsg.state)
5
6// Wait for reply.
7futexWaitWake(&sentryMsg.state, atomic.LoadUint32(&sentryMsg.state))

Because SECCOMP_RET_TRAP filters on guest threads stack arbitrarily deep and syscallThread falls back to _RUN_SYSCALL_LOOP without requesting a second user-notification listener, gvx runs cleanly inside runsc and supports recursive nesting (gvx inside gvx).

2. Native overlayfs on Single-File ext4lite Volumes

To avoid mounting overlayfs on top of the outer gVisor container's gofer (9p/lisafs) root filesystem, -v gvx-docker:/var/lib/docker backs /var/lib/docker with gvx's internal ext4lite filesystem stored in a single regular file (gvx-docker.ext4).

To outer runsc, gvx-docker.ext4 is just one regular file accessed via pread64(2) and pwrite64(2). Inside gvx, ext4lite provides a JBD2-journaled ext4 filesystem with trusted.overlay.* and security.capability extended attributes, character device whiteout nodes (0:0), d_type directory entries, and RENAME_EXCHANGE, allowing the inner dockerd to use its native overlayfs storage driver.

3. Self-Contained netstack with docker0 and iptables-legacy

Inside gvx, dockerd does not interact with the outer runsc container's network interfaces or netfilter tables. Instead:

  • dockerd creates its docker0 bridge and veth pairs inside gvx's own user-space netstack.
  • All iptables-legacy rules (-t nat -N DOCKER, MASQUERADE, conntrack, addrtype) execute against gvx's internal netfilter implementation.
  • Nested runc containers mount their /proc, /sys, and /sys/fs/cgroup hierarchies inside gvx's in-memory VFS2 mount table, never calling the outer runsc mount(2) syscall.

Verifying Nested docker:dind Inside gVisor (or gvx in gvx)

You can verify both scenarios (rungv inside runsc, or rungv inside rungv) with the following commands:

 1# Inside an outer gVisor (runsc) container or an outer rungv container:
 2mkdir -p ~/.gvx
 3rungv run \
 4  -p unix:~/.gvx/docker.sock:2375 \
 5  -v gvx-docker:/var/lib/docker \
 6  -e DOCKER_TLS_CERTDIR= \
 7  docker:dind --tls=false &
 8DIND_PID=$!
 9
10export DOCKER_HOST="unix://$HOME/.gvx/docker.sock"
11until docker info >/dev/null 2>&1; do sleep 0.5; done
12
13# Verify that the inner Docker daemon is using native overlayfs (not vfs)
14docker info --format 'Storage Driver: {{.Driver}}'
15# Output: Storage Driver: overlayfs
16
17# Run a multi-layer build and execute a nested container
18mkdir -p /tmp/nested-test && cd /tmp/nested-test
19cat <<'EOF' > Dockerfile
20FROM alpine:3.20
21RUN adduser -D -u 1001 appuser && mkdir -p /app && chown appuser:appuser /app
22USER appuser
23WORKDIR /app
24RUN echo "nested-build-ok" > status.txt
25CMD ["cat", "/app/status.txt"]
26EOF
27
28docker build -t nested-test:latest .
29docker run --rm nested-test:latest
30# Output: nested-build-ok
31
32kill -INT $DIND_PID

Further Reading in This Series

  1. gvx: Running Linux Containers, Docker, and Kubernetes in One Unprivileged Process: Overview of gvx across Windows (whpx, systrap), macOS (hvf), and Linux (systrap).
  2. Running Docker, docker build, and docker compose Without Root or User Namespaces: How gvx runs docker:dind when root, sudo, unshare(CLONE_NEWUSER), mount(2), and /dev/fuse are blocked.
  3. Running Docker-in-Docker Inside gVisor and Nested Sandboxes with gvx (this post)
  4. Fast Isolated Docker Environments for AI Agent RL Rollouts with gvx: How to use gvx and single-file ext4lite copy-on-write snapshots to run hundreds of parallel SWE-bench and RL rollout environments with 100-millisecond resets.