WhitepaperSandboxAI Infrastructure

StateSet Sandbox: A Kubernetes-Native Execution Platform for AI Agents

Version 3.1 · March 2026

StateSet Engineering

Platform Engineering

Mar 4, 202635 min read

0% complete

StateSet Sandbox: A Kubernetes-Native Execution Platform for AI Agents

Version 3.1 — March 2026

Authors: StateSet Engineering

Abstract

Large language models are increasingly deployed as autonomous agents that generate, execute, and iterate on code in real time. This creates a new class of infrastructure requirement: secure, multi-tenant execution environments that can be provisioned in milliseconds, enforce strict isolation boundaries, and tear down without residue - all at a pace that does not break the agentic loop. Latencies above 200 milliseconds degrade the quality of agentic reasoning; latencies above 2 seconds break interactive workflows entirely. Existing solutions force a tradeoff between speed (local execution, WebAssembly), security (VMs, gVisor), and operational maturity (Kubernetes) - no single approach satisfies all three.

StateSet Sandbox is a Kubernetes-native platform purpose-built for this workload. It reduces sandbox creation latency from the 3-10 seconds typical of cold Kubernetes pod scheduling to under 100 milliseconds through a Redis-backed warm pod pool with O(1) age-priority claiming via sorted sets. Every sandbox runs inside an isolated pod with configurable runtime classes - gVisor, Kata Containers, Firecracker microVMs, or standard containers - with a read-only root filesystem, dropped capabilities, and no access to the host kernel or cloud metadata services. An 8-layer defense-in-depth security model ensures that a compromise at any single layer does not grant access to the host, other tenants, or cloud infrastructure.

The platform provides a comprehensive agent runtime: code execution with SSE streaming, interactive REPL sessions (Jupyter kernels), computer-use capabilities (screenshot, mouse, keyboard), 10-endpoint file management, and native Model Context Protocol (MCP) integration. SDKs span 10 languages, a CLI, and 8 AI framework integrations. This paper describes the architecture, security model, performance characteristics, failure modes, and empirical benchmarks of StateSet Sandbox as deployed in production.

Introduction
Design Principles
System Architecture
Sandbox Lifecycle
Warm Pool Architecture
Execution Engine
Security Model
Agent Primitives
State Management
Failure Analysis and Recovery
Empirical Performance Evaluation
Competitive Landscape
Future Work
Conclusion
References

Appendices: A: API Surface · B: SDK Ecosystem · C: Deployment Architecture · D: Configuration Reference · E: Database Schema

1. Introduction

1.1 The Problem

The transition from LLMs as text generators to LLMs as autonomous agents has exposed a fundamental gap in infrastructure. When a model generates code and needs to execute it - to test a hypothesis, run a build, install dependencies, or interact with an API - it requires a runtime environment. That environment must satisfy competing demands:

Speed: An agent in a tight generate-execute-observe loop cannot wait seconds for its environment. Latencies above 200ms degrade reasoning quality [1]; above 2 seconds, interactive workflows break.
Security: Code is untrusted by definition - generated by a probabilistic model that may produce malicious commands, attempt privilege escalation, or exfiltrate data [2].
Multi-tenancy: Resource isolation, metered billing, and per-tenant rate limits must be enforced without adding latency to the hot path.
Statefulness: Agents need to persist intermediate results, restore previous states, and share artifacts across sessions.
Richness: Modern agents require REPL sessions, GUI automation, structured file operations, and integration with external tools via standardized protocols (MCP) [3].

Existing approaches fail to satisfy all five:

Approach	Speed	Security	Multi-tenancy	Statefulness	Richness
Local execution	Excellent	None	None	Process-level	Limited
Docker (no orchestration)	Good	Moderate	Manual	Volume mounts	Moderate
Kubernetes pods (cold)	Poor (3-10s)	Good	Good	emptyDir only	Good
Firecracker / QEMU	Poor (1-5s)	Excellent	Good	Snapshot-based	Excellent
WebAssembly	Excellent	Good	Limited	Limited	Limited

1.2 The StateSet Sandbox Approach

StateSet Sandbox resolves these tradeoffs by combining Kubernetes orchestration with a warm pod pool, multi-tier kernel-level isolation, and a purpose-built execution protocol. The key insight: Kubernetes scheduling is slow, but a pre-scheduled pod claimed from a Redis sorted set is fast. By decoupling pod creation from pod assignment, the platform achieves sub-100ms provisioning while retaining the security and operational maturity of the Kubernetes ecosystem.

2. Design Principles

Latency first: Pre-warm pods so creation is bounded by Redis RTT (~2ms), not Kubernetes scheduling (~3-10s). Defer all K8s API calls off the critical path.

Isolation as the floor: gVisor kernel interception, non-root UIDs, read-only root filesystems, dropped capabilities, and network policies are non-negotiable defaults. Five isolation tiers (container, gVisor, Kata, Firecracker, WASM) allow operators to choose the right tradeoff.

Defense in depth: Eight independent security layers ensure a breach at any single layer does not compromise the host, other tenants, or cloud infrastructure (Section 7).

Fail-fast configuration: All config is validated at startup with Zod schemas (Node.js) or typed env parsing (Rust). Missing required values cause process.exit(1). No random fallback keys, no wildcard CORS, no silent degradation.

Agent-native primitives: REPL sessions, computer-use, checkpoints, MCP lifecycle, agent sessions with budget enforcement, and per-sandbox RBAC are first-class concepts, not afterthoughts.

3. System Architecture

3.1 Component Overview

Note: ASCII diagrams are used for portability. Vector renderings (SVG) are available in docs/diagrams/.

┌──────────────────────────────────────────────────────────────────┐
│                        Client Layer                              │
│  10 SDKs · CLI · 8 AI Framework Integrations                    │
└──────────────────────────┬───────────────────────────────────────┘
                           │ HTTPS / WebSocket / SSE
┌──────────────────────────▼───────────────────────────────────────┐
│                    Controller Layer (3 replicas)                  │
│  Node.js (production) │ Rust (next-gen) │ Cloud SQL proxy        │
│  46 routes · 19 middleware · 45 services                         │
└──────────────────────────┬───────────────────────────────────────┘
          ┌────────────────┼─────────────────┐
┌─────────▼──────────┐ ┌──▼─────────┐ ┌─────▼──────────┐
│   PostgreSQL       │ │   Redis    │ │  Kubernetes     │
│  orgs, billing,    │ │  warm pool │ │  sandbox pods,  │
│  checkpoints,      │ │  (ZADD/    │ │  RBAC, network  │
│  audit, secrets    │ │  ZPOPMIN)  │ │  policies       │
└────────────────────┘ └────────────┘ └─────▼──────────┘
                                            │
                                ┌───────────┴────────────┐
                                │     Sandbox Pods       │
                                │  Go execd daemon:      │
                                │  ├─ exec + streaming   │
                                │  ├─ file ops (10 ep)   │
                                │  ├─ Jupyter/REPL       │
                                │  ├─ GUI/computer-use   │
                                │  └─ Prometheus :9998   │
                                │  MCP servers           │
                                └────────────────────────┘

3.2 Data Flow

Create: Agent calls POST /api/v1/sandbox/create. Controller authenticates, checks RBAC + rate limits + concurrency budgets, claims a warm pod via ZPOPMIN. Returns sandbox_id in ~36ms (p50).
Execute: Agent calls POST /api/v1/sandbox/:id/execute. Controller routes to the in-pod Go execd daemon over persistent TCP. Output streams via SSE.
Interact: File ops, REPL sessions, computer-use, MCP servers, checkpoints, artifacts - each enforcing org ownership, RBAC roles, plan limits, and security rules.
Complete: DELETE /api/v1/sandbox/:id. Pod recycled or terminated. Usage events recorded. Lifecycle events emitted.

3.3 Dual-Controller Strategy

The platform ships two controller implementations with identical API surfaces:

Node.js (Express, TypeScript): Production. Mature, full feature parity. Baseline: 256-512Mi.
Rust (Axum, Tokio): Next-generation. ~50Mi baseline, deterministic latency, better throughput. Feature parity in progress.

Traffic routes exclusively to Node.js in production. The Rust controller runs in shadow mode for benchmarking. Per-route canary shifting (1% → 10% → 50% → 100%) via Layer 7 routing enables incremental migration. Routes requiring features not yet in Rust remain pinned to Node.js.

4. Sandbox Lifecycle

4.1 Creation Pipeline

Request ──────────────────────────────────────────► Response
  │                                                    │
  ├─ 1. Schema validation (Zod)                        │
  ├─ 2. Auth: JWT or API key → org_id, scopes          │
  ├─ 3. Per-sandbox RBAC permission check              │
  ├─ 4. Rate limit (Redis sliding window)              │
  ├─ 5. Concurrency budget (per-plan limit)            │
  ├─ 6. Warm pool claim (ZPOPMIN) or cold create       │
  ├─ 7. Pod configuration (sync / async / deferred)    │
  └─ 8. Response: sandbox_id, expires_at, metrics ────►│

Isolation selection: container (default) | gvisor (guest kernel) | kata (lightweight VM) | firecracker (microVM) | wasm (WebAssembly). Each maps to a Kubernetes RuntimeClass.

Configuration modes control the latency/readiness tradeoff:

Mode	K8s calls on create	First-exec overhead	Latency
Synchronous	Patch labels + env	None	~50ms
Async configure	None (background)	None (usually done)	~20ms
Deferred configure	None	Patch on first call	~10ms

Production recommendation: WARM_POOL_FAST_CLAIM=true + WARM_POOL_DEFERRED_CONFIGURE=true.

4.2 Expiration, Recycling, and Extension

Each sandbox has an expiresAt timestamp. A background reaper runs every 60 seconds with a circuit breaker (pauses after 5 consecutive K8s API failures). When WARM_POOL_REUSE_PODS=true, stopped sandboxes are sanitized (kill processes → wipe workspace → increment epoch → validate health) and returned to the pool. Epoch tokens prevent stale clients from executing against recycled pods.

5. Warm Pool Architecture

The warm pool is the platform's primary performance mechanism. It decouples pod scheduling from pod assignment.

5.1 Data Structure

Redis Key Space (per namespace, per profile):

  warm-pool:{ns}:available:{profile-hash}  → SortedSet<score=timestamp, member=pod_name>
  warm-pool:{ns}:creating:{profile-hash}   → Set<pod_name>
  warm-pool:{ns}:claimed                   → Set<pod_name>
  warm-pool:{ns}:leader                    → String (controller_id, TTL 30s)

Sorted sets store pods with Date.now() as score, enabling age-priority claiming - oldest pods claimed first, preventing staleness.

5.2 Claim Algorithm

def claim_warm_pod(profile):
    # O(1) age-priority claim
    [pod_name, score] = ZPOPMIN warm-pool:{ns}:available:{profile.hash}
    if pod_name is None:
        return MISS → cold create (3-10s)

    if FAST_CLAIM:
        SADD warm-pool:{ns}:claimed pod_name
        return pod_name                      # skip K8s verify

    # Synchronous verify
    pod = k8s.get(pod_name)
    if pod is None or pod.status != Running:
        retry (up to 3 times)
    k8s.patch(pod_name, labels={org_id, sandbox_id})
    SADD warm-pool:{ns}:claimed pod_name
    return pod_name

Fast claim trades consistency for speed: if a pod was evicted between insertion and claim, ZPOPMIN returns a dead name. Retry rate in practice: <0.1%.

Local fallback: When Redis is unavailable, the pool degrades to an in-memory localAvailableSorted array sorted by createdAt. Pool sizes use ZCARD (Redis) or array length (local).

5.3 Leader Election and Replenishment

Multiple controllers run behind a load balancer. Only the leader (elected via SET NX EX 30) calls fillPool() every 10 seconds. Non-leaders handle claims but do not create pods. Kubernetes Lease-based election provides stronger guarantees for production.

5.4 Profile Configuration

Heterogeneous workloads are supported through profile-based pooling. Each profile is a tuple of (cpus, memory, isolation, image). Image-specific profiles allow warm pools for specialized workloads (Claude Code, Playwright+Chromium, desktop/VNC).

6. Execution Engine

6.1 Go Execd Daemon

The Go execd daemon (docker/execd/, 37 source files) runs inside each sandbox pod on port 9999. It provides:

Command execution with configurable timeout, output limits (10MB default), and SSE streaming
File operations: glob, mkdir, rm, mv, cp, chunked read/write
Jupyter kernel management: HTTP + WebSocket client for REPL sessions
GUI automation: screenshot (scrot), mouse/keyboard (xdotool), screen dimensions
Prometheus metrics on port 9998

6.2 Binary Protocol

┌─────────┬──────────┬────────────────┬─────────────────┐
│ version │ msg_type │ payload_length │ payload (JSON)  │
│ 1 byte  │ 1 byte   │ 4 bytes (LE)   │ variable        │
└─────────┴──────────┴────────────────┴─────────────────┘

13 message types: EXEC (1-2), HEARTBEAT (3-4), SANITIZE (5-6), SET_EPOCH (7), FILE (8-9), JUPYTER (10-11), STREAMING (12), GUI (13).

Every request includes an HMAC-SHA256 signature over the payload using EXEC_AGENT_HMAC_SECRET. The controller maintains persistent per-sandbox TCP connections, pre-warmed at pod claim time. Output is truncated at MAX_OUTPUT_BYTES with a [truncated] marker.

6.3 Streaming

SSE streaming supports resumable execution via X-Execution-ID headers. Clients reconnect at byte offsets after network disconnects. A distributed stream coordinator ensures consistency across controller replicas.

7. Security Model

7.1 Threat Model

Malicious LLM-generated code: privilege escalation, container escape, data exfiltration, host kernel exploitation [2]
Cross-tenant probing: unauthorized access to other tenants' sandboxes, secrets, or network traffic
Cloud metadata exploitation: access to provider metadata services (169.254.169.254) for credential theft [4]
Insider threat: legitimate user accessing sandboxes belonging to other users within the same organization

7.2 Defense-in-Depth Stack

Layer 7: Per-sandbox RBAC (owner/operator/viewer permissions)
Layer 6: Security rules engine (command/file content scanning, regex patterns)
Layer 5: Egress allowlists + TCP enforcement (domain-level, fail-closed proxy)
Layer 4: Kubernetes NetworkPolicy (pod-to-pod isolation, metadata API block)
Layer 3: Runtime isolation (gVisor / Kata / Firecracker / WASM)
Layer 2: Linux security context (non-root, read-only root, no capabilities, seccomp)
Layer 1: Kubernetes RBAC (minimal service account, no automount)
Layer 0: API authentication (JWT + API key with scoped permissions)

7.3 Authentication

JWT: HS256, JWT_SECRET min 32 chars (enforced at startup). Claims: org_id, user_id, exp (24h). Used by dashboards, SSO.

API Key: sk-sandbox-{32-char-random}. SHA-256 hashed in PostgreSQL; plaintext never stored after creation. In-process validation cache (5min TTL). Fine-grained scopes (sandbox:create, sandbox:read, sandbox:write).

7.4 Pod-Level Isolation

Every sandbox pod runs with a non-negotiable security context:

securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities: { drop: ["ALL"] }
  seccompProfile: { type: RuntimeDefault }
automountServiceAccountToken: false

Writable mounts: /workspace, /tmp, /var/tmp (all emptyDir), /dev/shm (optional, for browser automation).

7.5 Network Security

Sandbox egress: HTTP (80), HTTPS (443), DNS (53 UDP/TCP) only
Pod-to-pod: sandboxes cannot reach each other
Metadata API: blocked (169.254.169.254, metadata.google.internal)
Egress TCP enforcement: defense-in-depth NetworkPolicy denying direct TCP, allowing only DNS + egress proxy (port 15001)
Egress auditing: all outbound connections logged to egress_audit_events (configurable retention)

7.6 Per-Sandbox RBAC

Role	Permissions
owner	All operations (create, read, write, execute, delete)
operator	Read, write, execute (no delete, no config changes)
viewer	Read-only (list, get, file read, screenshot)

Enforced at the route level via requireSandboxPermission(permission) middleware on all 46 route files. Falls back to permissive when no role is assigned (backwards compatibility).

7.7 Encryption

Secrets at rest: AES-256-GCM, scrypt KDF (N=16384, r=8, p=1). No random fallback keys.
API keys: SHA-256 hashed; plaintext never stored.
Env injection: written to temp file inside pod, then sourced (prevents shell interpolation attacks).

8. Agent Primitives

8.1 REPL Sessions

Interactive code execution sessions backed by Jupyter kernels maintain persistent interpreter state across multiple execute calls:

POST /sandbox/:id/repl/sessions              → create session (Python, JS, Go, Rust...)
POST /sandbox/:id/repl/sessions/:sid/execute → execute code (returns outputs, status)
POST /sandbox/:id/repl/sessions/:sid/interrupt → interrupt execution
DELETE /sandbox/:id/repl/sessions/:sid       → destroy session

Sessions maintain kernel state (variables, imports, open files) across executions. Each sandbox supports multiple concurrent sessions. Per-execution timeout up to 600s.

8.2 Computer Use

GUI automation for desktop applications, browsers, and graphical interfaces:

GET  /sandbox/:id/screenshot     → base64 PNG
POST /sandbox/:id/mouse          → click, move, double_click, drag
POST /sandbox/:id/keyboard       → type, key, hotkey
GET  /sandbox/:id/screen         → width, height

Execution path: controller sends GUI_REQUEST (msg type 13) to Go execd. Falls back to shell commands (scrot, xdotool, xdpyinfo) when native GUI ops are unavailable.

8.3 MCP Integration

Native Model Context Protocol server lifecycle management. 14 pre-installed domain-specific servers (Shopify, Stripe, Gorgias, Zendesk, Klaviyo, ShipStation, Loop Returns, etc.) plus support for any MCP-compatible server.

8.4 Agent Sessions

Higher-level abstraction combining sandbox, MCP servers, and budget management:

Budget enforcement: hard stops on cost cap or iteration limit
Automatic rotation: new sandbox after configurable interval, with optional file preservation
Session lifecycle: start → pause → resume → stop / cancel / reattach
Tool registration: custom tools alongside MCP servers

8.5 Multi-Tenancy and Billing

Per-org rate limiting (Redis sliding window), per-plan concurrency budgets (3/10/25/100 concurrent sandboxes), queue mode with backpressure (202 Accepted), trial credits ($2.00 on registration), Stripe metered billing on three dimensions (compute, storage, egress).

9. State Management

9.1 Checkpoint v2: Content-Addressable Storage

Checkpoints capture complete sandbox state. v2 uses content-addressable, deduplicated storage:

Data model: checkpoint_chunks (keyed by SHA-256 hash, ref-counted) + checkpoint_manifests (file → chunk mapping)
Write path: enumerate files → rolling content-defined chunking (64KB window) → dedup by hash → upload only new chunks
Read modes: prefer (default, v2 first) | fallback (v1 first) | strict (v2 only)
GC: periodic sweep deletes chunks with ref_count <= 0 beyond stale threshold

Result: 80-95% storage savings for successive checkpoints of mostly-unchanged workspaces.

9.2 Artifacts

Files persisted to cloud storage (GCS/S3/Azure Blob) with presigned download URLs (1-24h). Multipart upload for large files (initiate → upload parts → complete). Max 100 MiB.

10. Failure Analysis and Recovery

10.1 Failure Modes

Failure	Impact	Detection	Recovery	RTO
Redis outage	No warm pool claims; cold-start fallback	Health check failure	In-memory `localAvailableSorted` array; rate limiting degrades to per-instance	<1s (automatic)
Controller crash	1 of 3 replicas lost; K8s restarts	Liveness probe fail	K8s restart + readiness gate; PDB ensures 2 always available	15-30s
K8s API saturation	Pool replenishment stalls; cold creates fail	Circuit breaker trips after 5 failures	Warm pool drains naturally; existing sandboxes unaffected; breaker auto-resets	Self-healing
PostgreSQL outage	Auth cache hits serve traffic; writes fail	Connection error	Cloud SQL proxy reconnects; advisory lock prevents migration conflicts	5-30s (managed DB failover)
Node failure (sandbox)	Running sandboxes on that node are lost	Pod status → Failed	Client receives error; retry creates new sandbox from warm pool (~36ms)	<100ms (client retry)
Network partition	Split-brain risk for leader election	Lease expiry (30s)	Multiple leaders may create duplicate pods (harmless); claims still work	30s (lease expiry)
Exec-agent connection loss	Execution fails for that sandbox	TCP error / timeout	Controller retries connection; stale connections detected automatically	<5s (reconnect)
Warm pool exhaustion	All claims miss	`hit_rate < 85%` alert	Cold-start fallback (3-10s); leader auto-replenishes on next fill cycle	10s (fill interval)

10.2 Consistency Guarantees

The platform provides at-most-once execution semantics by default. The Idempotency-Key header enables exactly-once semantics for creation operations (24h cache, PostgreSQL-backed in production).

Warm pool consistency: ZPOPMIN is atomic - no two controllers can claim the same pod. Fast claim mode trades verification for speed: a claimed pod may have been evicted (race window ~ms). Retry logic (3 attempts) and cold-start fallback make this self-healing. In production, the conflict rate is <0.1%.

Checkpoint consistency: v2 chunk ref-counts are updated in a PostgreSQL transaction. Concurrent checkpoints of the same sandbox are serialized by advisory lock. GC only deletes chunks with ref_count <= 0 beyond a stale threshold, preventing premature deletion during concurrent writes.

10.3 Recovery Time Objectives

Component	RTO	RPO	Strategy
Controller	15s	0 (stateless)	K8s restart, 3 replicas, PDB
Warm pool state	<1s	~10s (last fill)	Redis persistence + in-memory fallback
Database	30s	0 (synchronous replication)	Managed DB failover (Cloud SQL / RDS Multi-AZ)
Active sandboxes	N/A	N/A	Ephemeral by design; recreate from checkpoint if needed
Audit/usage events	0	0	Write-ahead to PostgreSQL; retained per policy

10.4 Graceful Degradation Hierarchy

Full system         → All features available, warm pool active
Redis down          → Cold-start only, per-instance rate limiting
PostgreSQL down     → Cached auth works, no new orgs/keys, no checkpoints
K8s API saturated   → Existing sandboxes work, no new creates
Controller at 1/3   → Reduced throughput, all features available

The system is designed so that each dependency failure degrades a specific capability rather than causing a full outage. Existing sandboxes continue to execute commands even during controller or infrastructure failures - the exec-agent runs independently inside the pod.

11. Empirical Performance Evaluation

11.1 Methodology

All benchmarks were conducted on the production GKE cluster:

Cluster: GKE Standard, us-central1, Kubernetes 1.33
Controller: 3 replicas, 250m-1000m CPU, 512Mi-1Gi memory
Sandbox nodes: c2d-standard-8 (8 vCPU, 32 GB), gVisor runtime class
Warm pool: 40 pods (mixed 1-CPU and 2-CPU profiles)
Redis: Memorystore, 1 GB, single replica
PostgreSQL: Cloud SQL, 2 vCPU, 8 GB RAM, SSD

Load generator: k6 scripts (benchmarks/) running from a VM in the same region. Each test runs 5 iterations of 1,000 requests. Reported values are medians across iterations with p50/p95/p99 percentiles.

11.2 Sandbox Provisioning Latency

Path	p50	p95	p99	n
Warm (fast claim + deferred)	36ms	72ms	96ms	5,000
Warm (sync configure)	52ms	110ms	180ms	5,000
Cold (image cached)	4.2s	6.8s	9.1s	500
Cold (image pull)	12.4s	18.2s	28.6s	100

The warm path is dominated by auth middleware (1-3ms), validation (2-5ms), Redis ZPOPMIN (1-2ms), and response serialization (1-2ms). The cold path is dominated by image pull (when not cached) and gVisor runtime initialization (~500ms).

11.3 Command Execution Overhead

Backend	p50	p95	p99	n
Go execd (persistent TCP)	8ms	14ms	22ms	10,000
K8s exec API	32ms	58ms	95ms	5,000
kubectl subprocess	72ms	120ms	185ms	5,000

Overhead measured as time from controller receiving the request to first byte of output. Command: echo ok (minimal execution time).

11.4 Comparative Provisioning Latency

We benchmarked warm-start provisioning latency against publicly reported numbers from competing platforms. Where possible, we reproduced benchmarks using each platform's public API and SDK. Where not possible (no public access), we cite the platform's own published numbers.

Platform	Warm p50	Warm p99	Method
StateSet Sandbox	36ms	96ms	Measured (this study)
E2B [5]	~150ms	~400ms	E2B published benchmarks (2025)
Daytona [6]	~90ms	~250ms	Daytona blog post (2025)
Cloud Run	~1,800ms	~4,200ms	Measured (us-central1, min-instances=1)
AWS Lambda (provisioned)	~12ms	~45ms	Measured (us-east-1, 512MB)
AWS Lambda (cold)	~800ms	~3,500ms	Measured (us-east-1, 512MB)

Note: Lambda provides function-level isolation (no filesystem, no REPL, no MCP), not sandbox-level isolation. It is included as a latency reference point, not a feature-equivalent comparison.

11.5 Throughput Under Load

Sustained request rate with 40-pod warm pool, 3 controller replicas:

Concurrent clients	Requests/sec	Pool hit rate	p99 create latency
10	48	99.8%	68ms
50	142	98.2%	95ms
100	198	94.1%	145ms
200	210	82.3%	4,200ms (cold fallback)

At 200 concurrent clients, the warm pool begins to exhaust, causing cold-start fallback. The pool replenishment rate (~4 pods/10s) becomes the bottleneck. Scaling the pool to 80 pods maintains >95% hit rate at 200 concurrent clients.

11.6 Memory Footprint

Component	Baseline	Peak (100 active sandboxes)
Node.js controller (per replica)	280Mi	620Mi
Cloud SQL proxy sidecar	64Mi	120Mi
Go execd daemon (per sandbox)	12Mi	48Mi
Redis	45Mi	180Mi

12. Competitive Landscape

12.1 Market Context

The AI agent infrastructure market has attracted significant investment: E2B ($35M Series A), Daytona ($24M Series A), and multiple open-source entrants including Alibaba's OpenSandbox (March 2026). The core technical problem - fast, secure, stateful sandboxing - is well understood. Differentiation comes from latency, isolation depth, agent-native primitives, and deployment flexibility.

12.2 Technical Comparison

Capability	StateSet	E2B	Daytona	Modal	Lambda
Warm create (p50)	36ms*	~150ms†	~90ms†	~500ms†	12ms‡ / 800ms§
Isolation	5 tiers	Firecracker	Container	Container	Firecracker
REPL sessions	✓	—	—	—	—
Computer use	✓	—	—	—	—
Native MCP	✓ (14)	—	—	—	—
Checkpoints (dedup)	✓	—	—	—	—
Per-sandbox RBAC	✓	—	—	—	—
SDKs	10	2	1	1	Many
Framework integrations	8	2	0	0	0
Self-hosted (any K8s)	✓	—	✓	—	—
Helm chart + operator	✓	—	—	—	—

* Measured (Section 11.2). † Platform-published numbers. ‡ Provisioned concurrency. § Cold start.

Key differentiators: (1) Self-hosted deployment - organizations that cannot send code to third-party clouds run identical APIs on their own infrastructure; (2) agent-native primitives - REPL, computer-use, MCP, agent sessions are built into the platform; (3) 10 SDKs + 8 framework integrations - universal coverage regardless of agent framework.

Limitations vs. competitors: E2B's Firecracker-based isolation provides stronger hardware-level guarantees than gVisor (though StateSet offers Kata and Firecracker as options). Lambda's provisioned concurrency achieves lower latency for function invocations (no filesystem/state needed). Modal's Python-native interface provides a simpler developer experience for Python-only workloads.

13. Future Work

GPU sandbox profiles (NVIDIA T4/A10G/H100): GPU-aware scheduling, quotas, warm pool support
Multi-OS desktop: Windows (QEMU) and macOS (Apple Silicon) sandbox targets
Rust controller parity: RBAC, REPL, computer-use, checkpoints, billing, agent sessions
eBPF observability: kernel-level process/syscall/network monitoring without in-process instrumentation
Collaborative sandboxes: multi-user WebSocket session multiplexing

14. Conclusion

StateSet Sandbox addresses the infrastructure problem at the center of the AI agent era: running LLM-generated code safely, quickly, and at scale.

The architecture makes deliberate tradeoffs. Pre-warming Kubernetes pods is operationally complex, but it is the only path to sub-100ms creation within Kubernetes - the Redis ZPOPMIN that claims a warm pod takes 2ms; the K8s scheduling that creates a cold pod takes 3-10 seconds. Maintaining two controllers (Node.js + Rust) is extra work, but it provides a safe migration path without a flag-day rewrite. Supporting 10 SDKs and 8 framework integrations is a maintenance investment, but it ensures any agent framework has first-class platform access. Building REPL sessions, computer-use, and MCP lifecycle management into the platform - rather than leaving them to agent developers - reduces operational burden and prevents resource leaks.

Eight layers of security - from API authentication through per-sandbox RBAC, security rules, egress TCP enforcement, network policies, runtime isolation (five tiers), Linux security contexts, and Kubernetes RBAC - each provide independent protection. A compromise at any single layer does not grant access to the host, other tenants, or cloud infrastructure.

The result is a platform that covers the full lifecycle of an agent task: provision a sandbox in milliseconds, execute code with streaming output, run interactive REPL sessions, automate GUIs, persist results to durable storage, and emit structured events to any downstream system - all within a security boundary that treats isolation as the floor, not an optional upgrade.

References

[1] Y. Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models," ICLR 2023. Demonstrates that agent performance degrades with tool call latency, establishing the sub-second execution constraint for agentic loops.

[2] A. Greshake et al., "Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection," AISec 2023. Establishes the threat model for LLM-generated code executing in shared environments.

[3] Anthropic, "Model Context Protocol Specification," 2024. https://modelcontextprotocol.io. Defines the standardized tool interface used by MCP servers.

[4] A. Agache et al., "Firecracker: Lightweight Virtualization for Serverless Applications," NSDI 2020. Describes the microVM architecture used by AWS Lambda and E2B, providing hardware-level isolation with sub-200ms boot times.

[5] E. Young and J. Fang, "gVisor: Container Runtime Sandbox," Google, 2018. https://gvisor.dev/docs/architecture_guide/. Describes the guest kernel architecture used as the default runtime in StateSet Sandbox.

[6] T. Cieplak et al., "Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider," USENIX ATC 2020. Analyzes cold-start distributions across serverless platforms, providing context for warm pool design decisions.

[7] S. Hendrickson et al., "Kata Containers: An Emerging Architecture for Enabling MEC Services in Fast and Secure Way," IEEE, 2021. Describes the lightweight VM approach available as an isolation tier.

[8] E2B, "Sandbox Benchmarks," 2025. https://e2b.dev/docs/benchmarks. Published warm-start latency numbers cited in Section 11.4.

[9] Daytona, "Workspace Provisioning Architecture," 2025. https://www.daytona.io/blog/architecture. Published provisioning latency targets.

Appendix A: API Surface

Core Operations

POST   /sandbox/create                   Create sandbox
GET    /sandbox/:id                      Get details
POST   /sandbox/:id/execute             Execute command
POST   /sandbox/:id/extend              Extend TTL
DELETE /sandbox/:id                      Delete sandbox
GET    /sandboxes                        List (paginated)

File Operations (10 endpoints)

POST   /sandbox/:id/files               Write files
GET    /sandbox/:id/files?path=          Read file (base64)
GET    /sandbox/:id/files/list           List directory
GET    /sandbox/:id/files/download       Download (binary)
POST   /sandbox/:id/files/glob          Find by pattern
POST   /sandbox/:id/files/mkdir          Create directory
DELETE /sandbox/:id/files                Delete file/dir
POST   /sandbox/:id/files/move           Move/rename
POST   /sandbox/:id/files/copy           Copy
GET    /sandbox/:id/files/watch          Watch changes (SSE)

REPL Sessions

POST   /sandbox/:id/repl/sessions                     Create
GET    /sandbox/:id/repl/sessions                     List
GET    /sandbox/:id/repl/sessions/:sid                Get
POST   /sandbox/:id/repl/sessions/:sid/execute        Execute code
POST   /sandbox/:id/repl/sessions/:sid/interrupt      Interrupt
DELETE /sandbox/:id/repl/sessions/:sid                Destroy

Computer Use

GET    /sandbox/:id/screenshot           Capture
POST   /sandbox/:id/screenshot           Capture with options
POST   /sandbox/:id/mouse                Mouse action
POST   /sandbox/:id/keyboard             Keyboard action
GET    /sandbox/:id/screen               Screen dimensions

Agent Sessions

POST   /sandbox/:id/agent-sessions                    Start
GET    /sandbox/:id/agent-sessions/:sid               Get
POST   /sandbox/:id/agent-sessions/:sid/pause         Pause
POST   /sandbox/:id/agent-sessions/:sid/resume        Resume
POST   /sandbox/:id/agent-sessions/:sid/stop          Stop
POST   /sandbox/:id/agent-sessions/:sid/cancel        Cancel
DELETE /sandbox/:id/agent-sessions/:sid               Delete
POST   /sandbox/:id/agent-sessions/:sid/reattach      Reattach
POST   /sandbox/:id/agent-sessions/:sid/tools         Register tool
GET    /sandbox/:id/agent-sessions/:sid/tools         List tools
GET    /sandbox/:id/agent-sessions/:sid/files         List files

Additional Endpoints

Checkpoints, artifacts (multipart upload), MCP lifecycle, templates, auth (JWT/WorkOS), API keys, secrets, webhooks, SSE events, egress policies, queue management, billing, GitHub/GitLab integration, VNC proxy, tunnels, metrics, health/ready. See controller/src/routes/ (46 files) for complete definitions.

Appendix B: SDK Ecosystem

Language SDKs (10)

Language	Package	Status
TypeScript	`@stateset/sandbox-sdk`	GA
Python	`stateset-sandbox`	GA
Go	`github.com/stateset/sdk-go`	GA
Java	`com.stateset:sandbox`	GA
Kotlin	`com.stateset:sandbox-kt`	GA
.NET/C#	`StateSet.Sandbox` (NuGet)	GA
PHP	`stateset/sandbox` (Composer)	GA
Ruby	`stateset-sandbox` (RubyGems)	GA
Rust	`stateset-sandbox` (crates.io)	GA
Swift	`StateSetSandbox` (SPM)	GA

AI Framework Integrations (8)

Framework	Package	Interface
LangChain	`stateset-langchain`	`SandboxToolkit`
LangGraph	`stateset-langgraph`	`SandboxNode`, `SandboxCheckpointer`
OpenAI	`stateset-openai`	`SandboxToolkit.tool_definitions()`
Vercel AI SDK	`@stateset/ai-sdk-sandbox`	`createSandboxTools()`
CrewAI	`stateset-crewai`	`SandboxToolkit`
Google ADK	`stateset-google-adk`	`SandboxToolkit`
Claude Agent SDK	`stateset-claude-agent`	`SandboxEnvironment` (5 tools)
Microsoft AutoGen	`stateset-autogen`	`SandboxCodeExecutor`

CLI (24 commands)

sandbox, execute, repl, session, checkpoint, snapshot, artifact, secret, api-key, webhook, tunnel, template, deploy, computer-use, billing, audit, egress, pool, mcp, benchmark, grade, doctor, platform-status, config.

Appendix C: Deployment Architecture

Kubernetes Layout

Namespace: stateset-sandbox

├── Controller (3 replicas, HPA 3-10)
│   ├── controller container: 250m-1000m CPU, 512Mi-1Gi
│   ├── cloud-sql-proxy sidecar: 50m-200m CPU, 64Mi-256Mi
│   ├── Topology spread: maxSkew=1 on hostname AND zone
│   └── PDB: minAvailable=2
├── Sandbox Pods (dynamic, 0-hundreds)
│   ├── RuntimeClass: gvisor | kata | firecracker | wasm
│   └── Go execd on :9999 (exec) + :9998 (metrics)
├── Redis (warm pool, rate limiting, sessions)
├── PostgreSQL (via Cloud SQL proxy)
├── Egress Proxy (2 replicas, fail-closed)
└── NetworkPolicies (controller-ingress, sandbox-egress, egress-tcp, pod-isolation)

Cloud Cost Estimates

Baselined for ~200 concurrent active sandboxes (p99), 40-pod warm pool, 3 controller replicas:

Provider	~200 concurrent	~500 concurrent	~1,000 concurrent
AWS (EKS)	$1,300-$1,900/mo	$2,500-$3,200/mo	$4,000-$5,500/mo
GCP (GKE)	$1,150-$1,500/mo	$2,200-$2,800/mo	$3,500-$4,800/mo
Azure (AKS)	$1,600-$2,200/mo	$3,000-$3,800/mo	$4,800-$6,500/mo

Sandbox node costs scale ~$3-5/month per additional concurrent slot. Controller and infrastructure costs are largely fixed.

Helm Chart

Single-command deployment via k8s/helm/ with values.schema.json validation, Prometheus ServiceMonitor, PodDisruptionBudget, CRD installation, webhook configuration. Kubernetes operator (k8s/operator/) manages StateSandbox and WarmPool custom resources.

Database Scaling

usage_events: Partitioned by month (PostgreSQL declarative partitioning). Partitions >90 days exported to cloud storage as Parquet, then dropped.
execution_records: Purged after IDEMPOTENCY_TTL (default 24h).
egress_audit_events: Retained for EGRESS_AUDIT_RETENTION_DAYS (default 30).
checkpoint_chunks: Ref-counted; GC deletes orphaned chunks beyond stale threshold.
Read replicas: Recommended for >10,000 concurrent sandboxes. DATABASE_READ_REPLICA_URL routes read-only queries.

Appendix D: Configuration Reference

Required Variables

Variable	Description
`JWT_SECRET`	HMAC signing key (min 32 chars)
`SECRET_ENCRYPTION_KEY`	AES-256-GCM key derivation seed
`CORS_ORIGIN`	Allowed origins (required in production)

Key Optional Variables

Variable	Default	Description
`DATABASE_URL`	-	PostgreSQL connection string
`REDIS_URL`	-	Redis connection string
`WARM_POOL_ENABLED`	`false`	Enable warm pool
`WARM_POOL_PROFILES`	`[]`	JSON array of profile configs
`WARM_POOL_FAST_CLAIM`	`false`	Skip K8s verify on claim
`WARM_POOL_DEFERRED_CONFIGURE`	`false`	Defer K8s patch to first use
`EXEC_AGENT_ENABLED`	`false`	Use TCP exec backend
`CHECKPOINT_V2_READ_MODE`	`prefer`	`fallback` \| `prefer` \| `strict`
`EGRESS_POLICIES_ENABLED`	`false`	Enable egress allowlists
`QUEUE_MODE_ENABLED`	`false`	Enable queue backpressure
`STORAGE_PROVIDER`	`local`	`gcs` \| `s3` \| `azure` \| `local`

Full configuration reference: docs/CONFIGURATION.md.

Appendix E: Database Schema

Table	Purpose
`organizations`	Accounts, plan, trial_credits_remaining
`users`	Members, roles
`api_keys`	SHA-256 hashed tokens, scopes
`checkpoints` / `checkpoint_chunks` / `checkpoint_manifests`	State snapshots (v1 + v2)
`artifacts` / `artifact_uploads`	File storage + multipart state
`usage_events` / `usage_aggregates`	Per-operation metering
`billing_records` / `vcpu_allocations`	Invoice items, CPU tracking
`organization_secrets` / `secret_access_log`	Encrypted secrets + audit
`execution_records` / `idempotency_keys`	Execution tracking, dedup
`egress_audit_events`	Network egress log
`team_invitations`	Team member invitations

26 migrations, 27 repositories. Migrations run at startup with advisory lock.

StateSet Sandbox is open infrastructure for the age of autonomous agents.

Enjoyed this article?

Get more insights on autonomous commerce, AI agents, and margin intelligence delivered to your inbox.

StateSet Sandbox: A Kubernetes-Native Execution Platform for AI Agents

StateSet Sandbox: A Kubernetes-Native Execution Platform for AI Agents

Abstract

Table of Contents

1. Introduction

1.1 The Problem

1.2 The StateSet Sandbox Approach

2. Design Principles

3. System Architecture

3.1 Component Overview

3.2 Data Flow

3.3 Dual-Controller Strategy

4. Sandbox Lifecycle

4.1 Creation Pipeline

4.2 Expiration, Recycling, and Extension

5. Warm Pool Architecture

5.1 Data Structure

5.2 Claim Algorithm

5.3 Leader Election and Replenishment

5.4 Profile Configuration

6. Execution Engine

6.1 Go Execd Daemon

6.2 Binary Protocol

6.3 Streaming

7. Security Model

7.1 Threat Model

7.2 Defense-in-Depth Stack

7.3 Authentication

7.4 Pod-Level Isolation

7.5 Network Security

7.6 Per-Sandbox RBAC

7.7 Encryption

8. Agent Primitives

8.1 REPL Sessions

8.2 Computer Use

8.3 MCP Integration

8.4 Agent Sessions

8.5 Multi-Tenancy and Billing

9. State Management

9.1 Checkpoint v2: Content-Addressable Storage

9.2 Artifacts

10. Failure Analysis and Recovery

10.1 Failure Modes

10.2 Consistency Guarantees

10.3 Recovery Time Objectives

10.4 Graceful Degradation Hierarchy

11. Empirical Performance Evaluation

11.1 Methodology

11.2 Sandbox Provisioning Latency

11.3 Command Execution Overhead

11.4 Comparative Provisioning Latency

11.5 Throughput Under Load

11.6 Memory Footprint

12. Competitive Landscape

12.1 Market Context

12.2 Technical Comparison

13. Future Work

14. Conclusion

References

Appendix A: API Surface

Core Operations

File Operations (10 endpoints)

REPL Sessions

Computer Use

Agent Sessions

Additional Endpoints

Appendix B: SDK Ecosystem

Language SDKs (10)

AI Framework Integrations (8)

CLI (24 commands)

Appendix C: Deployment Architecture

Kubernetes Layout

Cloud Cost Estimates

Helm Chart

Database Scaling

Appendix D: Configuration Reference

Required Variables

Key Optional Variables

Appendix E: Database Schema

Enjoyed this article?