STACX — STandAloneCompleX
A Modular Infrastructure for End-to-End
Agentic Reinforcement Learning.
Modular Orchestration
Declarative Recipes
Pluggable Model Specializations
Agent-as-a-Service
A Modular Infrastructure for End-to-End
Agentic Reinforcement Learning.
Standard RLHF stacks assume a single model, a reward signal, and a relatively closed training loop. Agentic RL breaks this assumption: policies call tools, run inside scaffolds, and interact with long-horizon environments that must be executed, scheduled, and scaled.
The core shift: Agentic RL turns post-training from single-model optimization into two infrastructure requirements: unifying heterogeneous models and agent scaffolds, and scaling agent-environment interaction across distributed sandboxes.
Agentic RL is no longer dominated by model compute alone. STACX decouples training, rollout, and sandboxed environment execution into independently schedulable services, allowing each component to scale, evolve, and be optimized around its own bottleneck.
Keep algorithms lightweight; move system complexity into stable service boundaries: training, rollout, inference serving, sandbox execution, and data buffering.
STACX system loop. Training, rollout, inference-time serving, and environment execution are separated into services connected by shared data buffering and weight synchronization.
Orchestrates inference, agent scaffolds, and environment execution to generate training-ready trajectories.
Consumes trajectory batches and updates policy weights through recipe-defined objectives.
Runs long-lived sandbox environments behind a narrow, scalable interface.
STACX decomposes training, rollout, scheduling, and environment execution into stable, independently extensible services.
Decouples training, rollout, model serving, data buffering, and environment execution into independently scalable subsystems.
Supports post-training for LLMs across SFT, RL, reward modeling, and evaluation workflows.
Plugs in agent scaffolds, tools, and multi-turn task logic through a standard rollout interface.
Scales long-running, containerized environments across multi-node clusters for parallel agent-environment interaction
We now zoom into the rollout side of the system loop: the Rollout Engine receives samples, schedules evaluator instances, and runs multi-turn agent–environment interactions to produce training-ready trajectories.
Service loop across trainer, rollout, inference, environment, and data buffer
Task scheduling and evaluator dispatch
One multi-turn trajectory execution
This expands the Customized Rollout Logic box from the system overview. The RolloutManager pulls samples from DataSource, allocates resources through TaskScheduler, and dispatches task-specific Evaluators to execute trajectories.
Task environments are normalized into a unified Harbor-style format, with shared metadata, prompts, tools, stopping conditions, and reward definitions, so new tasks can be added with minimal glue code.
Zooming in: Each evaluator dispatched by the RolloutManager runs the multi-turn loop below.
Each Evaluator runs an async-native multi-turn loop, enabling hundreds of trajectories to execute concurrently.
This expands a single Evaluator launched by the RolloutManager. Each Evaluator runs a multi-turn episode by coordinating agent actions, environment steps, reward computation, resource management, and trajectory construction.
Resources are acquired before each env.step() and released immediately after, including on exceptions.
Only model-generated tokens are trained on; environment observations are masked out.
The scheduler groups pending tasks by type and dispatches them to evaluators. Scheduling policies are pluggable: FIFO, priority, fair-share, or task-aware strategies can be swapped in without changing evaluator logic.
Processes batches in arrival order. Simple and predictable for homogeneous workloads.
Interleaves task types across batches to balance multi-task training and reduce catastrophic forgetting.
Tasks are grouped into batches by task_type, agents are managed per task, then the active strategy (FIFO or Task-Aware) sets execution order before dispatch to the matching evaluators.
Users specify a global resource budget, and STACX allocates per-task resources before launching evaluators or sandboxes. Resources are acquired atomically and released when each episode finishes.
Evaluators draw from a global budget to form local pools, while each task acquires the resources it needs at runtime.
Before scheduling, STACX verifies that the pool can satisfy each evaluator’s budget, ensuring all-or-nothing resource acquisition.
STACX plugs external agent scaffolds into the RL loop as black-box services. A lightweight proxy captures actions, tool calls, observations, rewards, and traces from systems such as Terminus2 or OpenHands, then converts them into training-ready rollout trajectories.
Connect existing agent frameworks without modifying their internals.
Normalize heterogeneous agent executions into trajectories that support evaluation, reward computation, and policy training.
The same rollout interface closes the RL loop in two complementary ways: Batch RL collects high-throughput trajectories from parallel simulated environments, while On-Call RL learns continuously from a deployed agent's real-world interactions.
Offline & high-throughput. Agents run across simulated environments in parallel; trajectories are buffered and consumed by the trainer for reproducible large-scale policy updates.
Online & continuous. A deployed agent produces interaction logs in live scenarios; a daemon monitors those logs and triggers training when new data is ready, closing the loop with real-world feedback.
A modular optimization layer that consumes rollout batches, applies recipe-defined objectives on distributed GPU workers, and publishes the updated policy weights. New algorithms, model specializations, and training backends drop in cleanly — without touching rollout or environment execution.
A single training path covers GRPO, PPO, DPO, SFT, and online reward modeling — so there is no need to maintain a separate trainer implementation for each method.
StacxTrainer separates shared infrastructure from role-specific algorithm logic: the backend handles the mechanics of distributed training, while role workers and model specializations carry the method-specific behavior.
Every step ingests a batch of rollout data, applies the recipe's optimization objective, and dispatches trainable roles to distributed GPU workers — coordinating shared weights safely throughout.
Each semantic model role maps to a RoleWorker group on a Ray pool, cleanly separating algorithm-level responsibilities from physical resource placement.
Backends own system concerns — sharding, checkpointing, and weight synchronization — while model specializations define the objectives and role-specific behavior, reusable across model families.
The specialization layer decouples a model's identity from its training role, unlocking flexible deployment across algorithms.
Any model maps to any role through its specialization — covering policy-only, actor-critic, reference-model, reward co-training, and heterogeneous experiments alike.
A single checkpoint can serve as actor, critic, and reference at once. DualRole shares those weights across roles for PPO-colocated, GRPO, and online reward-modeling setups.
Heterogeneous models (for example, DeepSeek-V3 alongside Qwen-72B) can fill the same semantic role within one training loop, enabling ensembles and head-to-head model comparison.
Algorithms are expressed as configuration rather than hard-coded trainer paths. Each recipe declares its model roles, specializations, training behavior, weight synchronization, and inter-role data dependencies.
# Algorithm = data @recipe("grpo") def grpo_config(): return TrainingConfig( name="grpo", roles=(ModelRole(name="policy", specialization_cls=ActorSpecialization),), syncs=(), # no inter-role syncs ) # Adding a new algorithm? @recipe("my_algo") def my_algo_config(): return TrainingConfig( name="my_algo", roles=(policy_role, value_role), syncs=(DataSync(source="value", target="policy", keys=("values",)),), )
All model-specific logic sits behind three methods the backend calls; everything else remains shared infrastructure.
Every algorithm lives as a specialization.
The core of the algorithm — returns (loss, metrics) and computes advantages when the objective requires them.
Transforms raw rollout data into model-ready micro-batches.
Pulls the right tensor out of the model's forward pass.
Write a recipe and, where the objective demands it, a specialization. The trainer loop, workers, sync infrastructure, and weight export all stay untouched.
The execution layer for agent actions. Built on ROCK, it runs isolated sandbox sessions that manage container lifecycle, tool execution, and resource isolation, while exposing only a narrow HTTP gateway to the rollout engine.
Scales sandbox execution independently from training and rollout on separate CPU/GPU resource pools.
Routes agent actions to the appropriate sandbox and returns normalized observations, logs, exit status, and artifacts while hiding sandbox lifecycle details.
Assigns each episode to its own sandbox session and releases resources predictably when the episode completes or fails.
Represents all task environments through a common Harbor-style interface for metadata, prompts, tools, stopping conditions, and reward definitions.
The rollout engine interacts with environments only through a narrow gateway; The environment engine manages sandbox sessions, routing, execution, and cleanup behind the interface.
We validate STACX across three agentic RL settings: end-to-end RL improves tool-integrated math reasoning, multiple post-training strategies improve software-engineering agents over their base models, and rollout-only evaluation shows that STACX reproduces reference infrastructure behavior across diverse agentic task suites.
# Tool-integrated math RL: GRPO on DAPO-Math-17K with the ReTool scaffold git clone https://github.com/STACX/stacx.git && cd stacx # 1. Start a slime container with repo + data + model mounts docker run --gpus all -d --name stacx_retool --network host --shm-size 16g \ --ulimit memlock=-1 --ulimit stack=67108864 --ulimit nofile=524288:524288 \ -v $(pwd)/rl_engine:/root/rl_engine \ -v $(pwd)/external/data:/root/data \ -v $(pwd)/external/models:/root/models \ -v $(pwd)/external/ckpts:/root/experiments \ lichangh20/slime-rl:stable sleep infinity # 2. Run GRPO training inside the container (rl.sh handles Ray setup) docker exec -it stacx_retool bash -c \ 'cd /root/rl_engine/examples/retool && WANDB_API_KEY=<key> bash scripts/rl.sh'
We train Qwen3-4B-Instruct with GRPO on DAPO-Math-17K, using ReTool as the tool-integrated rollout scaffold, and evaluate checkpoints across math reasoning benchmarks. Accuracy improves consistently over training on MinervaMath, MATH, GSM8K, and AIME 2024, demonstrating that STACX supports the full rollout-to-training RL loop.
We post-train software-engineering agents end-to-end with STACX on SWE-Gym, generating and evaluating trajectories through the OpenHands scaffold. With a Qwen3-Coder-30B teacher, STACX supports a range of post-training strategies: SFT, On-Policy Distillation, and DAgger- / AggreVaTe-style mixture-policy imitation, that improve Qwen3-4B / 8B students on both SWE-Gym Holdout and SWE-Bench Verified.
# SWE-agent post-training: Qwen3-4B/8B student × Qwen3-Coder-30B teacher git clone https://github.com/STACX/stacx.git && cd stacx # Stage models + data under external/, then start the ROCK sandbox admin # (one-time setup — see scripts/train/swe/QUICKSTART_OPD.md) rock admin start & # Pick a post-training strategy (append _8b for the 8B student) export WANDB_KEY=<your-wandb-key> bash scripts/train/swe/sft/sft_4b.sh # SFT bash scripts/train/swe/opd/opd_4b.sh # On-Policy Distillation bash scripts/train/swe/online_dagger/dagger_4b.sh # DAgger bash scripts/train/swe/online_dagger/aggrevate_4b.sh # AggreVaTe
Training-data scaling under matched effective-sample budgets (4B student), comparing four post-training strategies supported by STACX on SWE-Gym Holdout and SWE-Bench Verified-100.
| Method | Agent Scaffold | Train Data | SWE-Gym Holdout | SWE-Bench Verified |
|---|---|---|---|---|
| Qwen3-4B | OpenHands | — | 5.0% | 11.2% |
| + SFT | OpenHands | SWE-Gym | 15.0% | 22.9% |
| + OPD | OpenHands | SWE-Gym | 16.0% | 23.4% |
| + DAgger | OpenHands | SWE-Gym | 17.0% | 27.3% |
| + AggreVaTe | OpenHands | SWE-Gym | 16.0% | 24.5% |
| Qwen3-8B | OpenHands | — | 2.0% | 7.7% |
| + SFT | OpenHands | SWE-Gym | 12.0% | 23.4% |
| + OPD | OpenHands | SWE-Gym | 16.0% | 26.2% |
| + DAgger | OpenHands | SWE-Gym | 19.0% | 29.8% |
| + AggreVaTe | OpenHands | SWE-Gym | 17.0% | 27.3% |
Task-resolution rate, where higher is better. Under the same OpenHands scaffold, STACX supports multiple post-training strategies that improve Qwen3-4B and Qwen3-8B agents over their base initialization on both SWE-Gym Holdout and SWE-Bench Verified.
We run rollout-only evaluation under identical agent scaffolds and benchmark settings, comparing STACX against Harbor across five agentic task suites. STACX closely matches the reference infrastructure across OpenHands and Terminus2, validating it as a drop-in execution layer for agentic RL workloads.
# Rollout-only eval: BENCH ∈ swebench | terminal_bench | seta | kernel_bench | algotune git clone https://github.com/STACX/stacx.git && cd stacx # 1. Download the Harbor task tree, convert → tasks.jsonl, build images python rl_engine/examples/shared/download_tasks.py <bench>@<ver> \ -o external/harbor-datasets/<bench> python rl_engine/examples/shared/prepare_jsonl.py \ --tasks-dir external/harbor-datasets/<bench> \ --output rl_engine/examples/<bench>/config/tasks.jsonl --registry <bench> bash rl_engine/examples/shared/build_images.sh \ rl_engine/examples/<bench>/config/tasks.jsonl # 2. Serve the model, then evaluate (SCAFFOLD = openhands | terminus2) MODEL=<hf-model> PORT=9001 bash rl_engine/examples/shared/serve_sglang.sh BENCH=<bench> SCAFFOLD=openhands SGLANG_ROUTER_PORT=9001 \ bash rl_engine/examples/shared/docker_run_eval.sh
| System + Scaffold | SWE-bench Verified | Terminal-Bench 2.0 | SETA | KernelBench | AlgoTune |
|---|---|---|---|---|---|
| STACX + OpenHands | 47.00% | 24.34% | 47.80% | 19.00% | 1.14 |
| Harbor + OpenHands | 47.00% | 24.71% | 45.67% | 19.00% | 1.19 |
| STACX + Terminus2 | 57.33% | 28.46% | 51.67% | 36.00% | 1.28 |
| Harbor + Terminus2 | 57.67% | 31.46% | 51.83% | 36.00% | 1.14 |
SWE-bench Verified, Terminal-Bench 2.0, SETA, and KernelBench report average success rate; AlgoTune reports harmonic-mean speedup reward.
More agentic results coming soon. We are actively expanding evaluations across additional scaffolds, task suites, and algorithms. Numbers above reflect the current public snapshot.
@misc{stacx2026,
author = {STACX Contributors},
title = {STACX: A Modular Infrastructure for End-to-End Agentic Reinforcement Learning},
year = {2026},
howpublished = {\url{https://github.com/STACX/stacx}},
urldate = {2026-07-15}
}