Release notes from prime-rl
Articles about Release notes from prime-rl
V0.9.1.dev213: refactor(trainer): add iter_moe_blocks for the MoE layer lookup (#3843)
Summary Add iter_moe_blocks(model) in trainer/model.py. It yields the MoE MLP of each decoder layer. It replaces the five copies of the decoder-layer MoE lookup in freeze_moe_router, apply_fp32_moe_router, get_full_offload_dtype_policy, apply_force_balanced_routing and get_load_balance_stats. What changed New iter_moe_blocks next to the helpers that use it. The helpers stay in model.py, so no imports change.
V0.9.1.dev212: feat(trainer): freeze MoE router by default in RL (#3797)
Make trainer.model.freeze_moe_router default to on for RL. What changes model.freeze_moe_router is now bool | "auto", default "auto". It resolves to True on the RL TrainerConfig and to False on SFTConfig, following the per-trainer pattern of moe_router_dtype. An explicit true/false still wins. freeze_moe_router() no longer raises on non-MoE models; it does nothing, so dense RL runs keep working with the new default. docs/training.md knobs table lists the field.
Release v0.9.1.dev211 · PrimeIntellect-ai/prime-rl
fix(trainer): run FA2 and FA3 through custom ops so fullgraph compile
Release v0.9.1.dev210 · PrimeIntellect-ai/prime-rl
fix(inference): let the LoRA adapter take the base model name on vLLM
Release v0.9.1.dev209 · PrimeIntellect-ai/prime-rl
fix(logger): disable loguru diagnose so crashes with large tensors ex
Release v0.9.1.dev208 · PrimeIntellect-ai/prime-rl
fix(eval): resolve endpoints and credentials through Verifiers client
Release v0.9.1.dev207 · PrimeIntellect-ai/prime-rl
chore(models): sqrtsoftplus in the shared router, drop the DeepseekV4
V0.9.1.dev206: fix(dashboard): keep replay Home/End mode when the output fits (#3878)
The dashboard smoke check at the end of the alphabet_sort integration test flakes at tests/integration/dashboard_smoke.py:164. It asserts that #replay-top is active after Home. It failed on #3767 and #3774, which don't touch the dashboard. Root cause. The smoke test presses t and then Home. When thinking is hidden, a trace that overflowed the replay pane gets shorter and fits. The browser then clamps scrollTop and queues a scroll event. That event fires after Home. At that point the pane fits, so the scroll
V0.9.1.dev205: chore: bump vllm to 0.31.0 (#3888)
Bump vLLM 0.30.0 → 0.31.0 (exempt flashinfer-python 0.7.0.post1 from the exclude-newer cooldown). Read enable_return_routed_experts from VllmConfig.aux_output_config (moved in 0.31). Router replay now always uses the V2 runner; router replay with disaggregated P/D is rejected at config time (0.31 has no NIXL routed-expert capture). Resync the copied LoRA loader in _patch_lora_key_prefix with 0.31 (modules_to_save). Removed patches: monkey_patch_deepseek_v4_request_tools_placement: fixed upstream (vllm#51856).
Release v0.9.1.dev204 · PrimeIntellect-ai/prime-rl
chore(tests): delete permanently skipped tests and the vendored HF AF
V0.9.1.dev203: fix(ci): move all 2-GPU GPU Tests jobs to h200 k8s runners (#3835)
Moves every GPU Tests job that fits in 2 GPUs onto the prime-rl-2xh200 ARC scale set, so runs stop queueing behind the two vm runners. Stacked on #3834. Assignment prime-rl-2xh200 (9): unit-tests, reverse_text, reverse_text_sft, reverse_text_sft_lora, reverse_text_lora, reverse_text_moe, reverse_text_rl_opd, reverse_text_rl_sft, alphabet_sort, gsm8k_eval 4xa6000 (1): benchmark_regression, unchanged The vm runners are left registered and untouched - just no longer on the GPU Tests path. Sizing checks
V0.9.1.dev202: fix(docker): include FlashMLA in docker image (#3887)
Hosted image v0.9.1.dev200 fails when GLM-5 selects model.dsa_backend = "cudnn_flashmla": the forward path imports flash_mla, but Docker's explicit sync extras omit flash-mla. The dev200 build log confirms the missing extra; nvidia-cudnn-frontend is already installed. Add --extra flash-mla to the dependency-image sync and the source-override entrypoint sync. Add an amd64 package-presence check before publishing the dependency image. The existing Linux/x86_64 marker
V0.9.1.dev201: chore: remove legacy config options (#3770)
Removes legacy config options and their code paths. They have had no changes of their own in 4+ months. Removed SGD optimizer (optim.type = "sgd"): SGDConfig, its union member, and the torch.optim.SGD branch in trainer/optim. AdamW, Muon, and SignSGD stay. ckpt.skip_progress (orchestrator and trainer ckpt configs): resume now always restores progress. This affects the orchestrator ckpt loading and SFT resume. Jackmin108/ model-name special case in validate_shared_model_name. Trainer and orchestrator model names now
Release v0.9.1.dev200 · PrimeIntellect-ai/prime-rl
perf(dsa): cuDNN + FlashMLA kernels for GLM-5 sparse attention (model
Release v0.9.1.dev199 · PrimeIntellect-ai/prime-rl
feat(docker): reuse dependency images for fast Docker source builds (
Release v0.9.1.dev198 · PrimeIntellect-ai/prime-rl
refactor(algo): declare action_loss_type once on the config; drop fin
V0.9.1.dev197: refactor(orchestrator): fold packing.py into batch.py (#3842)
Moves BatchPacker (35 lines) from orchestrator/packing.py into orchestrator/batch.py and deletes packing.py. The class only wraps build_bin_cost and prepare_batch, which already live in batch.py. Item 6 of the orchestrator review. Behavior change: none (pure move; the import in orchestrator.py is updated). Checks pytest tests/unit/orchestrator tests/unit/test_configs* -m "not gpu": 291 passed ruff check, ruff format --check: clean python -c "import prime_rl.orchestrator.orchestrator" OK
V0.9.1.dev196: chore(deps): drop duplicate and unused pyproject dependencies (#3856)
Removes duplicate and unused entries from pyproject.toml (rev-infra P9). The set of installed packages does not change for any extra combination (checked below). Removed Reason second orjson>=3.10 in dependencies Duplicate of orjson>=3.11.0 a few lines above. pandas>=2.0 Nothing imports it: no import pandas / pd. in src/, tools/, scripts/, tests/, examples/, skills/, packages/. It was added for the nano bridge (7089ba1). It stays installed because datasets depends on it. pyarrow>=21.0.0 No direct use anywhere in those directories.
Release v0.9.1.dev195 · PrimeIntellect-ai/prime-rl
docs(scaling): match ep=“auto” to the code; NIXL wheel is nixl_cu13 (
Release v0.9.1.dev194 · PrimeIntellect-ai/prime-rl
chore: remove Kubernetes deployment path (Helm chart, trainer metrics
V0.9.1.dev193: chore: remove legacy dead code (#3769)
Removes legacy leftovers that were last touched 2025-11 to 2026-02 or were orphaned by features that are already gone. Behavior does not change. orchestrator/patches.py: deletes the OpenAI chat-type monkeypatches, which were written for verifiers 0.1.8. The orchestrator and eval processes no longer parse ChatCompletion or message objects in-process. Rollouts run in separate env-server processes, which never applied the patch, and prefill_logprobs reads the raw httpx.Response.
Release v0.9.1.dev192 · PrimeIntellect-ai/prime-rl
fix(dashboard): use relative static/api paths for path-prefixed rever
V0.9.1.dev191: chore(orchestrator): remove token_batch_size batching (#3775)
Removes the orchestrator token_batch_size option (token-budget batch cutting). Batches are now always cut by sample count (batch_size). Why: the train sink had two parallel batch-cut paths. Sample-count batching is the one every shipped config uses, including GLM-5.3. Removing the token path drops the pending_tokens / payload_tokens accounting and the token branches in process_batch, _drop_stale, _maybe_batch and _record_zero_output. Changes OrchestratorConfig.batch_size is now a plain int = 128 field.
V0.9.1.dev190: feat: opt-in exact prompt-mean loss aggregation for GRPO (#3798)
Adds GRPOAlgoConfig.loss_aggregation: Literal["token", "prompt"] = "token". echo gets it too because it subclasses GRPO. Why: the trainer averages the rl loss over every loss token in the batch. Prompts whose groups produce long trajectories therefore dominate the gradient. SWE groups can differ by about 10x in total tokens. "prompt" implements MiMo-V2.6's prompt-mean aggregation exactly (Eq. 1, p8): L = mean_q (1/T_q) Σ_{i,t ∈ q} ℓ_{i,t}. MiMo uses it to keep response length from growing too fast. How:
Release v0.9.1.dev189 · PrimeIntellect-ai/prime-rl
chore(trainer): remove HF model-implementation fallback (model.impl)
Release v0.9.1.dev188 · PrimeIntellect-ai/prime-rl
feat(configs): default optim.weight_decay to “auto” (0.0 for RL, 0.01
Release v0.9.1.dev187 · PrimeIntellect-ai/prime-rl
feat(trainer): MoE expert-load metrics (cv, max/mean, cold fraction)
V0.9.1.dev186: refactor(configs): define the weight-broadcast config once (#3788)
Summary The weight-broadcast config (filesystem | nccl | nixl) was declared four times: the inference WeightBroadcastConfig, the trainer *WeightBroadcastConfig, the orchestrator *WeightBroadcastConfig, and the rl.py Shared*WeightBroadcastConfig. RLConfig.auto_setup_weight_broadcast then copied each field between them by hand. Every new transport field had to be added in four places and in the copy code. Design: one discriminated union WeightBroadcastConfig (FileSystem / NCCL / NIXL variants) in configs/shared.py. The trainer,
V0.9.1.dev185: feat(orchestrator): optional length-weighted GRPO baseline (#3789)
Adds length_weighted_baseline: bool = False to GRPOAlgoConfig. When enabled, the GRPO group baseline is the token-length-weighted mean reward b = sum_i(L_i * r_i) / sum_i(L_i) instead of the plain mean. L_i is the number of trainable (mask-True, policy-sampled) tokens of rollout i, summed across all turns. Advantage is r_i - b. The baseline is applied after the length penalty, same as the plain mean. There is no fallback for a group with zero trainable tokens: then every trace has L_i = 0, so assign_advantages writes no tokens, and no
Release v0.9.1.dev184 · PrimeIntellect-ai/prime-rl
refactor(launcher): share local launch-and-monitor code between rl an
Release v0.9.1.dev183 · PrimeIntellect-ai/prime-rl
refactor: move packing to orchestrator, DSv4 eager reference to tests
Release v0.9.1.dev182 · PrimeIntellect-ai/prime-rl
perf(metrics): count attention FLOPs with the real head dim and repor
Release v0.9.1.dev181 · PrimeIntellect-ai/prime-rl
chore(trainer): remove torch full-offload backend, fold gradient offl
Release v0.9.1.dev180 · PrimeIntellect-ai/prime-rl
refactor(inference): register all vLLM patches in the general plugin
V0.9.1.dev179: feat(models): add native Qwen3.8 Flash Next (#3434)
Merged main at e094f0860. Main's expert parameters remain in the transformer block's FSDP group; prefetching retains that behavior and Qwen3.8's final hyper-connection mixer handling. Adds Qwen3.8 Flash Next text training for qwen4_exp and qwen4_exp_text: hyper-connection residual streams, Gated DeltaNet, indexed sparse attention, PLE N-gram embeddings, and sigmoid-gated MoE, with HF checkpoint conversion. Head parallelism PLE uses ordinary nn.Embedding parameters. EmbeddingParallel, an
V0.9.1.dev178: chore: remove fusions.raise_on_fail and dead ProgressTracker (#3778)
Removes two small things: trainer.model.fusions.raise_on_fail. A requested fusion that no module supports now always logs a warning and is skipped. The default fusions list (["gate_up", "qkv"]) is applied to every model, including models that lack those modules, so warning is the right single behavior instead of a flag. Dead ProgressTracker class in prime_rl/utils/logger.py, plus the _progress branch in build_log_entry that only it could reach. Nothing in the repo, deps/verifiers, or deps/prime-envs references them.
Release v0.9.1.dev177 · PrimeIntellect-ai/prime-rl
chore(configs): remove configs/basic and unreferenced debug configs (
V0.9.1.dev176: chore: remove KL penalty term from RL policy losses (#3810)
Removes the KL penalty/regularizer term from the RL policy losses. What changed IPOLossConfig.kl_tau removed (packages/prime-rl-configs/.../configs/trainer.py); IPOLoss no longer adds kl_tau * log_ratio^2 (src/prime_rl/trainer/rl/loss.py). ref_kl_loss_fn: removed the hard-coded 1e-3 * log_ratio^2 drift regularizer. The ref_kl component itself (reverse KL to the reference model as the PG signal for OPD/OPSD) is unchanged. mismatch_kl metrics are unchanged. Docs (docs/algorithms.md IPO formula + knob table, IcePop wording)
V0.9.1.dev175: fix(monitors): omit training arrays from live trace records (#3763)
Live MoE trace deltas contain binary routed_experts and sampling_mask payloads. FileMonitor.log_live() currently serializes those bytes through default=str, expanding them into escaped text and adding serialization work to live monitoring. Filter these two fields from copied node dictionaries inline before writing live records, and omit routing_repairs since the corresponding arrays are absent from those records. The original deltas and training payloads remain unchanged; message content, token IDs, calls, semantic
V0.9.1.dev174: refactor(trainer): move SignSGD into trainer/optim (#3764)
What does this PR do? Moves sign_sgd.py from prime_rl/trainer/ into prime_rl/trainer/optim/ so all optimizer implementations live in one place. Pure file move — git detects a 100% rename so history and blame are preserved. No functional changes. Updated the three import sites: trainer/optim/__init__.py trainer/optim/offload.py tests/unit/train/test_cpu_adam.py No other references to prime_rl.trainer.sign_sgd exist; configs, docs, and example TOMLs use the sign_sgd optimizer type value, which is unchanged.
Release v0.9.1.dev173 · PrimeIntellect-ai/prime-rl
feat(dashboard): add persistent trace URLs (#3759)
Release v0.9.1.dev172 · PrimeIntellect-ai/prime-rl
fix(dashboard): wrap trace viewer toolbar (#3760)
Release v0.9.1.dev171 · PrimeIntellect-ai/prime-rl
fix(dashboard): group collapsible tool results with calls (#3761)
Release v0.9.1.dev170 · PrimeIntellect-ai/prime-rl
chore: bump verifiers to 4fd5921 and prime-envs to d4fb80a (#3762)
Release v0.9.1.dev169 · PrimeIntellect-ai/prime-rl
feat(configs): auto-resolve moe_router_dtype (fp32 for RL, bf16 for S
Release v0.9.1.dev168 · PrimeIntellect-ai/prime-rl
perf(trainer): keep EP experts in the block’s FSDP unit so MoE blocks
Release v0.9.1.dev167 · PrimeIntellect-ai/prime-rl
fix(trainer): run FA4 through a custom op so compile can trace attent
Release v0.9.1.dev166 · PrimeIntellect-ai/prime-rl
fix(docs): mark Qwen3/Qwen3.5 VLM CP as supported, drop stale ’Other
Release v0.9.1.dev165 · PrimeIntellect-ai/prime-rl
chore: bump verifiers to 185b67d and prime-envs to e82de8b (#3750)
Release v0.9.1.dev164 · PrimeIntellect-ai/prime-rl
fix(sft): reshard FSDP2 params after validation before grad clipping
V0.9.1.dev163: feat(sft): map dataset columns in data.columns (#3744)
Summary add [data.columns], which names the dataset columns the SFT trainer reads. messages, prompt, and completion hold lists of OpenAI chat messages and tools holds OpenAI function-calling schemas; each defaults to today's name. SFTDataset reads the configured names per row, so no dataset is rewritten up front. The defaults are optional because a dataset carries either messages or prompt/completion; a name set in the config must exist in the dataset add [data.columns.renderer], a renderer field = dataset column
V0.9.1.dev162: feat: custom renderers from local modules and files (#3743)
Summary pin deps/renderers to d43c430, the PrimeIntellect-ai/renderers#168 merge on main, which adds CustomRendererConfig and renderers.custom. [renderer] name = "custom" with import_path = "my_module.Class" or "path/to/file.py:Class" selects a renderer that lives outside the renderers package, for SFT and for RL via [orchestrator.renderer]. The renderer's own config_class validates the remaining [renderer] keys, which sit inline like every other renderer config; the key names match type = "custom" + import_path
V0.9.1.dev161: examples: 2-node Qwen3-30B-A3B SFT config (#3728)
Adds a multi-node SFT example for Qwen/Qwen3-30B-A3B-Instruct-2507 on 2 train nodes. It uses the same layout as the GLM-5.3 SFT example (examples/advanced/glm-5.3/sft/h200/). Files examples/advanced/qwen3-30b-a3b/sft/h200/base.toml: custom MoE implementation, ep = 8, FA3, full AC with offloading, AdamW (lr 1e-5, linear schedule, 50 warmup steps), multi_node with num_train_nodes = 2 and a [slurm] block. Trains only, with no inference nodes. sft/h200/math-10k.toml: data overlay for
V0.9.1.dev160: feat(moe): add SonicMoE BF16 expert compute backend (#3535)
Adds model.moe.compute.backend = "sonicmoe" for BF16 MoE expert computation. Sonic consumes already-dispatched expert rows and fused gate/up weights; dispatchers retain routing-score application and combination. GroupedExperts owns its parameters and delegates execution through the structural ExpertCompute protocol. Concrete backends do not inherit from the protocol. set_compute() validates compatibility before assignment: Sonic requires fused gate/up weights, standard SwiGLU, and bias-free experts.
Release v0.9.1.dev159 · PrimeIntellect-ai/prime-rl
fix(disagg): disable breakable CUDA graphs on DeepEP high-throughput
Release v0.9.1.dev158 · PrimeIntellect-ai/prime-rl
feat(dashboard): show cancelled and failed subagents in the semantic
Release v0.9.1.dev157 · PrimeIntellect-ai/prime-rl
fix(dashboard): detect index rewrite on resume in index_rows (#3732)
V0.9.1.dev156: feat: dashboard global filter and train overview tiles (#3725)
Summary Overview keeps its scroll position across re-renders. Every new eval episode (and every new metric key on a training run) rebuilt #overview-body and snapped it to the top, which made analysing anything below the fold on a live eval impossible. renderMetricsBody now restores scrollTop after the rebuild. One filter shared by the overview, the traces table and the trace viewer. The filter menu (status, kind, env, outcome) is a single cloned into the three toolbars over one state.filter; a
V0.9.1.dev155: feat: unify how source groups pass defaults (#3714)
Summary Add SourceGroupConfig, the one base of every source group: [orchestrator.train] (TrainConfig), [orchestrator.eval] and the SFT [eval] block (ScheduledEvalConfig), and the top level of uv run eval (EvalConfig). It has one rule: every field that a group and its source type both declare is a default. Each source inherits the fields that the group sets, merged under its own with vf.merge_defaults: nested blocks merge key by key, a source's own values win, and a block whose type/id differs is the source's alone.
V0.9.1.dev154: fix: shard single-row gate weights along columns for Muon (#3723)
Qwen shared-expert gates can have weight shape [1, hidden_size]. Default FSDP row sharding gives some ranks empty local tensors, which is incompatible with the equal-shard layout needed by the distributed Muon update. Shard 2D single-row parameters along dimension 1 when their width is divisible by the parameter-sharding degree, obtained directly from parallel_dims.get_mesh("dp_shard_cp").size(). Other parameters retain the existing fused-parameter placement callback or default placement. FSDP still uses the HSDP mesh;
V0.9.1.dev153: fix(utils): throttle BetterStack heartbeat pings (#3715)
Problem Since #3664, eval runs beat the Better Stack heartbeat on every landed episode. Heartbeat only coalesced concurrent requests (at most one in flight), so once a ping returned, the very next landed episode fired a new one. A busy eval lands many episodes per second and blows past Better Stack's heartbeat rate limit: The trainer/sft/orchestrator beat once per batch and stay under the limit only because of step duration. Changes Heartbeat now rate-limits itself: at most one ping every
V0.9.1.dev152: feat: bump vllm-router to 0.2.2 (#3718)
Summary Bump vllm-router from 0.2.1 to 0.2.2 (amd64 and arm64 release wheels). The new router picks up PrimeIntellect-ai/router#58. The default health-check interval drops from 60s to 20s, and every health-check cycle refreshes /v1/models. Workers that are still loading at startup no longer stay under unknown and miss model-filtered traffic for minutes. Verification uv lock resolves with vllm-router v0.2.1 -> v0.2.2. Installed router reports version 0.2.2 and
Release v0.9.1.dev151 · PrimeIntellect-ai/prime-rl
fix(docs): reword bare ‘
V0.9.1.dev150: feature: prime monitor for SFT trainer (#3614)
Motivation Hosted SFT runs (the dedicated FFT path) currently report nothing to the platform: SFTConfig.monitors was the base MonitorsConfig (wandb/file only), so a hosted SFT trainer has no platform run — no metrics, no billing surface, no "view on platform" link. RL runs get all of this through PrimeTrainMonitor (wired in the RL orchestrator); this PR gives the SFT trainer the same platform integration. What it adds Config: SFTConfig.monitors is now TrainMonitorsConfig, so
V0.9.1.dev149: fix(docker): pin the NIXL wheel build backend (#3720)
Problem Image builds on main have failed since #3667 merged. Every build_image.yaml run since 2026-09-28 dies in the INCLUDE_NIXL_FROM_SOURCE step, so no amd64/arm64 image is published. The failing command is the step's own linkage check — the wheel builds and installs fine: Cause The NIXL wheel build is not reproducible: NIXL's [build-system] requires is an unpinned list, so pip's build isolation resolves the newest backends at build time. meson-python 0.22.0 (uploaded 2026-09-26 08:33Z — after the
Release v0.9.1.dev148 · PrimeIntellect-ai/prime-rl
Delegate prompt metadata to vLLM 0.30 (#3710)
V0.9.1.dev147: feat: depend on the prime CLI instead of prime-sandboxes (#3717)
Summary Replace prime-sandboxes>=0.3.2 with prime>=0.8.0 in the core dependencies. prime 0.8.0 drops its verifiers==0.2.0 pin (prime#940). prime-sandboxes now comes in transitively and stays locked at 0.3.2. Remove the separate uv pip install 'prime>=0.7' step and its verifiers override from Dockerfile.cuda. uv sync now installs the CLI into /app/.venv at the locked version. Remove the same replay step from the PRIME_RL_REF source overlay in scripts/docker-entrypoint.sh. Remove uv tool install prime from scripts/install.sh. uv sync
V0.9.1.dev146: feat: bump vllm to 0.30.0 (#3716)
Summary Bump vLLM to v0.30.0. Bump the nvidia-cutlass-dsl override to 4.7.1. vLLM 0.30.0 and its quack-kernels 0.6.5 require it. Set --enable-scale-out on the inference server. vLLM 0.30.0 no longer registers /inference/v1/generate on vllm serve without it, and the orchestrator generates through that endpoint. Set inference.vllm.enable_scale_out = false to opt out. Remove two DeepSeek V4 patches that upstream now covers: monkey_patch_deepseek_v4_per_layer_rope: fixed by
V0.9.1.dev145: feat: select taskset tasks per train and eval source (#3703)
Summary Give every train and eval source a select block (EnvConfig.select, a vf.SelectConfig): include/exclude by task idx, ids, keys or names, then shuffle, skip and limit. The orchestrator loads each source's tasks with vf.load_taskset(...).select(source.select). prime-rl no longer shuffles or slices tasksets itself, so it picks the same tasks as vf-eval for the same selection.
V0.9.1.dev144: Add new chunked loss impl based on torchtitan (#3673)
Port of torchtitan based chunked loss, which greatly imporves perf on hopper for smaller models: Model Old chunked head New chunked head Gain Qwen3-30B-A3B, 8×H200, no AC, chunk 1024 6.71s 6.20s −7.6% step time (+8.2% throughput) Qwen3-235B-A22B, 4×8 H200, full AC, chunk 2048 14.09s 14.00s −0.6% step time Note Medium Risk Changes core LM-head autograd and SFT loss computation; RL/temperature paths are mostly untouched but shared head code affects all training modes using injected heads. Overview
V0.9.1.dev143: feat(rl): add PPO and CISPO loss options (#3697)
Motivation The trainer exposes IPO and IcePop but has no built-in PPO or CISPO loss for controlled math-RL comparisons. PPO clips the token importance-ratio surrogate; CISPO clips a detached importance weight multiplying the current log probability, so an out-of-band ratio still has a policy gradient. Both need to operate on the trainer's existing eligible-token masks and optional per-token weights. Design Add PPOLossConfig / PPOLoss with a sign-aware clipped surrogate (ratio_low=.8, ratio_high=1.2 by default). A separate
Release v0.9.1.dev142 · PrimeIntellect-ai/prime-rl
Train from inline multimodal rollouts (#3320)
V0.9.1.dev141: fix(rl): stabilize IPO, IcePop, and ref-KL loss masking (#3696)
Motivation Packed RL batches can contain masked tokens with invalid or extreme log probabilities. IPO, IcePop, and ref-KL previously formed importance ratios before removing those tokens, so a nominally excluded token could still produce 0 * inf or 0 * NaN. Accepted rare actions could also overflow exp(log p_trainer - log p_inference), including in mismatch-KL diagnostics. Design Select eligible tokens before ratio and KL arithmetic; IcePop and ref-KL also select accepted tokens before exponentiating their policy terms.
V0.9.1.dev140: fix: bump verifiers and prime-envs for timeouts (#3709)
Summary Bump Verifiers to c35ebd32. This includes the merged timeout fix: an unset agent rollout timeout no longer falls back to four hours. Bump prime-envs to 565e8a74. Its Lean helper now works with Verifiers' taskset API. Breaking An unset agent rollout timeout now inherits the task timeout, or has no separate limit. Set a positive env..timeout.rollout to retain a stage limit. env..timeout.rollout = 0 is rejected. Remove the field to request no separate stage limit.
Release v0.9.1.dev139 · PrimeIntellect-ai/prime-rl
[ENG-6355] feat(docker): in-place PRIME_RL_REF source overlay for cha
V0.9.1.dev138: Require prime-sandboxes>=0.3.2 (#3682)
Raises the pyproject floor to prime-sandboxes>=0.3.2 and refreshes the lock accordingly (resolves 0.3.2, wheel/sdist hashes cross-checked against PyPI). Why The hosted-eval stress runner bootstraps this repo with uv sync --no-dev --frozen, so the 429-retry fixes in prime-sandboxes 0.3.2 (status-aware backoff on platform API and gateway retries) only reach eval rollouts through this lock. 0.3.0 satisfied the old >=0.3.0 floor indefinitely - lock-check enforces consistency, not freshness - so the
Release v0.9.1.dev137 · PrimeIntellect-ai/prime-rl
fix(docker): build NIXL from source by default; isolate the prime CLI
Release v0.9.1.dev136 · PrimeIntellect-ai/prime-rl
fix(docker): keep the editable verifiers when installing the prime CL
V0.9.1.dev135: feat(eval): allow any config override on resume (#3690)
Summary uv run eval --resume used to compare the resolved config against the interrupted attempt's eval.json. It refused any change outside a short allowlist, which left out the solver model, everything under source.*.env (including where each judge is served) and tasks_per_minute. So a paused eval couldn't move to a new endpoint serving the same weights under another name, such as a self-hosted router becoming a hosted API. The only way out was --clean. This PR removes that validation (RESUMABLE, config_diff,
V0.9.1.dev134: feat: treat timed-out rollouts as their own outcome (#3702)
Summary Companion to PrimeIntellect-ai/verifiers#2693, which stops a rollout whose stage deadline expires as _timeout with Trace.is_timeout set (scored, not errored). Training keeps main’s
V0.9.1.dev133: feat: shared env block for train and eval sources (#3695)
Summary Group env block: [orchestrator.train], [orchestrator.eval], the [eval] block of sft, and the top level of uv run eval each take an env block. Every source of the group inherits it through vf.merge_env_defaults, and a source's own env values win. Only shared knobs: the block is typed as vf.SharedEnvConfig. It takes retries, timeout, max_concurrent_agents, interception, taskset.task, and taskset.system_prompt. Env ids, taskset ids, and agent roles stay on each source and are rejected in the group block. So
V0.9.1.dev132: fix(examples): set group_size=8 in GRPO recipes (#3693)
Summary Set orchestrator.group_size = 8 in the INTELLECT-3.1 and Qwen3 math/SWE standalone RL examples. Prevent GRPO's default group size of 1 from giving every rollout a zero group-relative advantage. Keep existing explicit group sizes, including GLM base configs used by overlays. Verification Parsed each changed TOML through RLConfig.model_validate; all three resolve to group size 8 for every train source. git diff --check passed. Co-authored-by: Konstantin Dunas git@hallerite.com
V0.9.1.dev131: fix: update eval dependency pins (#3694)
Summary Pin verifiers to 6a82e5d, which includes the Bash/Null certifi fix and the nano-rlm certifi pin. Pin prime-envs to 647ff7a, which includes the GPQA judge fix, DeepSWE missing-test scoring, and Apex Compose host images. Refresh uv.lock. Exclude tau2-bench from the shared workspace because its new Tau2 dependency requires LiteLLM below 1.82.7, while Harbor requires 1.92.0 or newer. Breaking uv sync --all-packages no longer installs tau2-bench. Install and run it in a separate environment until its Tau2 and Harbor LiteLLM
V0.9.1.dev130: chore: drop CLAUDE.md in favor of native AGENTS.md (#3691)
Summary Delete CLAUDE.md. It only imported @AGENTS.md, and Claude Code reads AGENTS.md natively since v2.1.277 (docs). Update the skills line in AGENTS.md to name both the .claude/skills (Claude Code) and .agents/skills (Codex) symlinks. Neither tool reads the other's skills folder, so both symlinks stay. Breaking Claude Code sessions older than v2.1.277 read only CLAUDE.md, so they no longer load the repo instructions. On Bedrock or with telemetry disabled, the minimum is v2.1.281. Migration: update Claude Code.
V0.9.1.dev129: fix(trainer): measure MoE violation per EP group and microstep (#3684)
Summary max_vio is intended to capture the largest expert routing violation for each training microbatch across an entire EP group. The metric uses expert token counts, not the number of tokens allocated to an EP rank. For each microbatch, we take the maximum across MoE layers and EP groups; max_vio/mean and max_vio/max then summarize those microbatch maxima within the logged optimizer step. Why the old numbers depended on topology The router records selections for every expert on each rank before
V0.9.1.dev128: fix: keep workstation blackwell on fa2 in auto-attn (#3681)
Summary Gate attn='auto' → FA4 on SM10x/SM11x (major in (10, 11)) instead of major >= 10. FA4's own interface uses the same families for its native tcgen05/TMEM Blackwell path. B200/GB200 (SM100) and B300/GB300 (SM103) keep FA4, so the goal of #3655 stays intact. Hopper keeps FA3. Workstation Blackwell (SM12x, e.g. RTX PRO 6000) resolves to FA2 again. #3655 sent it to FA4 as a side effect. Update the attn comment in docs/scaling.md. GPU CC attn: auto H100/H200 SM90 FA3 B200/GB200 SM100 FA4 B300/GB300 SM103 FA4 RTX PRO 6000 SM120
Release v0.9.1.dev127 · PrimeIntellect-ai/prime-rl
perf(deepseek_v4): dynamic sparse attention slot count, no recompiles
V0.9.1.dev126: feat(eval): add a BetterStack heartbeat to eval runs (#3664)
Summary The trainer, sft, and orchestrator already beat a Better Stack heartbeat; evals (and the sft online-eval, which shares the EvalRunner) were invisible to it. This gives the eval its own heartbeat config: one ping per landed episode. There is deliberately no startup ping: the first landed episode is the first beat. The gap between run start and the first episode (pool boot, env servers, the first episode's full duration) would otherwise sit inside the ping stream as an abnormally long silence that flips a healthy run stale.
V0.9.1.dev125: add sft example glm5 (#3666)
Note Low Risk Documentation and example TOML only; no changes to training, auth, or runtime code paths. Overview Adds supervised fine-tuning (SFT) to the GLM-5.3 advanced example alongside the existing RL/inference guides. New TOML overlays under sft/h200/ target 8-GPU H200 nodes (8 training nodes, cp/ep = 8, 131k seq_len): base.toml wires zai-org/GLM-5.3-BF16, custom MoE training settings, sign_sgd, and Slurm; math-10k.toml points at PrimeIntellect/INTELLECT-3-SFT-10K (math split); fake.toml swaps in fake data for dry runs.
V0.9.1.dev124: chore: bump prime-envs to latest main (#3672)
Summary Bump the deps/prime-envs submodule from 1213b7336 to b677502a2 (latest main). Pulls in: prime-envs#818: separate original OfficeQA from repaired PI-OfficeQA (new pi-officeqa-pro-v2 env). prime-envs#832 (+ #835, #836): switch swebench_pro to SWE-bench Pro V2, constraining image repository choices. prime-envs#824: SWE-Atlas Harbor environments (new swe-atlas-qna, swe-atlas-rf, swe-atlas-tw envs).
V0.9.1.dev123: Mario/increase moe mfu split size tensorize (#3665)
This PR changes the split representation to use CPU tensors. Performance Benchmarked on Qwen3-30B-A3B with 8× H200, EP 8, balanced routing, bf16, and no activation checkpointing. Version Step ti
V0.9.1.dev122: docs: drop the standalone RL trainer entrypoint from the docs (#3634)
Problem uv run trainer calls prime_rl.trainer.rl.train directly, outside torchrun, so the env:// rendezvous has no RANK/WORLD_SIZE/MASTER_ADDR/MASTER_PORT and the process dies before any training starts: Every documented use of that entrypoint was therefore broken. Scott hit it during onboarding on the README's own validation steps (#3633). Fix Rather than defaulting the rendezvous env for a single-process group (
Release v0.9.1.dev121 · PrimeIntellect-ai/prime-rl
docs(readme): drop stale FA3 source-build step; flash_attn_3 comes fr
Release v0.9.1.dev120 · PrimeIntellect-ai/prime-rl
fix(orchestrator): save final checkpoint before finalizing the run (#
Release v0.9.1.dev119 · PrimeIntellect-ai/prime-rl
perf(deepseek_v4): fused in-place interleaved RoPE for DeepSeek-V4, m
V0.9.1.dev118: chore: bump verifiers to latest main (#3663)
Advance the deps/verifiers submodule from 9e7ec2746 to 69cc0f948, the latest verifiers main, on top of prime-rl origin/main. This picks up interrupted completion stream retries and the follow-up refactor to tenacity. Dependency metadata is unchanged, so no lockfile update is needed. Validation: uv lock --check --offline passed. All four upstream stream-retry regression cases passed with uv run --no-sync pytest --noconftest deps/verifiers/tests/v1/test_e2e.py::test_chat_harness_preserves_streamed_reasoning -q.
V0.9.1.dev117: fix: pre-download data before trainer ranks (#3659)
Summary pre-download Hub data in the SFT launcher, beside model predownload, before trainer ranks start call data predownload only for SFT data, and skip generated fake data in the launcher store the snapshot in the configured Hub cache and pass its local path to the trainer log skipped local paths and download duration like model predownload add optional data.revision and honor it when loading data directly Verification before: research-prod debug job 1400 failed before its first step when
V0.9.1.dev116: chore: update verifiers for buffered streams (#3656)
Summary Advance the deps/verifiers submodule from 9cf0df2 to 9e7ec27, the last VF main commit before Matej's retry fix. Include buffered SSE keepalives and retry replay from verifiers#2650, the empty-tools fix from verifiers#2669, and Prime job-output recovery from verifiers#2670. Include the nano-rlm default revision bump from verifiers#2672. Keep Matej's stream-consumption retry PR verifiers#2654 out of this pin. VF main merged it during this load test. Breaking VF streams now send keepalives while they buffer a model turn.
V0.9.1.dev115: chore(deps): bump dion to 6f9242d (uneven FSDP shards) (#3658)
Summary Bumps the dion pin from 99e5f7e to 6f9242d, picking up samsja/dion#4 (merged into sami/fix-ep): Uneven and empty FSDP shards in Muon: local shards are padded to a uniform peer-transfer size for the all-to-all and trimmed back before/after orthogonalization, so the update is exactly the unpadded Muon update. Previously the optimizer asserted the sharded dim is divisible by the optimizer world size — full Muon failed on the first step for matrices with e.g. 4 or 24 rows on a 64-rank FSDP group.
Release v0.9.1.dev114 · PrimeIntellect-ai/prime-rl
feat(trainer): expose the torch.compile mode (reduce-overhead) in the
V0.9.1.dev113: fix(trainer): b300 fallback to FA4 (#3655)
Problem Two GPU-capability gates in src/prime_rl/trainer/model.py matched (10, 0) exactly, but B300/GB300 are SM103 (compute capability 10.3, per https://developer.nvidia.com/cuda-gpus): resolve_auto_attn: attn: auto on B300/GB300 resolved to flash_attention_2 instead of FA4, silently losing the Blackwell-native attention kernels on every default-config B300 run. apply_quantization (MXFP8 gate): MXFP8 quantization raised requires SM100 (Blackwell) on B300 even though Blackwell Ultra supports it.
Release v0.9.1.dev112 · PrimeIntellect-ai/prime-rl
fix: increase default router timeout to four hours (#3654)
V0.9.1.dev111: fix(inference): align DeepSeek V4 request tool placement (#3646)
Summary attach request-level tools to the first existing DeepSeek V4 system message preserve vLLM's existing synthetic-system behavior when a request has no system message cover both paths and ensure the caller's messages are not mutated vLLM 0.29's Python DeepSeek V4 tokenizer prepends a synthetic tool-only system message even when the request already has a system prompt. That differs from DeepSeek's reference encoder, vLLM's Rust renderer, and prime-rl's training renderer, producing a different token prefix at inference time.
V0.9.1.dev110: fix: mark dashboard episodes failed by ok, not by carried errors (#3649)
Summary The dashboard marks an episode as failed only when ok is false. Retry history no longer turns a successful episode red: the episode table row, the rollout list item, the error-rate tile on the eval overview, and the server's errors_only filter all read ok alone. Episode errors and trace errors render in different places, each tagged with its scope. A failed episode's own errors sit above the agent and branch selectors, since they belong to the whole episode. A failed
V0.9.1.dev109: fix(trainer): preserve Nemotron Mamba parameter dtype (#3557)
Summary Initialize Nemotron Mamba A_log and D parameters with the active default dtype so FSDP units do not mix float32 and bfloat16 storage. Keep the existing float32 state-space computation by casting at use time. Add a focused bfloat16 parameter-dtype regression test. Validation ruff check src/prime_rl/trainer/models/nemotron_h/mamba.py tests/unit/train/models/test_nemotron_h.py Focused in-cluster GPU probe: A_log=torch.bfloat16, D=torch.bfloat16 One-step Nemotron 3.5 Super VLM + MATH + Dynamo run: optimizer
Release v0.9.1.dev108 · PrimeIntellect-ai/prime-rl
feat: scope verifiers limiters by run id and bump verifiers to main (
V0.9.1.dev107: perf: reduce DSV4 indexer stalls and overlap CP gathers (#3643)
Packed sequence lengths were causing repeated FP8-indexer autotuning during training. Other DP groups then arrived late at the layer-4 FSDP prefetch, making a normal ~1–3 ms parameter gather appear to take 1.78 s. Together with narrower checkpoint state, avoiding a query-layout copy, and overlapping CP communication, the matched BF16 benchmark improves from 22.741 s / 40.23% MFU after #3638 to 18.999 s / 47.79% MFU. Bucket the indexer's autotune key by compressed-key length, also keying on head count and dimension.
V0.9.1.dev106: feat(eval): make task dispatch rate configurable (#3602)
Note Low Risk Optional config with default unchanged; reuses existing orchestrator dispatch rate limiting with no auth or data-path changes. Overview Eval runs can now cap how fast episodes are dispatched via a new tasks_per_minute setting on ServedEvalConfig (shared by standalone eval and SFT online eval). None keeps the previous behavior with no rate limit; a positive value throttles dispatch through the existing Dispatcher AsyncLimiter, intended to pace sandbox provisioning during autoscaling.
Release v0.9.1.dev105 · PrimeIntellect-ai/prime-rl
fix(trainer): match vLLM’s per-token FP8 activation quantization bit
Release v0.9.1.dev104 · PrimeIntellect-ai/prime-rl
fix: retain two reduce-scatter input buffers for FP32 MoE routers (#3
Release v0.9.1.dev103 · PrimeIntellect-ai/prime-rl
fix(docker): install the Prime CLI into the image venv for hosted-run
V0.9.1.dev102: fix: read serialized dashboard token counts (#3632)
Summary pin Verifiers main at the merge of Verifiers #2652, including the computed fields from Verifiers #2651 read serialized token counts directly from completed episode and live trace records remove dashboard-specific token reconstruction and prompt-token summation The previous implementation summed every full prompt in a multi-turn trace. This counted the shared conversation once per request and inflated long rollouts. Verification Before this change, a two-turn trace with prompt usages of 100 and 150 tokens
V0.9.1.dev101: chore: bump prime-envs to latest main (#3630)
Summary Bump the deps/prime-envs submodule from 1f1e050ab to 1213b7336 (latest main). Pulls in: prime-envs#791: strip v0 remnants. prime-envs#795: restore solver network access for training environments. prime-envs#797: 500-trace SWE cheating recall benchmark (new trace-cheating-recall-500 judge env). prime-envs#796: run MCP Atlas and Enterprise Ops Gym services in VMs. prime-envs#798: fair-use solver prompts for SWE and terminal training tasksets.
V0.9.1.dev100: fix(inference): extend startup readiness timeout (#3626)
Summary increase the default inference readiness timeout from 30 minutes to 60 minutes keep the direct check_health default aligned with ClientConfig Large models can exceed 30 minutes on a cold node while vLLM compiles architecture-specific kernels and captures CUDA graphs. The previous default can tear down an otherwise healthy launch just before readiness. Test plan PYTHONPATH="$PWD/src:$PWD/packages/prime-rl-configs/src" pytest -q tests/unit/orchestrator/test_clients.py (10 passed) Note Low Risk
V0.9.1.dev99: feat(inference): configurable vLLM router request timeout (#3620)
Summary Add VllmRouterConfig.request_timeout_secs (default 1800 — the router's own built-in default, ge=1): the maximum lifetime of a routed request, including streamed completions. Wire it through both router launch paths — the Python entrypoint (--request-timeout-secs) and the _launch_router.sh.j2 template. When unset, the router behaves exactly as before. Long agentic rollouts set it explicitly, e.g. request_timeout_secs = 3600 for one-hour SWE eval requests.
Release v0.9.1.dev98 · PrimeIntellect-ai/prime-rl
feat(orchestrator): unified taskset shuffle for train and eval source
V0.9.1.dev97: feat(sft): per-row reasoning_effort renderer override (#3621)
Summary A reasoning_effort column in the dataset overrides the [renderer] config's reasoning_effort field per row, so one SFT run can mix reasoning efforts with the configured renderer serving as the default for rows that do not set one (or set null). Requires a typed renderer config that has the field: renderer.name = "auto" and field-less renderers raise a clear ValueError. Renderer configs are frozen, so resolved configs are cached and rows resolving to the same config share one renderer instance. Doc paragraph in docs/training.md;
Release v0.9.1.dev96 · PrimeIntellect-ai/prime-rl
feat(monitors): attach eval runs to a platform-created evaluation (#3
V0.9.1.dev95: chore(deps): bump vLLM to 0.29.0 (#3617)
Bumps vLLM from 0.28.0 to 0.29.0 and reconciles prime-rl's runtime monkeypatches with 0.29's reorganized entrypoint layout. Three patches are now obsolete and removed, each backed by a merged upstream PR. Rebound the API server layer onto vllm.entrypoints.launchers.*: on 0.29 the old openai.api_server path is a deprecation shim, so without this the custom routes and the PrimeRlServingTokens swap would silently not install. Dropped monkey_patch_fp32_lm_head: vLLM does the fp32 lm_head projection natively via head_dtype
V0.9.1.dev94: chore: whitelist the Environments Hub index for uv installs (#3612)
Summary Add an explicit [[tool.uv.index]] named prime-hub for https://hub.primeintellect.ai/primeintellect/simple/ with exclude-newer = false. Hub wheels carry no upload time, so the repo's 7-day exclude-newer cooldown filtered all of them out. uv pip install gsm8k --index prime-hub and uv add gsm8k --index prime-hub now resolve inside prime-rl. The index is explicit, so uv sync never consults it and uv lock --check passes unchanged. Only the named index inherits the setting: passing the same URL via
Release v0.9.1.dev93 · PrimeIntellect-ai/prime-rl
improvement(docker): bake the dashboard extra into the cuda image (#3
V0.9.1.dev92: feat: default enable_ep_weight_filter to true (#3608)
What Adds a typed inference.vllm.enable_ep_weight_filter field defaulting to true, so every vLLM server prime-rl launches now passes --enable-ep-weight-filter (previously this flag only appeared as an explicit pass-through in the GLM-5.3 advanced examples). Drops the now-redundant explicit enable_ep_weight_filter = true from the GLM-5.3 examples (examples/advanced/glm-5.3/swe.toml, examples/advanced/glm-5.3/infer/pd-llmd.toml). Asserts the default lands on the server namespace.
Release v0.9.1.dev91 · PrimeIntellect-ai/prime-rl
chore(examples): glm-5.3 swe.toml base + swe-llmd.toml append, serve
V0.9.1.dev90: fix: select GLM-5.3 renderer in examples (#3606)
Summary configure both GLM-5.3 example runs to use the matching glm-5.3 renderer Requires the GLM-5.3 renderer pinned by #3605.
V0.9.1.dev89: chore: bump renderers for GLM-5.3 (#3605)
Summary pin deps/renderers to the GLM-5.3 renderer implementation Depends on PrimeIntellect-ai/renderers#156. Note Cursor Bugbot is generating a summary for commit 8c41fdb. Configure here.
V0.9.1.dev88: fix: do not abort on consecutive empty train batches (#3595)
Summary Remove the run-level aborts on consecutive empty train batches: The orchestrator raised after 10 consecutive no-sample train batches. MAX_CONSECUTIVE_EMPTY_BATCHES and its counter are gone; an empty batch now logs a warning and is skipped. The train sink raised after 10 batch-equivalents of finalized groups that shipped no payload. The tally now only throttles the warning (_warn_zero_output); there is no abort. Keep both warnings, so a run with no training signal stays visible in the logs.
V0.9.1.dev87: fix: specialize FP8 indexer quantization constants (#3594)
Compiled GLM-5.3 training fails on its first forward pass because eps, fp8_min, and fp8_max reach the FP8 indexer quantization kernel as runtime FP64 scalars. Their arithmetic promotes the quantization result to FP64, and Triton rejects the resulting conversion with Unsupported conversion from f64 to f8E4M3FN with rounding mode rtne. Mark these three fixed scalar arguments as tl.constexpr so Triton specializes them at compile time. Validation: reproduced the failure on an 8-node H200 BF16 trainer with
V0.9.1.dev86: chore(examples): rename GLM example folder to glm-5.3 (#3593)
The advanced GLM presets now train GLM-5.3, but the directory was still named glm-5.2. Rename it and update every inbound link and the "GLM-5/GLM-5.2" prose in the READMEs and docs. Note Low Risk Documentation and example path updates only; no training, inference, or runtime code changes. Overview Renames the advanced GLM example from glm-5.2 to glm-5.3 and updates every inbound link and launch snippet so docs match the presets that target GLM-5.3. README, docs (advanced.md, configuration.md,
Release v0.9.1.dev85 · PrimeIntellect-ai/prime-rl
fix(examples): express GLM-5.2 SWE shared model as a [model] table (#
V0.9.1.dev84: Update swe.toml (#3588)
Note Low Risk Example config-only change with no application code; verify the stack resolves the correct vLLM checkpoint now that [inference.vllm] no longer sets a separate zai-org/GLM-5.3 id. Overview Centralizes the Hugging Face model id for the GLM-5.2 SWE advanced example by adding a top-level model = "zai-org/GLM-5.3-BF16" and dropping duplicate name / model entries under [trainer.model], [inference.vllm], and [orchestrator.model]. Inference still uses the same vLLM settings (BF16 dtype,
V0.9.1.dev83: docs(examples): update GLM SWE presets to 5.3 (#3587)
Note Low Risk Documentation and example TOML checkpoint IDs only; no runtime or auth logic changes. Overview Updates the GLM-5.2 advanced SWE example presets and README so runs target GLM-5.3: trainers load zai-org/GLM-5.3-BF16, while disaggregated vLLM inference serves the zai-org/GLM-5.3 block-FP8 checkpoint (with short comments explaining the train/serve split). swe.toml and swe-llmd.toml swap prior GLM-5 / GLM-5.1 Hugging Face IDs on [inference.vllm], [trainer.model], and [orchestrator.model]. The base
V0.9.1.dev82: Default IPO to eps 0.3 without KL (#3586)
Summary make IPO use eps = 0.3 by default disable the IPO KL regularizer by default with kl_tau = 0.0 update the algorithm documentation to match Explicit [trainer.loss] values continue to override these defaults. Test plan instantiated IPOLossConfig and the default TrainerConfig.loss, confirming eps == 0.3 and kl_tau == 0.0 ruff check packages/prime-rl-configs/src/prime_rl/configs/trainer.py
V0.9.1.dev81: feat: update Prime sandbox SDK (#3585)
Summary Replace the full Prime CLI dependency with prime-sandboxes 0.3.0. Preserve API key and team discovery from prime login through the sandbox SDK config reader. Advance the verifiers submodule to merged PrimeIntellect-ai/verifiers#2626. Document the Prime CLI as a separate tool for login and sandbox cleanup commands. Breaking Remove the Prime runtime vm and guaranteed fields. Delete these fields from configs. Prime now always creates a VM sandbox. Stop publishing ports from Prime service runtimes. Colocate the
Release v0.9.1.dev80 · PrimeIntellect-ai/prime-rl
Replace trainer-side FP8 transfer with online inference quantization
V0.9.1.dev79: Add quantization type for inference configuration (#3580)
Restrict VllmConfig.quantization to fp8_per_block or None using a consistently named QuantizationType alias beside the other vLLM type alias. None continues to let vLLM infer quantization from the checkpoint. Update configs/debug/concurrency.toml to use fp8_per_block so it validates against the typed field. Validation: Ruff lint and formatting checks passed. A config smoke check passed for imports, the default, acceptance of fp8_per_block, rejection of fp8, and the updated concurrency config's vLLM section. Note Medium Risk
V0.9.1.dev78: fix(dashboard): compress large HTTP responses (#3579)
Large run dashboards transfer every metrics chunk without compression, making full-history loads expensive over remote connections. Enable the existing GZip middleware for responses of at least 1,000 bytes, using compression level 1 to limit CPU overhead. Browsers negotiate gzip automatically. JSON payloads and pagination offsets are unchanged, and server-sent events remain uncompressed. This reduces network transfer size while adding compression CPU overhead; it does not reduce the browser's JSON parsing or chart-rendering work.
Release v0.9.1.dev77 · PrimeIntellect-ai/prime-rl
chore(trainer): update the reduce_scatter API and fix a ring attentio
Release v0.9.1.dev76 · PrimeIntellect-ai/prime-rl
feat(examples): add 4-node and 2-node GLM-4.5-Air scaleswe budget var
V0.9.1.dev75: feat: split dashboard overview and metrics tabs (#3578)
Summary The dashboard's Metrics tab is now two tabs for every run type. Overview is the curated view: the W&B overview sections for training runs (train, eval, stability, inference, performance) and the episode pane for eval runs (progress bar, summary tiles, swarms, usage and timing). Metrics is every key in metrics.jsonl, one pane per key, with a regex filter, smoothing and collapse/expand. The Overview/All toggle is gone; the tabs replace it. Eval runs load metrics.jsonl next to their episode series, so the
V0.9.1.dev74: chore: require prime-runs 0.1.3 (#3571)
Require prime-runs[train]>=0.1.3 and update the lockfile from 0.1.2 to 0.1.3. FFT image builds use uv sync --locked, so the PyPI release alone leaves them on the previous SDK and its dual-upload behavior. With the new SDK, training episodes upload only to Prime Traces when the account has access; accounts without access retain legacy samples. Run lifecycle and metrics continue through Platform. Only the SDK requirement and its locked artifact change; pinned submodules and other resolved packages are preserved.
V0.9.1.dev73: feat(inference): sticky least-loaded routing (#3563)
Summary Makes sticky_least_loaded the default routing algorithm on vllm-router (ported from upstream into our fork in PrimeIntellect-ai/router#55). New session are routed to the engine with least active sessions, any requests in an active sessions are sticky-routed to maximize prefix cache hits. So this is a mix of least loaded and consistent hashing which should be strictly better for RL workloads. pin vllm-router v0.2.1 release wheels for x86_64 and aarch64 make sticky_least_loaded the default vllm-router policy while
V0.9.1.dev72: chore: bump verifiers to latest main (#3572)
Summary Bump the deps/verifiers submodule from 5a69fb4d (feat: stream served episodes turn by turn (#2512)) to 28f08939 (latest main). Commits this pulls in: vf#2616 (ff3b8dc5): fix(v1): recover the routed-experts row dropped at each turn boundary — a reused sampled prefix node keeps a placeholder final routing row until a later turn's prefill forwards it; it is now repaired. vf#2618 (28f08939): fix(serve): stream repaired routing rows — served multi-turn episodes had the worker's #2616 repair but the delta stream never delivered it,
V0.9.1.dev71: fix: log periodic gauges through the monitors (#3565)
Summary The PeriodicLogger printed its pipeline view to the console and called wandb.log directly, so the dispatcher/*, concurrency/*, watcher/*, and event_loop_lag/* gauges only ever reached W&B. The file monitor (and with it the local dashboard's metrics.jsonl) and the Prime monitor never saw them; with W&B disabled the gauges existed only as console log text. The periodic payload now fans out through monitors.log(payload, step=None) like the inference-metrics collector, so every registered
V0.9.1.dev70: fix(rl): launch env servers in multi-node RL runs (#3568)
Multi-node RL runs currently launch no environment servers. The orchestrator waits 600 seconds for an address file that nobody writes, then times out, and the trainer sits in the weight broadcast until its own timeout. The flat-to-nested refactor in #3471 replaced format_log_message's two flat train_env_names / eval_env_names parameters with a single env_names: dict[str, list[str]] and updated all four call sites in rl.py. Two of them really are format_log_message calls and were correct. The
Release v0.9.1.dev69 · PrimeIntellect-ai/prime-rl
fix(trainer): declare contiguous outputs in the fp8 and ring attentio
V0.9.1.dev68: chore: ask before changing repository skills (#3564)
Require agents to suggest specific repository skill changes, explain the benefit, and wait for user approval before editing. Existing explicit requests or approvals remain sufficient; ordinary code, workflow, documentation, and PR requests do not authorize skill changes. Reading and following skills remains allowed. Changes only AGENTS.md, replacing its automatic skill-maintenance requirement with the same approval guidance as PrimeIntellect-ai/prime-envs#814. Validation: git diff --check passed; file-scoped pre-commit hooks
Release v0.9.1.dev67 · PrimeIntellect-ai/prime-rl
perf(deepseek-v4): fuse the mHC chain and route unweighted RMS norms
V0.9.1.dev66: fix(orchestrator): taper concurrency growth by KV usage (#3556)
Summary The adaptive controller currently grows the admission cap by 25% per pipeline turnover until GPU-KV usage crosses 60%, then stops growing until the 80% soft-cap. That discontinuity can carry the cap into a corrective trim too quickly. This changes growth to taper linearly from 1.20× per turnover at 0% GPU-KV usage to 1.00× at the 80% soft-cap. Above the soft-cap, the existing soft-trim behavior remains unchanged. The controller also exposes the active growth multiplier as a gauge.
V0.9.1.dev65: feat: consolidate eval cli (#3471)
Summary Migrates the eval CLI from verifiers into the prime-rl ecosystem: uv run eval is prime-rl's entrypoint for evaluating any model on any environment, standalone or as the online evals of a training run. Why: consolidation of features: adaptive concurrency, the local dashboard, resume from the trace stream, live rollouts easier code sharing between the CLI entrypoints (rl, sft, eval) direct access to the pre-installed environments (prime-envs submodule) one workspace for post-training: training and evals Important:
V0.9.1.dev64: fix(inference):Fix native disk KV cache offload tier type (#3538)
Summary Align the disk-backed KV cache secondary tier with vLLM's registered filesystem tier name: fs, not fs_python.#43600 Problem The inference configuration emitted a secondary tier of the form but the tier factory only registers the filesystem secondary tier under the name fs: Because the identifier did not match, vLLM aborted engine initialization with:
V0.9.1.dev63: perf(optim): reuse pinned CPU buffers in CPUOffloadOptimizer (#3552)
Allocate CPUOffloadOptimizer's CPU-side buffers once, pinned, and copy optimizer state into them non-blocking, instead of allocating a fresh pageable buffer plus a fresh pinned buffer on every step. The old path copied each state tensor into fresh pageable memory blocking, then .pin_memory() allocated a second buffer and copied again host to host. Tensor.to cannot allocate pinned memory, and non_blocking was set to not self.pin_memory, which disabled the async path in the one case where a pinned destination would have made it work.
V0.9.1.dev62: fix(docker): drop removed gpt-oss extra from uv sync (#3549)
#3424 removed the gpt-oss optional-dependency group from pyproject.toml, but Dockerfile.cuda still passed --extra gpt-oss to uv sync, which fails on undefined extras and broke every image build since. Note Low Risk Build-only Dockerfile alignment with pyproject extras; no application or dependency logic changes beyond fixing the sync command. Overview Repairs CUDA Docker image builds that broke after the gpt-oss optional extra was removed from pyproject.toml (
Release v0.9.1.dev61 · PrimeIntellect-ai/prime-rl
chore: glm-4.5-air swe example to muon + 2 trainer / 4 inference node
V0.9.1.dev60: chore(trainer): unify context-parallel setup across models (#3533)
Unifies all context parallel setup so that a single frozen CPContext dataclass is shared by all cp-aware modules, stored under a self.cp_contex: CPContext attr. Co-authored-by: Claude Opus 5 (1M context) noreply@anthropic.com
Release v0.9.1.dev59 · PrimeIntellect-ai/prime-rl
docs(examples): restore launch READMEs, add vlm section, slim advance
V0.9.1.dev58: feat(trainer): add optional IcePop loss (#3526)
Summary Add IcePop as an explicit, opt-in RL loss while preserving IPO as the default. Accepted tokens use the importance-weighted policy-gradient term; tokens outside the inclusive trainer/inference ratio band are dropped. IcePop has no separate KL term. adv_tau remains configurable. The acceptance test is performed in log space and masked ratios are replaced before exponentiation. This is mathematically equivalent for accepted tokens while preventing a rejected extreme ratio from producing 0 * inf = NaN in the loss or gradient.
V0.9.1.dev57: chore(examples): drop empty [monitors.prime] sections (#3536)
What Removes the bare [monitors.prime] sections (empty tables, no keys) from the example configs: examples/advanced/glm-4.5-air/search.toml examples/advanced/glm-4.5-air/swe.toml examples/advanced/glm-4.5-air/terminal.toml examples/advanced/nemotron-3-super/swe.toml Why An empty [monitors.prime] table silently enables Prime Lab platform monitoring (streaming metrics/episodes to the platform) for anyone who starts a run from these examples. Dropping the section keeps the
V0.9.1.dev56: fix(qwen3.5): address follow-up review cleanup (#3534)
Follow up on Garrett's review of #3429. Composite MoE models store expert counts in text_config, so is_tt_moe_model missed them and the trainers skipped their load-balancing statistics. Resolve the text config before checking expert counts, with a CPU check covering dense/MoE text and VLM configs. Remove the unused-argument del statements while preserving the public signatures. Drop the zero-initialization assertion and the two-rank GDN test that ordinary CI always skipped, as requested in review. Keep the
V0.9.1.dev55: feat(models): add native Qwen3.5 family implementation (#3429)
Summary Unify Qwen3.5 dense, MoE, and VLM training under a native implementation, including the Qwen3.8 checkpoint that uses this architecture. Remove the separate MoE package and global Transformers model patches. Keep Qwen3_5PreTrainedModel(PreTrainedModelPrimeRL) shared by the backbone and causal LM; remove unused Transformers TP/PP plans, split-module metadata, cache arguments, and redundant initialization. Modules own their initialization, including zero-centered RMSNorm. Use Prime's shared attention and MoE runtime.