SaylorTwift/lfm-opencode-lora-rl-code

OpenCode-only RL on LFM2.5-2.6B with LoRA

Reproduction of the OpenCode-only run from The ultimate guide to multi-harness RL, but training a LoRA adapter instead of full weights, so the result is a small, distributable adapter on top of LiquidAI/LFM2.5-2.6B.

Stack: OpenEnv (capture proxy + Harbor integration) → Harbor (tasks + HF Sandboxes) → TRL AsyncGRPOTrainer with HarnessRolloutWorker → vLLM policy server. Every API here was verified against the released wheels (trl 1.16.0, openenv 0.10.0, harbor 0.24.0), the harbor_env source, and the FineEnvs production env Space.

Version pins that matter: trl==1.15.0 (1.16.0 made tree packing unconditional in AsyncGRPOTrainer and refuses hybrid models — LFM2.5 has conv layers; 1.15.0 has the harness worker but predates tree packing) and vllm/vllm-openai:v0.31.0 (top of trl 1.15.0's supported vLLM band, and predates its mcp>=2 requirement which conflicts with openenv's fastmcp 3.x).

Sandboxes are HF Sandboxes (Harbor's hf-sandbox backend, harbor>=0.24): each rollout boots an HF Jobs cpu-basic machine running the task's prebuilt image (savatar101/env-data-agent-train:base — the same image every task's Dockerfile is a one-line FROM of). Only HF_TOKEN is needed; no Daytona/E2B account.

Why merged-weight sync (source-verified)

Generation goes harness → capture proxy → vLLM, addressed by a fixed model name. TRL's adapter-only sync publishes versioned adapters (trl-policy-v{N}) and only the trainer-built default worker learns the versioned name; an injected HarnessRolloutWorker never gets one. So the server runs without --enable-lora and the trainer merges the LoRA into the base and NCCL-streams full weights each step (~5.2 GB for 2.6B — trivial on one node; vLLM also serves adapters 1.33–1.39x slower per token). Checkpoints still contain only the LoRA adapter — that is what you ship.

Files

File	Purpose
train_lora.py	Worker wiring + AsyncGRPOConfig + LoraConfig (r=32 on q/k/v/out_proj, w1/2/3). Task dataset is discovered from the env server via HarborSessionFactory.prompt_rows() — create() rejects any prompt the server doesn't serve
serve_policy.sh	vLLM with capture-proxy flags + NCCL weight transfer
serve_env.sh	openenv harbor serve with production env vars (MAX_CONCURRENT_ENVS, pinned OpenCode 1.18.31, 4096 output cap)
prepare_tasks.py + data/	Materializes the pinned 1,000 train / 250 test tasks (hash-checked, from the FineEnvs Space) and patches docker_image into each task.toml — the one change HF Sandboxes require
run_job.sh	Single-container entrypoint for HF Jobs (setup + all 3 processes)
export_adapter.py	Publish any adapter checkpoint to the Hub (+ optional merged model)

Launch on HF Jobs

# 0. One-time: put this folder on the Hub so the job can pull it
hf repos create YOU/lfm-opencode-lora-rl-code --type model --private
hf upload YOU/lfm-opencode-lora-rl-code . \
  --include "*.py" --include "*.sh" --include "*.md" --include "data/*.json"

# 1. Smoke run (~1-2h): validates weight sync, capture contract, sandbox boots
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 6h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=20 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'

# 2. Watch: merged weight transfers (no .vllm_lora/), sandbox Jobs booting under
#    your account (hf jobs ps), rollouts returning rewards, no CaptureContractError.
hf jobs logs <job-id> --follow

# 3. Full run (the article's OpenCode-only run took ~32h on 2x H100)
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 72h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=1000 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'

Checkpoints (adapter-only) push to ADAPTER_REPO every 100 steps, so a dead job loses little. SANDBOX=daytona/e2b + the matching API key secret still works.

What you distribute

Every checkpoint contains adapter_model.safetensors + adapter_config.json (~50 MB at r=32). export_adapter.py adds a model card and pushes:

python export_adapter.py checkpoints/checkpoint-1000 YOU/LFM2.5-2.6B-opencode-lora-RL [--also-merged]

Consumers load it with PeftModel.from_pretrained("LiquidAI/LFM2.5-2.6B", YOU/...) or vllm serve LiquidAI/LFM2.5-2.6B --enable-lora --max-lora-rank 32 --lora-modules opencode-lora=YOU/... (Lfm2ForCausalLM implements SupportsLoRA).

Recipe choices carried over from the article (don't regress these)

• top_p=1.0, top_k=0 — truncation biases the policy and breaks logprob agreement.
• Same 4096-token output cap in training and evaluation (OPENENV_MAX_OUTPUT_TOKENS
  • worker max_tokens). A Qwen run with 16k training answers collapsed at 4k eval.
• Correctness + <=0.1 tool-efficiency bonus on correct answers (else all-correct GRPO groups carry no signal and tool calls creep up).
• num_generations=8, one harness per group, so advantage compares actions.
• max_tool_calling_iterations=32 on the worker — the loop-owning default would silently cap the agent at 8 model calls.
• max_staleness=4 (config default) matches the article's async setup.

Honest caveats

• HarnessRolloutWorker + peft_config and sandbox="hf-sandbox" are both untested-in-print combinations on this stack. All pieces are released and the interfaces line up (verified in source) — the smoke run exists to prove it before you spend 32+ GPU-hours.
• HF Sandboxes are experimental (per HF docs) and each rollout is an HF Job under your token: mind your Jobs concurrency quota (MAX_CONCURRENT_ENVS=8 bounds it) and per-minute billing of thousands of short cpu-basic jobs over a full run.
• Sandbox boot time (job scheduling + first image pull) adds latency per rollout vs Daytona/E2B's warm pools; expect slower wall-clock than the article's 32h.
• Expect the article's single-harness shape: gains concentrate in OpenCode (34 → 58% there in their run) and transfer unevenly elsewhere.
• LoRA r=32 may cap peak accuracy vs full FT; no published comparison on this stack.