Four LoRA runs on our own agent decisions: what $327 of GPU time bought
We run AI agents for real businesses. Much of their work runs on frontier models, and I want an open model that can carry the everyday part reliably. So over four days in October (20 to 23) we fine-tuned Qwen/Qwen3.6-35B-A3B four times on reviewed decisions from our own production agents. Each run had a pass bar written down before any GPU was rented. The full log, including the failures, is now a dataset: Scalably/qwen-agent-sft-experiments.
The training examples and adapters are not in it. They come from our clients' real sessions, and our clients own that data.
The setup
Every candidate example is one decision an agent made in production: which tool to call, with which arguments, after which context. Two model families (Anthropic's Claude and OpenAI's Codex) reviewed each one blind, and only examples both passed went into training: 442 of 800 in the first run. Training was LoRA (rank 16) in ms-swift on rented GPU boxes, mostly 2x H200, one night per run. Evaluation replays held-out decisions, including decisions from clients the model never saw in training.
What moved
- X17, plain SFT: right tool and arguments went from 44.7% to 63.3% and 64.4% on two seeds, and the gain held on clients never seen in training. It did not learn to avoid calls that had failed in production: on that test one seed was no different from the base model and the other was slightly worse.
- X18, reasoning SFT: we trained the model to think before acting, with rationales written by a teacher model. Right tool and arguments: 36.7% to 58.7%. Loading the required skill before a gated tool: 55.6% to 98.4%. IFEval didn't drop (0.750 to 0.765).
- X18, DPO on top: 60.2%, statistically the same as SFT alone, though only 158 of 301 preference pairs fit in memory. We dropped DPO.
- X19, skills it never saw: we held six skills out of the training examples (their names still appear in system catalogs). The skill gate on those went from 54.9% to 93.7%, and X18 scored even higher (99.1%), so the habit transfers. The test turned out easier than designed: the block message names the skill to load. Everyday accuracy fell from 60.2% (X18) to 51.5%.
- X20, rebalanced: doubling the weight of everyday decisions brought accuracy back to 63.6%, but the correct first step after loading a held-out skill fell from 65.8% in X19 to 36.8%. X19's gain there was also not significant against the base model (p 0.12).
What I take from it
Reasoning SFT on carefully reviewed decisions is worth doing; preference training on top wasn't, at this size. Imitation teaches the model what to call, not when to stop. Every mix trades one skill for another, which is why each run had a preregistered bar and why the next phase trains on whole multi-step tasks, then reinforcement learning, rather than a fifth single-step mix.
Cost
4 runs, about $327 of rented GPU time ($110.92 invoiced, the rest estimated from hourly price and box time). Data review and teacher rationales ran on AI subscriptions at no marginal cost. X20 closed its own box unattended: verify the outputs, wipe, destroy.
How this was made
AI agents did the engineering: Claude ran the pipeline and the evaluations, Codex and Claude reviewed the data and the results, and I set the questions and approved every spend. No outside expert has reviewed it yet. One seed per run except X17 and small test sets (38 to 264 cases per line), so check the p-values in the runs subset before trusting a gap, and open a discussion if something looks off.
