| --- |
| license: apache-2.0 |
| task_categories: |
| - text-generation |
| language: |
| - en |
| tags: |
| - code-generation |
| - tool-use |
| - agent |
| - knapsack |
| - synthetic-data |
| - runtime-semantics |
| pretty_name: "Agents Learn Their Runtime - Evaluation Task Instances" |
| size_categories: |
| - n<1K |
| configs: |
| - config_name: tasks |
| default: true |
| data_files: |
| - split: easy |
| path: easy/knapsack/*.json |
| - split: hard |
| path: hard/knapsack/*.json |
| --- |
| |
| # Agents Learn Their Runtime -- Task Definitions |
|
|
| Paper: [Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics](https://arxiv.org/abs/2603.01209) |
|
|
| 200 procedurally generated **Opaque Knapsack** tasks, split into 100 Easy and 100 Hard instances. These are the shared evaluation problems solved by all models in the paper's experiments. |
|
|
| ## The Opaque Knapsack Task |
|
|
| A partially observable constrained optimization problem. An agent is given a set of items identified only by opaque IDs and must: |
|
|
| 1. Call `inspect(item_id)` to reveal an item's weight, value, and class (costs one unit of a limited inspection budget) |
| 2. Call `take_item(item_id)` to select items, maximizing total value without exceeding a weight capacity |
| 3. Respect class-validity constraints (only certain item classes are allowed) |
|
|
| Item properties are hidden behind random IDs, so the task is unsolvable by memorization. The agent must track running weight totals, budget usage, and candidate rankings across multiple steps. |
|
|
| ## Related Datasets |
|
|
| | Dataset | What it contains | |
| |---|---| |
| | **This dataset** | Task definitions (the problems) | |
| | [Training traces](https://huggingface.co/datasets/AutomatedScientist/agents-learn-runtime-train) | 2,000 teacher solutions by **Gemini 3 Flash** (1K persistent + 1K stateless), used to fine-tune LoRA adapters | |
| | [Benchmark traces](https://huggingface.co/datasets/AutomatedScientist/agents-learn-runtime-benchmarks) | 1,200 inference traces from **Qwen3-8B** (base + 2 LoRA adapters) solving these exact tasks across 12 conditions | |
|
|
| ## Structure |
|
|
| ``` |
| tasks/ |
| ├── easy/knapsack/ |
| │ └── knapsack-0000000000.json ... knapsack-0000000099.json |
| └── hard/knapsack/ |
| └── knapsack-0000000000.json ... knapsack-0000000099.json |
| ``` |
|
|
| ## File Schema |
|
|
| Each JSON file fully specifies a single task instance: |
|
|
| ```json |
| { |
| "task_id": "unique identifier", |
| "family": "knapsack", |
| "seed": 12345, |
| "difficulty": { |
| "n_items": 36, |
| "capacity": 34, |
| "budget_coverage": 0.58, |
| "p_valid": 0.2, |
| "optimal_set_size": 3, |
| "max_item_dominance": 0.38 |
| }, |
| "public": { "capacity": 34, "budget": 21, "valid_classes": ["A", "C"] }, |
| "private": { "items": { "item_abc123": {"weight": 5, "value": 12, "class": "A"} } }, |
| "reference": { "optimal_value": 47, "optimal_items": ["item_abc123", "..."] }, |
| "nl": { "title": "...", "instructions": "...", "output_format": "..." } |
| } |
| ``` |
|
|
| | Field | Description | |
| |---|---| |
| | `public` | Parameters revealed to the agent (capacity, budget, valid classes) | |
| | `private` | Ground-truth item properties, hidden behind `inspect()` at runtime | |
| | `reference` | Optimal solution for scoring | |
| | `nl` | Natural-language prompt given to the agent | |
| | `difficulty` | Generation parameters controlling problem hardness | |
|
|
| ## Reproduction |
|
|
| Tasks are generated via `make tasks` in the [source repo](https://github.com/mrcabbage972/agents-learn-runtime), which calls `pythonformer.cli` with config files from `pythonformer/task_configs/`. |
|
|
| ## License |
|
|
| Apache License 2.0 |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{may2026agents, |
| title={Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics}, |
| author={May, Victor and Salgarkar, Aaditya and Wang, Yishan and Misra, Diganta and Nguyen, Huu}, |
| journal={arXiv preprint arXiv:2603.01209}, |
| year={2026} |
| } |
| ``` |
|
|
