CTBench is an agentic benchmark for evaluating AI agents in realistic telecom Network Operations and Maintenance (NetO&M) troubleshooting scenarios. Unlike benchmarks that only measure final-answer accuracy, CTBench evaluates whether an agent can behave like a competent telecom engineer: interact with diagnostic tools, collect relevant evidence, reason over network state, and produce an operationally valid answer.

CTBench Automatic Evaluation Framework

CTBench Automatic Evaluation Framework.

CTBench contains two task families:

Root Cause Analysis (RCA): The agent must localize the affected node and object, identify the correct root cause, and justify the diagnosis using diagnostic evidence.
Path Restoration: The agent must reconstruct service forwarding paths from source to destination by querying device outputs and reasoning over routing, interfaces, tunnels, policies, and forwarding behavior.

The current release includes 126 RCA tasks and 108 path-restoration tasks. Each task is constructed and reviewed by telecom domain experts. Agents are given only the task instruction and permitted diagnostic interface; they do not receive ground-truth answers, metadata labels, or expert solution paths during inference.

Key Features

CTBench provides expert-normalized answers and process-level annotations:

Normalized RCA answers: root-cause triples of the form
(fault_node, fault_object, root_cause).
Normalized path-restoration answers: ordered forwarding paths with nodes, interfaces, and path elements.
Golden actions: expert-validated diagnostic actions required to solve each task.
Task metadata: observability level, root-cause category, root-cause count, fault-propagation-chain length, restored-path count, protocol complexity, network heterogeneity, and golden-solution length.

These annotations enable evaluation beyond final correctness. CTBench measures whether an agent reaches the correct answer, whether it localizes the problem precisely, whether it identifies the correct root cause or restored path, and whether it collects the evidence needed to support its conclusion.

Evaluation Dimensions

CTBench reports task-specific capability metrics:

RCA Metrics

RCA Localization: whether the agent identifies the affected node and object.
RCA Identification: whether the agent identifies the correct root-cause label.
RCA Evidence: whether the agent collects relevant diagnostic evidence, measured against expert golden actions.

Path-Restoration Metrics

Path Localization: whether the agent identifies the correct source and destination endpoints.
Path Restoration: whether the agent reconstructs the correct forwarding path.
Path Evidence: whether the agent collects the evidence required to justify the reconstructed path.

CTBench also reports cost and efficiency metrics, including interaction rounds, latency, and token consumption.

Task Difficulty Metadata

Each task includes structural metadata to support fine-grained analysis of agent behavior:

Evidence observability: whether decisive evidence is directly observable or must be inferred from indirect observations.
Network heterogeneity: the diversity of vendors and device families involved.
Protocol complexity: the number and interaction depth of protocols, policies, and service mechanisms.
Fault-propagation chain: the causal depth between root cause and observed symptom.
Answer multiplicity: number of independent root causes or restored paths.
Golden solution length: number of expert-required diagnostic actions.

These metadata allow researchers to study which operational conditions most challenge current agents, rather than treating all tasks as equally difficult.

Intended Use

CTBench is designed for evaluating LLM-based agents, tool-using agents, and telecom-specialized models in realistic troubleshooting workflows. It can be used to study:

telecom-domain reasoning,
evidence-grounded diagnosis,
tool-use behavior,
robustness under partial observability,
multi-vendor and multi-device troubleshooting,
path reconstruction and forwarding reasoning,
capability-cost tradeoffs.

Downloads last month: 31

Total file size:

190 MB