inference-optimization/dspark-inkling-small

DSpark drafter for Inkling-Small-NVFP4

A DSpark speculative-decoding drafter trained for the verifier thinkingmachines/Inkling-Small-NVFP4 (multimodal MoE, NVFP4). DSpark drafts a whole block in one parallel pass (DFlash-style backbone) and injects intra-block dependency with a lightweight sequential Markov head, plus a per-position confidence head.

Model details

Algorithm	DSpark (greedy proposal)
Verifier	thinkingmachines/Inkling-Small-NVFP4
Backbone layers	6
Hidden size	4096
Attention heads / KV heads	32 / 8
Draft vocab size	201024 (full padded vocab)
Block size	16
Markov rank	256 (vanilla)
Confidence head	enabled (with Markov)
Aux hidden-state layers	2, 10, 18, 26, 34, 39
dtype	bfloat16

Acceptance evaluation

Inference acceptance measured with vLLM spec-decode over 9 task subsets (~300k drafts total). Mean acceptance length (AL) = expected number of tokens accepted per draft step (higher is better; block size 16).

Subset	Acceptance length	pos-0	pos-1	pos-2	pos-3	pos-4	pos-5	pos-6	pos-7
math_reasoning	4.55	0.776	0.611	0.476	0.376	0.297	0.232	0.182	0.141
HumanEval	3.90	0.740	0.533	0.389	0.288	0.215	0.165	0.129	0.101
rag	3.73	0.736	0.537	0.394	0.288	0.212	0.156	0.113	0.083
translation	3.56	0.736	0.535	0.382	0.269	0.188	0.132	0.094	0.067
summarization	3.40	0.731	0.530	0.375	0.265	0.178	0.119	0.077	0.050
tool_call	3.20	0.707	0.484	0.328	0.222	0.148	0.100	0.067	0.045
writing	3.11	0.679	0.447	0.296	0.198	0.136	0.095	0.068	0.050
qa	3.10	0.684	0.464	0.318	0.216	0.146	0.098	0.065	0.043
question	3.09	0.679	0.446	0.294	0.197	0.136	0.093	0.067	0.048
Average	3.40	0.709	0.493	0.344	0.241	0.170	0.121	0.087	0.063

Draft-weighted average across all subsets shown in the last row.

Per-position acceptance rate, draft-weighted mean over all subsets:

Full per-position numbers are in acceptance_eval.csv.

Usage (vLLM)

Serve the verifier with this drafter as the speculative model:

vllm serve thinkingmachines/Inkling-Small-NVFP4 \
    --speculative-config '{"model": "<this-repo>", "num_speculative_tokens": 16}' \
    --enforce-eager \
    --trust-remote-code

Required vLLM patches (both open at time of writing):

• vllm-project/vllm#52184 — adds Eagle3 hidden-state extraction support for Inkling (the verifier must expose the aux hidden states this drafter consumes).
• vllm-project/vllm#55133 — treats a full/padded-vocab draft (draft vocab_size 201024 vs. the verifier's logical 200058) as an identity map, so no d2t mapping is required to load it.

Also note: Inkling currently requires --enforce-eager (CUDA-graph capture of the custom ops crashes).

Training

Trained with speculators for 3 epochs (checkpoint_best = best validation epoch). Validation at the selected checkpoint: accept_len 3.67, accept_rate 0.40, pos-0 acc 0.776. Loss {"ce": 0.1, "tv": 0.9}, lr 1e-4, seq len 8192, block size 16.

Training data: orestis-z/Inkling-Small-NVFP4-Regenerated-Collection — on-policy data where Inkling-Small-NVFP4 regenerates the assistant responses (turn-by-turn, thinking effort 0.9) over Magpie + UltraChat prompts (~500k conversations). General chat/instruct mix.

Training curves (run dspark_inkling_small_v2, logged with trackio):

Provenance / reproducibility:

• train_command.txt — exact training command, git SHA, and package versions (speculators 0.7.0.dev155, vllm 0.27.1, transformers 5.15.0, torch 2.13.0+cu130, compressed-tensors 0.17.0).
• training_state.json — epoch / global step of this checkpoint.
• val_metrics.json — full validation metrics.