G2VLM-2B-MoT finetuned on VSI-590K
Finetuned weights for InternRobotics/G2VLM-2B-MoT on a balanced video subset of VSI-590K, trained as a video spatial-reasoning baseline (VSI-Bench style questions).
These are delta checkpoints: each file holds only the tensors that were finetuned (395 tensors, 1.78B parameters, fp32) and is loaded on top of the original G2VLM-2B-MoT weights.
Files
| Path | Training | Steps | Samples seen |
|---|---|---|---|
ft2ep_vsi590k_bal/delta_final.safetensors |
2 epochs over the subset | 3,500 | 28,000 |
ft_vsi590k_bal/delta_final.safetensors |
short run (100 min budget) | 366 | 2,928 |
Each folder also has a train_log.json with the per-step loss and the training arguments.
ft2ep_vsi590k_bal is the longer run and the one to use by default.
Training setup
- Data: 14,000 video QA samples from VSI-590K, 2,000 for each of 7 question types
(
absolute_count,absolute_distance_object,absolute_size_object,absolute_size_room,relative_distance_object,relative_direction_object,appearance_order), drawn from the ScanNet, ScanNet++ v2 and ARKitScenes training videos, shuffled with seed 0. No scene overlaps with the VSI-Bench test set. - Input: 16 uniformly sampled frames per video, official
chat_with_reconlayout (both the semantic and the geometry branch see the frames). - Trainable parameters: the semantic (understanding) expert of the language model, including the token embeddings and LM head. The geometry expert, DINO encoder, ViT and 3D heads stay frozen.
- Optimisation: AdamW (betas 0.9/0.95, no weight decay), learning rate 2e-5 with cosine decay to 10%, 50 warmup steps (20 for the short run), effective batch size 8 (2 GPUs x 4 accumulation steps), cross-entropy loss on the answer tokens only.
Usage
Clone the G2VLM code (tested at commit a45f2f4),
download the base model, then load a delta on top of it:
from huggingface_hub import hf_hub_download, snapshot_download
from safetensors.torch import load_file
from g2vlm_utils import load_model_and_tokenizer_from_path # from the G2VLM repo
base = snapshot_download("InternRobotics/G2VLM-2B-MoT")
model, tokenizer, new_token_ids, vit_transform, dino_transform = load_model_and_tokenizer_from_path(base)
delta = load_file(hf_hub_download("RunyiY/G2VLM", "ft2ep_vsi590k_bal/delta_final.safetensors"))
msg = model.load_state_dict(delta, strict=False)
assert not msg.unexpected_keys
The delta keys are plain G2VLM state-dict names (language_model.*), so
load_state_dict(..., strict=False) works on any G2VLM instance built from the base config.
Inference then follows the official inference_chat.py. Prompts used for finetuning are the
VSI-Bench style ones: the question followed by "Answer with the option's letter from the given
choices directly." for multiple choice, or "Please answer the question using a single word or
phrase." for numerical answers.
Patches needed for training
Finetuning the released code needed two one-line fixes (not needed for inference):
modeling/g2vlm/g2vlm.py, inG2VLM.forward: the returnedmse=mserefers to an undefined variable when only the text loss is used; replace withmse=None.modeling/g2vlm/qwen2vl.py, inQwen2VLModel.forward_train: guard the hidden-state collection withif output_hidden_states and intermediate_layers is not None:.
License
Apache 2.0, following the base model.