G2VLM-2B-MoT finetuned on VSI-590K

Finetuned weights for InternRobotics/G2VLM-2B-MoT on a balanced video subset of VSI-590K, trained as a video spatial-reasoning baseline (VSI-Bench style questions).

These are delta checkpoints: each file holds only the tensors that were finetuned (395 tensors, 1.78B parameters, fp32) and is loaded on top of the original G2VLM-2B-MoT weights.

Files

Path Training Steps Samples seen
ft2ep_vsi590k_bal/delta_final.safetensors 2 epochs over the subset 3,500 28,000
ft_vsi590k_bal/delta_final.safetensors short run (100 min budget) 366 2,928

Each folder also has a train_log.json with the per-step loss and the training arguments. ft2ep_vsi590k_bal is the longer run and the one to use by default.

Training setup

  • Data: 14,000 video QA samples from VSI-590K, 2,000 for each of 7 question types (absolute_count, absolute_distance_object, absolute_size_object, absolute_size_room, relative_distance_object, relative_direction_object, appearance_order), drawn from the ScanNet, ScanNet++ v2 and ARKitScenes training videos, shuffled with seed 0. No scene overlaps with the VSI-Bench test set.
  • Input: 16 uniformly sampled frames per video, official chat_with_recon layout (both the semantic and the geometry branch see the frames).
  • Trainable parameters: the semantic (understanding) expert of the language model, including the token embeddings and LM head. The geometry expert, DINO encoder, ViT and 3D heads stay frozen.
  • Optimisation: AdamW (betas 0.9/0.95, no weight decay), learning rate 2e-5 with cosine decay to 10%, 50 warmup steps (20 for the short run), effective batch size 8 (2 GPUs x 4 accumulation steps), cross-entropy loss on the answer tokens only.

Usage

Clone the G2VLM code (tested at commit a45f2f4), download the base model, then load a delta on top of it:

from huggingface_hub import hf_hub_download, snapshot_download
from safetensors.torch import load_file
from g2vlm_utils import load_model_and_tokenizer_from_path  # from the G2VLM repo

base = snapshot_download("InternRobotics/G2VLM-2B-MoT")
model, tokenizer, new_token_ids, vit_transform, dino_transform = load_model_and_tokenizer_from_path(base)

delta = load_file(hf_hub_download("RunyiY/G2VLM", "ft2ep_vsi590k_bal/delta_final.safetensors"))
msg = model.load_state_dict(delta, strict=False)
assert not msg.unexpected_keys

The delta keys are plain G2VLM state-dict names (language_model.*), so load_state_dict(..., strict=False) works on any G2VLM instance built from the base config.

Inference then follows the official inference_chat.py. Prompts used for finetuning are the VSI-Bench style ones: the question followed by "Answer with the option's letter from the given choices directly." for multiple choice, or "Please answer the question using a single word or phrase." for numerical answers.

Patches needed for training

Finetuning the released code needed two one-line fixes (not needed for inference):

  • modeling/g2vlm/g2vlm.py, in G2VLM.forward: the returned mse=mse refers to an undefined variable when only the text loss is used; replace with mse=None.
  • modeling/g2vlm/qwen2vl.py, in Qwen2VLModel.forward_train: guard the hidden-state collection with if output_hidden_states and intermediate_layers is not None:.

License

Apache 2.0, following the base model.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for RunyiY/G2VLM

Base model

Qwen/Qwen2-VL-2B
Finetuned
(1)
this model

Dataset used to train RunyiY/G2VLM