Multimodality support - #2232
Closed
OnePunchMonk wants to merge 14 commits into
Closed
Conversation
… and a new tokenizer utility.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
OnePunchMonk
requested review from
KaelanDt,
andyland,
k223kim,
lianakoleva and
t-vi
as code owners
April 9, 2026 19:44
OnePunchMonk
force-pushed
the
multimodality-support
branch
from
April 9, 2026 19:52
b2969ec to
d729598
Compare
for more information, see https://pre-commit.ci
OnePunchMonk
force-pushed
the
multimodality-support
branch
from
April 9, 2026 19:58
55cc41b to
7342e26
Compare
OnePunchMonk
marked this pull request as draft
April 11, 2026 05:41
OnePunchMonk
force-pushed
the
multimodality-support
branch
from
April 12, 2026 10:52
8af9af0 to
5dd8a28
Compare
…ismatched architecture litgpt's generic VisionEncoder/MultiModalProjector don't match Gemma3's SigLIP tower or Gemma3MultiModalProjector 1:1 (different parameter names and shapes, e.g. mm_input_projection_weight/mm_soft_emb_norm vs a plain nn.Linear), and none of the Gemma3 configs opt into config.is_multimodal. Loading the vision/mm-projector weights unconditionally broke state_dict loading for every Gemma3 checkpoint conversion (this was causing the CI failures on this PR). Only attempt the vision weight mapping when config.is_multimodal is set; otherwise fall back to the previous text-only behavior.
OnePunchMonk
marked this pull request as ready for review
August 16, 2026 14:36
…r on GPT subclasses These GPT subclasses bypass GPT.__init__ (calling nn.Module.__init__ directly), so they never received the vision_encoder/mm_projector attributes added for multimodal support. forward() unconditionally reads self.vision_encoder, so passing pixel_values through a LoRA/Adapter/AdapterV2 model raised AttributeError.
Contributor
Author
|
Closing in favor of #2301 — this branch had gone stale against |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #2173