Multimodality support

[OnePunchMonk]

closes #2173

Supersedes #2232 (which had gone stale against main and was hard to review after 4 months / 14 commits). This is the same feature rebased onto current main as a single commit, plus a fix for a bug found while chasing down a CI failure on the old PR: the LoRA/Adapter/AdapterV2 GPT subclasses construct their own state instead of calling GPT.__init__, so they never received the new vision_encoder/mm_projector attributes — passing pixel_values through those model types raised AttributeError.

Summary

• Adds litgpt/vision.py: VisionEncoder (HF backbone or conv-patch fallback), MultiModalProjector (linear/mlp2x), merge_input_embeds, ImagePreprocessor.
• Config gains optional vision_* fields and an is_multimodal property.
• GPT.forward accepts pixel_values and merges projected image-patch embeddings into the token embedding sequence at <image> placeholder positions. Same wiring added to LoRA/Adapter/AdapterV2 GPT subclasses.
• litgpt.generate.base / litgpt.chat.base thread pixel_values through generation; LLM.generate() accepts an image path/PIL input.
• convert_hf_checkpoint: only load Gemma3 vision tower / mm-projector weights when the target config explicitly declares a matching vision architecture, to avoid corrupting the state_dict when shapes don't match Gemma3's SigLIP tower.

Test plan

• tests/test_vision.py (new, 23 tests) covers VisionEncoder, MultiModalProjector, merge_input_embeds, ImagePreprocessor, and GPT integration.
• Ran full tests/test_lora.py, tests/test_adapter.py, tests/test_adapter_v2.py, tests/test_model.py locally on CPU against latest main — all pass.
• ruff check passes on all changed files.

AI Usage Disclaimer

• AI assistance (Claude Code) was used for this change.

[OnePunchMonk]

GPU test results (Modal, A10G)

Ran this PR's branch (multimodality-support-v2, 0f4e468) end-to-end on a real GPU via Modal, installing with pip install .[extra,test,compiler] on Python 3.11 / an NVIDIA A10.

Test suite

File	Result
tests/test_vision.py	23 passed
tests/test_lora.py	392 passed, 3 skipped
tests/test_adapter.py	15 passed, 6 xfailed
tests/test_adapter_v2.py	226 passed
tests/test_model.py	515 passed, 7 skipped, 62 xfailed, 22 xpassed
tests/test_chat.py	13 passed
tests/generate/test_main.py	9 passed, 1 xfailed

Total: 1193 passed, 0 failed across the vision, LoRA, Adapter, AdapterV2, base model, chat, and generate suites, confirming the GPT/LoRA GPT/Adapter GPT/AdapterV2 GPT vision wiring (mentioned in the PR description) doesn't regress any of those paths on GPU.

Backward-compatibility check

None of the existing suites explicitly call GPT.forward both with and without the new pixel_values kwarg on a non-multimodal config, so I added a standalone check:

config = Config.from_name("pythia-14m")  # no vision_* fields
model = GPT(config)
...
out_no_kwarg = model(idx)                    # old call signature
out_none_kwarg = model(idx, pixel_values=None)  # new call signature
assert torch.equal(out_no_kwarg, out_none_kwarg)

Result: BACKWARD_COMPAT_OK — identical output whether pixel_values is omitted or passed as None, so existing non-multimodal checkpoints are unaffected by this change.

[OnePunchMonk]

@bhimrazy could you provide a first review of this when you have some time to spare?