studio: allow specifying family override for image and video diffusion models (#8618) - #8622
studio: allow specifying family override for image and video diffusion models (#8618)#8622karan68 wants to merge 4 commits into
Conversation
…n models (unslothai#8618) - Add UI selector in Image and Video Generation Advanced Options panels for family_override. - Thread family_override through load and download planning requests. - Add delimiter flexibility and aliases in diffusion_families.py and video_families.py. - Add backend unit tests for family overrides and delimiter variations.
c255d27 to
613fdd5
Compare
karan68
changed the title
|
@codex review |
|
@karan68 this screenshot is not from Unsloth Desktop... have you manually verified and tested the fix? |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 129805030f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| attention_backend: attentionBackend === "auto" ? undefined : attentionBackend, | ||
| memory_mode: memoryMode === "auto" ? undefined : memoryMode, | ||
| transformer_cache: transformerCache === "auto" ? undefined : transformerCache, | ||
| family_override: familyOverride === "auto" ? undefined : familyOverride, |
There was a problem hiding this comment.
When an override is needed for an opaque custom repository, the image picker still calls defaultsFor(repoId), which falls back to the Z-Image recipe of 9 steps and guidance 0; the post-load resident seeding effect also skips this load because lastLoad.current is set. Consequently, for example, an opaque SDXL merge loaded with the new sdxl override immediately generates with 9/0 instead of SDXL's 30/7 unless the user manually corrects the sliders. Use the selected family when deriving defaults or reseed them from the resolved family after loading.
Useful? React with 👍 / 👎.
| ["ltx-2", "LTX-2"], | ||
| ["wan2.2-ti2v-5b", "Wan2.2-TI2V-5B"], | ||
| ["wan2.2-t2v-a14b", "Wan2.2-T2V-A14B"], | ||
| ["hunyuanvideo-1.5", "HunyuanVideo-1.5"], |
There was a problem hiding this comment.
For an opaque custom HunyuanVideo-1.5 720p fine-tune—the exact case requiring this selector—the only available Hunyuan option resolves to the 480p family. The backend has a distinct hunyuanvideo-1.5-720p family with a different base repository, scheduler shift, and disjoint resolution presets, and name-based 720p rerouting cannot help when an explicit generic override is supplied. Add the 720p family to this list so these checkpoints are not loaded and generated with the 480p configuration.
Useful? React with 👍 / 👎.
| ); | ||
| const [memoryMode, setMemoryMode] = useState<"auto" | "fast" | "balanced" | "low_vram">("auto"); | ||
| const [transformerCache, setTransformerCache] = useState<"auto" | "off" | "fbcache">("auto"); | ||
| const [familyOverride, setFamilyOverride] = useState<string>("auto"); |
There was a problem hiding this comment.
When the page mounts with an opaque custom pipeline already resident, this state resets to auto even though the backend now reports the original explicit choice in status.resolved.family_override; the Advanced reseeding effect restores quantization, memory, and attention but never this control. The resident-pipeline path still enables Reapply, so clicking it omits the override and the backend rejects the otherwise valid model as an unknown family. Seed this selector from the resolved family record just like the other load-time controls.
Useful? React with 👍 / 👎.
| family_override: | ||
| controls.familyOverride === "auto" ? undefined : controls.familyOverride, |
There was a problem hiding this comment.
When minimax-h3 is selected for an opaque custom pipeline, the override reaches the backend but the frontend's isH3PipelinePick checks only the repository's literal name. Such a pick therefore bypasses the H3 task dialog and sends no h3_task, which defaults the modular loader and download plan to the fl2va transformer; the separate 66 GB transformer_ref partition and References workflow become unreachable, and a reference fine-tune can load the wrong denoiser partition. Include the resolved override in the H3 task-routing decision.
Useful? React with 👍 / 👎.
|
@mahiatlinux Yes, here is a screenshot from the running Unsloth Studio interface showing the new Family override control under Advanced Options: Review Feedback Addressed & Verified:
|
2138516 to
d0062aa
Compare
d0062aa to
cfa6de2
Compare
3 participants
Fixes #8618
Problem
When loading fine-tunes, merges, or custom derivative checkpoints for diffusion image or video models (e.g.
user/FLUX.2_klein_9B,user/flux2_klein,user/klein-image-gen, or custom community LoRA/GGUF repacks), the backend raises a refusal because the repo id does not match canonical family substring heuristics:_ValueError: '{model_name}' is not a supported diffusion image model. Supported families: flux.1, flux.2-klein, flux.2-dev, ... If this is a variant of one of them, pass family_override with that family name.
While the backend load and download-plan endpoints already accepted a
family_overrideparameter, there was no way for users to set this override in the Studio UI. Additionally, delimiter variations in repo names (e.g., underscores_vs hyphens-or dots.) caused heuristic auto-detection to fail on common names likeFLUX.2_klein_9B.Changes
Frontend (
images-page.tsx&video-page.tsx):"Auto (detect)").FLUX.1,FLUX.2 Klein,FLUX.2 Dev,FLUX.1 Kontext,Qwen-Image,Qwen-Image-Edit,Z-Image,Krea 2,Lumina 2,HunyuanImage 2.1,HiDream-I1,Ideogram 4,SDXL.MiniMax-H3,LTX-2,Wan2.2-TI2V-5B,Wan2.2-T2V-A14B,HunyuanVideo-1.5.family_overridethroughLoadAdvanced/VideoLoadAdvanced,currentLoadAdvanced,loadDiffusionModel/loadVideoModel, and the download staging preflight (getDiffusionDownloadPlan/getVideoDownloadPlan).LoadedBuildSummaryandformatResolvedValue.Backend Delimiter Flexibility & Aliases (
diffusion_families.py&video_families.py):_token_in_needlewith_token_regexto match delimiter variations (_,.,-,/,\), allowing repo names such asFLUX.2_klein_9B,flux2_klein, andflux-klein-v1to auto-detect without requiring manual overrides."flux-klein","klein").video_families.py.family_overrideinbuild_resolved_recordfor runtime provenance.Tests (
test_diffusion_backend.py):family_overrideresolution.test_detect_family_delimiter_flexibilitycovering underscores, hyphens, and shorthand naming variants across families.Verification
user/FLUX.2_klein_9B->flux.2-kleinuser/flux2_klein->flux.2-kleinuser/flux-klein-v1->flux.2-kleinuser/klein-4b-lora->flux.2-kleinuser/FLUX.2_dev->flux.2-devuser/flux-2-dev->flux.2-devuser/qwen_image_lora->qwen-imageuser/z_image_turbo->z-imageuser/wan2_2_ti2v_5b->wan2.2-ti2v-5bfamily_overrideon unrecognized custom repo names for all supported image and video families.build_resolved_recordprovenance andformatResolvedValuetitle formatting.test_diffusion_backend.py(all passed).