QwenEdit InScene LoRAs (Beta)
Model Description
InScene and InScene Annotate are a pair of LoRA fine-tunes for QwenEdit that enhance its ability to generate images based on scene references. These models work together to provide flexible scene-based image generation with optional annotation support. Both models are currently in beta and will be improved significantly over time.
InScene
The main model that generates images based on scene composition and layout from a reference image. InScene is trained on pairs of different shots within the same scene, along with prompts describing the desired output. Its goal is to create entirely new shots within a scene while maintaining character consistency and scene coherence.
InScene is intentionally biased towards creating completely new shots rather than minor edits. This design choice overcomes Qwen-Image-Edit's internal bias toward making small, conservative edits, enabling more dramatic scene transformations while preserving the characters and overall scene identity.
InScene Annotate
InScene Annotate is trained on images with green rectangles drawn over specific regions. The model learns to generate images showing the subject within that green rectangle area. Rather than simply zooming in precisely on the marked region, it's trained to flexibly interpret instructions to show what's inside that area - capturing the subject, context, and framing in a more natural, composed way rather than a strict crop.
InScene and InScene Annotate are currently in beta.
How to Use
InScene
To use the base InScene model, start your prompt with:
Show a different image in the same scene of:
And then describe what you want to generate.
For example:
Show a different image in the same scene of: a bustling city street at night.
InScene Annotate
To use InScene Annotate:
- Draw a green rectangle over the subject or area of interest in your reference image
- Describe what you want to focus on and how it should change
For example:
Zoom in on the girl, make her turn to the side and laugh
Use with ComfyUI
A ready-to-use ComfyUI workflow is included in this repository:
- workflow.json - Complete ComfyUI workflow with model links and setup instructions
The workflow includes all necessary nodes and links to download the required models (base Qwen-Image-Edit model, VAE, text encoder, and Lightning LoRA).
Use with diffusers
InScene:
import torch
from diffusers import QwenImageEditPipeline
pipe = QwenImageEditPipeline.from_pretrained("Qwen/Qwen-Image-Edit", torch_dtype=torch.bfloat16)
pipe.to("cuda")
pipe.load_lora_weights("peteromallet/Qwen-Image-Edit-InScene", weight_name="InScene-0.7.safetensors")
InScene Annotate:
import torch
from diffusers import QwenImageEditPipeline
pipe = QwenImageEditPipeline.from_pretrained("Qwen/Qwen-Image-Edit", torch_dtype=torch.bfloat16)
pipe.to("cuda")
pipe.load_lora_weights("peteromallet/Qwen-Image-Edit-InScene", weight_name="InScene-Annotate-0.7.safetensors")
Strengths & Weaknesses
The models excel at:
- Capturing scene composition and spatial layout from reference images
- Maintaining consistent scene structure while varying content
- Understanding spatial relationships between elements
- Strong prompt adherence with scene-aware generation
- (Annotate) Precise control using annotated references
The models may struggle with:
- Very complex multi-layered scenes with numerous elements
- Extremely abstract or non-traditional scene compositions
- Fine-grained details that conflict with the reference scene layout
- Occasional depth perception issues
Training Data
The InScene and InScene Annotate LoRAs were trained on curated datasets focusing on scene composition and spatial relationships. InScene uses pairs of different shots within the same scene, while InScene Annotate uses annotated images with green rectangle markers.
The training data will be released publicly when it's in a more stable state.