Daniel Maddern
|
807bd64a82
|
Add direct first/last-frame (fl2va) keyframe conditioning
Wire full fl2va into the direct H3 runtime so first/last keyframes flow
through VAE encode -> Qwen vision tokens -> DiT cond segments:
- vae_encoder.py: direct encoder-only H3 video VAE (causal 3D convs,
reflect spatial padding, causal temporal padding, single-frame tap
truncation, tiling, FP32 moments + mean/std normalization).
- qwen3vl_vision.py: Qwen3-VL-32B visual tower (27 blocks, 2D rotary,
deepstack mergers) ported to match the Comfy reference exactly
(head_dim=72, no-bias proj, LayerNorm blocks, split-half apply_rope),
plus Qwen image preprocess, mrope ids/freqs, DiT token tags, keyframe
resize (first=stretch / last=center cover-crop matching Comfy
common_upscale), and build_fl2va_presentation.
- qwen3vl_text.py: split-half apply_rope, _embed_rows/_run_layers,
optional mrope position_ids + DeepStack injection at the first three
decoder layers at visual positions.
- packing.py: H3PromptPacker builds [text | cond | audio | video] with
tag-run text spans, cond rows (first/last cond_t anchors,
VISUAL_COND_TIMESTEP=0.999 noise augmentation via CPU-seeded RNG),
three-timestep row table (t_row*3 + modality_tag), and rope positions.
- runtime.py: load VAE encoder + vision tower; generate() accepts
first_frame/last_frame, builds the fl2va presentation, encodes keyframes,
and passes text_token_tags/cond_latents/frame_count/seed to the sampler.
- sampler.py: thread pack kwargs + seed.
- serve_hot_runtime.py / direct_t2v_preview.py: /generate and
--first-frame/--last-frame accept image paths or base64.
|
2026-08-19 20:17:41 +07:00 |
|