Integrate guarded FC2 cuBLASLt schedule
This commit is contained in:
Daniel Maddern
parent
0298ccd1cb
commit
a29b8960b0
21 changed files with 8879 additions and 60 deletions
|
|
@ -160,7 +160,8 @@ video and audio tensors. Spark enables the path with
|
|||
`SAGE2_BLACKWELL_DESIGN.md`, and
|
||||
`benchmarks/gb10-post-optimization-profile-summary.json`.
|
||||
|
||||
The isolated FC2 cuBLASLt scheduling study is complete. The production
|
||||
The FC2 cuBLASLt scheduling study and guarded canonical-shape integration are
|
||||
complete. The production
|
||||
heuristic's `_stream_k` kernel requests the same `25.664 GB` of operands as the
|
||||
retained public split-K-1 schedule, but its L2 hit rate is only `53.32%` versus
|
||||
`91.10%`; it incurs `9.853 GB` more L2 read misses and spends heavily in
|
||||
|
|
@ -168,8 +169,15 @@ synchronization polling. Algorithm 70, tile 20, stages 37, split-K 1 is
|
|||
byte-exact with zero workspace. It improves complete blocks 0, 24, and 49 by
|
||||
`8.16-8.88%`, the two-step trajectory by `7.50%`, and the canonical 12-step
|
||||
trajectory from `278.201 s` to `255.371 s` (`8.21%`) with exact video and audio
|
||||
latents. This remains a research-retained integration candidate: production
|
||||
dispatch and configuration are unchanged. See
|
||||
latents. The final hardened production method improves 20-round blocks by
|
||||
`7.86-9.36%` and the canonical 12-step trajectory from `286.431 s` to
|
||||
`262.979 s` (`8.19%`), with 600 successful dispatches, zero fallback, and exact
|
||||
latents. `H3_NVFP4_FC2_LT_SPLITK1=1` enables only the validated `M=37,810`
|
||||
descriptor; nearby row counts can differ by two BF16 elements and therefore
|
||||
retain the existing Comfy fallback. The extension and measured runtime ABI are
|
||||
prepared during H3 model loading rather than on the first canonical request.
|
||||
The Spark hot service is intentionally
|
||||
stopped until performance work is complete. See
|
||||
`research/fc2_nvfp4_scheduling/RESULTS.md`.
|
||||
|
||||
The Spark hot runtime was rebuilt and recreated with image
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ RUN python -m pip install --no-cache-dir --no-deps -e . \
|
|||
|
||||
ENV TORCH_EXTENSIONS_DIR=/opt/h3-blackwell-runtime/.torch_extensions
|
||||
RUN TORCH_CUDA_ARCH_LIST=12.1 python -c \
|
||||
"from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension; print(_vortex_scale_extension().__file__)"
|
||||
"from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension; from h3_blackwell_runtime.fc2_lt import _fc2_lt_extension; print(_vortex_scale_extension().__file__, _fc2_lt_extension().__file__)"
|
||||
|
||||
ENV H3_MODEL_PATH=/models/minimax_h3_ref2va_pruned_nvfp4.safetensors
|
||||
ENV PYTHONPATH=/opt/ComfyUI-sol-attn
|
||||
|
|
|
|||
|
|
@ -394,12 +394,18 @@ tensor-pipe activity from `25.01%` to `82.42%`. There is no material global
|
|||
partial-accumulator or output-reduction traffic; the baseline instead spends
|
||||
heavily in Stream-K synchronization polling and loses traversal locality.
|
||||
|
||||
The candidate is byte-exact and improves FC2 p50 from `52.521 ms` to
|
||||
The candidate is byte-exact and improves FC2 p50 from `53.618 ms` to
|
||||
`15.636 ms`. Complete blocks 0, 24, and 49 improve by `8.16-8.88%`; two-step
|
||||
and canonical 12-step trajectories improve by `7.50%` and `8.21%`, with exact
|
||||
video and audio latents. The custom persistent-kernel branch is therefore
|
||||
closed. Production integration remains separate work, so current dispatch is
|
||||
unchanged. See `research/fc2_nvfp4_scheduling/RESULTS.md` and
|
||||
closed. Guarded production integration is complete behind
|
||||
`H3_NVFP4_FC2_LT_SPLITK1=1`. Its direct 20-round block gate improves blocks
|
||||
0/24/49 by `9.36%`, `8.16%`, and `7.86%`; the exact canonical 12-step gate
|
||||
improves by `8.19%`
|
||||
with 600 successful dispatches and zero fallback. Shape testing found two-BF16
|
||||
differences at several nearby row counts, so only `M=37,810` is eligible and
|
||||
all variable shapes retain Comfy. The hot service remains stopped until the
|
||||
performance program is complete. See `research/fc2_nvfp4_scheduling/RESULTS.md` and
|
||||
`benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json`.
|
||||
|
||||
The follow-up real block-24 Sage2 decomposition now selects the next exact
|
||||
|
|
|
|||
2025
benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json
Normal file
2025
benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json
Normal file
File diff suppressed because it is too large
Load diff
1776
benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json
Normal file
1776
benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json
Normal file
File diff suppressed because it is too large
Load diff
1776
benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json
Normal file
1776
benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json
Normal file
File diff suppressed because it is too large
Load diff
2404
benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json
Normal file
2404
benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -20,6 +20,7 @@ services:
|
|||
H3_FUSED_ELEMENTWISE: "1"
|
||||
H3_NVFP4_MODULATE_FUSION: "1"
|
||||
H3_NVFP4_SWIGLU_FUSION: "1"
|
||||
H3_NVFP4_FC2_LT_SPLITK1: "1"
|
||||
H3_SAGE_QKV_LAYOUT: "strided_nhd"
|
||||
H3_SOL_QKV_LAYOUT: "native"
|
||||
command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--warmup"]
|
||||
|
|
|
|||
|
|
@ -37,6 +37,7 @@ services:
|
|||
H3_FUSED_ELEMENTWISE: "1"
|
||||
H3_NVFP4_MODULATE_FUSION: "1"
|
||||
H3_NVFP4_SWIGLU_FUSION: "1"
|
||||
H3_NVFP4_FC2_LT_SPLITK1: "1"
|
||||
H3_SAGE_QKV_LAYOUT: "strided_nhd"
|
||||
H3_SOL_QKV_LAYOUT: "native"
|
||||
command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--turbo-lora", "4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors", "--turbo-lora", "8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors", "--latent-upscaler", "/latent-upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors", "--warmup"]
|
||||
|
|
|
|||
|
|
@ -17,3 +17,6 @@ flash4 = ["flash-attn-4[cu13]==4.0.0b27"]
|
|||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
h3_blackwell_runtime = ["csrc/*.cpp", "csrc/*.cu"]
|
||||
|
|
|
|||
|
|
@ -2,36 +2,36 @@
|
|||
"metadata": {
|
||||
"algorithm": "SHA-256",
|
||||
"generated_date": "2026-08-25",
|
||||
"scope_note": "Local selected scope is the established 167-file set plus 10 retained FC2 scheduling artifacts. Spark records are the complete reproducible current top-level benchmark output set; the audit-retained 279-file/665950155-byte aggregate cannot be reconstructed because its path list was not retained.",
|
||||
"scope_note": "Local selected scope is the established 167-file set plus 14 retained FC2 scheduling and integration artifacts. Spark records are the complete reproducible current top-level benchmark output set; the audit-retained 279-file/665950155-byte aggregate cannot be reconstructed because its path list was not retained.",
|
||||
"summary": {
|
||||
"local": {
|
||||
"record_count": 177,
|
||||
"size_bytes": 230157548,
|
||||
"expected_record_count": 177,
|
||||
"expected_size_bytes": 230157548,
|
||||
"record_count": 181,
|
||||
"size_bytes": 230398680,
|
||||
"expected_record_count": 181,
|
||||
"expected_size_bytes": 230398680,
|
||||
"reconciled": true
|
||||
},
|
||||
"spark": {
|
||||
"record_count": 290,
|
||||
"size_bytes": 567477443,
|
||||
"record_count": 294,
|
||||
"size_bytes": 567718575,
|
||||
"expected_record_count": 279,
|
||||
"expected_size_bytes": 665950155,
|
||||
"reconciled": false,
|
||||
"record_count_delta": 11,
|
||||
"size_bytes_delta": -98472712
|
||||
"record_count_delta": 15,
|
||||
"size_bytes_delta": -98231580
|
||||
},
|
||||
"total": {
|
||||
"record_count": 467,
|
||||
"size_bytes": 797634991
|
||||
"record_count": 475,
|
||||
"size_bytes": 798117255
|
||||
}
|
||||
},
|
||||
"json_reconciliation": {
|
||||
"identical": 112,
|
||||
"mismatches": 2,
|
||||
"identical": 118,
|
||||
"mismatches": 0,
|
||||
"local_only": 40,
|
||||
"spark_only": 107,
|
||||
"local_total": 154,
|
||||
"spark_total": 221
|
||||
"local_total": 158,
|
||||
"spark_total": 225
|
||||
}
|
||||
},
|
||||
"artifacts": [
|
||||
|
|
@ -195,6 +195,70 @@
|
|||
"artifact_class": "benchmark_json",
|
||||
"location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "local",
|
||||
"path": "benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json",
|
||||
"size_bytes": 61575,
|
||||
"sha256": "a49664031af6c21a341cfb0787000e54801c4067cf8ad231cc614c11de251067",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "local",
|
||||
"path": "benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json",
|
||||
"size_bytes": 53189,
|
||||
"sha256": "3e0e42addf5a748462f1bc600d5bb08875b31da5177d2af2b2abddfb1078ebd3",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "local",
|
||||
"path": "benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json",
|
||||
"size_bytes": 53192,
|
||||
"sha256": "94a915acee1bf3d3c23a934f93bafc583ac2ae0f3e91ff1277ee57f628e2d254",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "local",
|
||||
"path": "benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json",
|
||||
"size_bytes": 73176,
|
||||
"sha256": "cbde2aab33ceb9df32e65c8649838dcdf47e0e135aaaae4165b2f9b6e8d2734b",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "spark",
|
||||
"path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json",
|
||||
"size_bytes": 61575,
|
||||
"sha256": "a49664031af6c21a341cfb0787000e54801c4067cf8ad231cc614c11de251067",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "spark",
|
||||
"path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json",
|
||||
"size_bytes": 53189,
|
||||
"sha256": "3e0e42addf5a748462f1bc600d5bb08875b31da5177d2af2b2abddfb1078ebd3",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "spark",
|
||||
"path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json",
|
||||
"size_bytes": 53192,
|
||||
"sha256": "94a915acee1bf3d3c23a934f93bafc583ac2ae0f3e91ff1277ee57f628e2d254",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "spark",
|
||||
"path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json",
|
||||
"size_bytes": 73176,
|
||||
"sha256": "cbde2aab33ceb9df32e65c8649838dcdf47e0e135aaaae4165b2f9b6e8d2734b",
|
||||
"artifact_class": "benchmark_json",
|
||||
"location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json"
|
||||
},
|
||||
{
|
||||
"scope": "local",
|
||||
"path": "benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json",
|
||||
|
|
|
|||
|
|
@ -504,22 +504,22 @@
|
|||
"id": "fc2-cublaslt-splitk1-schedule",
|
||||
"name": "FC2 cuBLASLt public split-K-1 schedule",
|
||||
"family": "nvfp4-library-scheduling",
|
||||
"status": "research_retained",
|
||||
"status": "production_accepted",
|
||||
"hypothesis": "A documented cuBLASLt schedule can preserve the exact FC2 reduction result while avoiding the production heuristic's traffic and synchronization regression.",
|
||||
"implementation_strategy": "Reproduce the exact Comfy Kitchen descriptors in an isolated extension, enumerate checked cuBLASLt algorithms, and compare one selected public split-K-1 schedule against the accepted FC2 path.",
|
||||
"source_locations": ["research/fc2_nvfp4_scheduling/README.md", "research/fc2_nvfp4_scheduling/RESULTS.md", "research/fc2_nvfp4_scheduling/fc2_nvfp4_lt.cpp", "tools/benchmark_fc2_nvfp4_algorithms.py"],
|
||||
"active_source_location": "research/fc2_nvfp4_scheduling/fc2_nvfp4_lt.cpp",
|
||||
"source_locations": ["research/fc2_nvfp4_scheduling/README.md", "research/fc2_nvfp4_scheduling/RESULTS.md", "src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp", "tools/benchmark_fc2_nvfp4_algorithms.py"],
|
||||
"active_source_location": "src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp",
|
||||
"commit_hash": null,
|
||||
"benchmark_artifacts": [{"path": "benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-block-gate-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-trajectory-2step-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-trajectory-12step-20260825.json", "exists": true}],
|
||||
"benchmark_artifacts": [{"path": "benchmarks/gb10-fc2-nvfp4-library-sweep-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json", "exists": true}],
|
||||
"profiler_artifacts": [{"path": "benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-baseline-20260825.csv", "exists": true}, {"path": "benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv", "exists": true}],
|
||||
"environment": {"gpu": "NVIDIA GB10", "cuda": "13", "driver": null, "pytorch": null, "triton": null, "container_image": "sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83", "commit_context": "isolated extension; production source and dispatch unchanged"},
|
||||
"metrics": {"fc2_p50_ms_baseline": 53.618, "fc2_p50_ms_candidate": 15.636, "block_improvement_pct": [8.16, 8.63, 8.88], "two_step_improvement_pct": 7.50, "canonical_12_step_improvement_pct": 8.21, "trajectory_measurement_note": "single baseline-then-candidate paired runs", "l2_hit_rate_pct_baseline": 53.32, "l2_hit_rate_pct_candidate": 91.10, "l2_read_miss_reduction_bytes": 9853094784},
|
||||
"correctness_evidence": ["Selected FC2 output is byte-exact.", "Blocks 0, 24, and 49 are byte-exact against paired baseline and retained traversal.", "Two-step and canonical 12-step video and audio latents are byte-exact."],
|
||||
"decision_rationale": "The library schedule passed all exactness and performance gates, explaining the prior traffic amplification without requiring a custom kernel. It is retained pending explicit production integration and deployment validation.",
|
||||
"reproducer_commands": ["python tools/benchmark_fc2_nvfp4_algorithms.py --mode sweep --rounds 20 --workspace-bytes 67108864", "python tools/benchmark_fc2_nvfp4_algorithms.py --mode block-gate --candidate research/fc2_nvfp4_scheduling/candidate_splitk1.json --rounds 20 --workspace-bytes 0"],
|
||||
"environment": {"gpu": "NVIDIA GB10", "cuda": "13", "driver": null, "pytorch": "2.9.1+cu130", "triton": null, "container_image": "sha256:d61c8d29e8b59306d140bfd279b0de31d827ad13bd92447ea72ea0a04182494d", "commit_context": "guarded canonical production integration"},
|
||||
"metrics": {"fc2_p50_ms_baseline": 53.618, "fc2_p50_ms_candidate": 15.636, "production_block_improvement_pct": [9.36, 8.16, 7.86], "production_two_step_seconds": {"baseline": 47.30056222799976, "candidate": 43.727866895999796, "improvement_percent": 7.553177306389591}, "production_canonical_12_step_seconds": {"baseline": 286.43065266899976, "candidate": 262.97871506399997, "improvement_percent": 8.187649396624085}, "production_12_step_dispatches": 600, "production_12_step_fallbacks": 0, "l2_hit_rate_pct_baseline": 53.32, "l2_hit_rate_pct_candidate": 91.10, "l2_read_miss_reduction_bytes": 9853094784},
|
||||
"correctness_evidence": ["Production blocks 0, 24, and 49 are byte-exact against paired baseline and retained traversal.", "Production two-step and canonical 12-step video and audio latents are byte-exact.", "Nearby M=37711/37725/37742 shape probes differ in two BF16 elements and are excluded by the canonical guard."],
|
||||
"decision_rationale": "The shape-guarded library schedule passed production-method block and trajectory gates without requiring a custom kernel. Noncanonical shapes retain the accepted fallback because AlgoCheck alone does not guarantee byte exactness.",
|
||||
"reproducer_commands": ["python tools/benchmark_fc2_nvfp4_algorithms.py --mode production-block-gate --candidate research/fc2_nvfp4_scheduling/candidate_splitk1.json --rounds 20 --workspace-bytes 0", "python tools/benchmark_fc2_nvfp4_algorithms.py --mode production-trajectory --candidate research/fc2_nvfp4_scheduling/candidate_splitk1.json --steps 12 --workspace-bytes 0", "python tools/benchmark_fc2_nvfp4_algorithms.py --mode shape-gate --candidate research/fc2_nvfp4_scheduling/candidate_splitk1.json --workspace-bytes 0"],
|
||||
"timestamp": "2026-08-25",
|
||||
"evidence_missing": ["Production Nvfp4Linear integration", "Production deployment smoke", "Portable validation outside GB10/SM121"],
|
||||
"production_behavior": "Not dispatched. Production still uses Comfy Kitchen's heuristic-selected FC2 path.",
|
||||
"evidence_missing": ["Resident-service deployment validation deferred until performance work completes", "Trajectory timings are single baseline-then-candidate pairs rather than repeated AB/BA trials", "Shape-gate artifact predates runtime-version fields in build_info", "Portable validation outside GB10/SM121"],
|
||||
"production_behavior": "H3_NVFP4_FC2_LT_SPLITK1 selects algorithm 70 only for the validated canonical FC2 descriptor; every other shape, device, library, LoRA, and failure case falls back to Comfy Kitchen.",
|
||||
"source_recovery": "The direct cuBLASLt extension, benchmark harness, selected candidate, raw NCU reports, and full gate artifacts are retained in this checkout."
|
||||
},
|
||||
{
|
||||
|
|
|
|||
|
|
@ -1,14 +1,15 @@
|
|||
# FC2 NVFP4 Library Scheduling
|
||||
|
||||
This directory contains the first isolated library-scheduling study for H3 FC2.
|
||||
It does not register a PyTorch operator, alter `Nvfp4Linear`, or participate in
|
||||
production dispatch. The extension is loaded only by
|
||||
`tools/benchmark_fc2_nvfp4_algorithms.py`.
|
||||
The validated extension source moved to
|
||||
`src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp` when the guarded production
|
||||
integration was added. The research harness and production loader share that
|
||||
source.
|
||||
|
||||
The completed measurements and decision are in `RESULTS.md`. Algorithm 70 with
|
||||
public split-K 1 is byte-exact and passes the block, two-step, and canonical
|
||||
12-step gates, but remains research-only until production integration and
|
||||
deployment validation are performed.
|
||||
12-step gates. Canonical-only production integration is complete behind an
|
||||
opt-in flag; resident-service deployment validation remains deferred.
|
||||
|
||||
## Exact Operation
|
||||
|
||||
|
|
@ -175,6 +176,10 @@ latents; see `RESULTS.md` and the linked benchmark artifacts.
|
|||
|
||||
## Prohibited Experiments
|
||||
|
||||
These restrictions governed the isolated study. Production integration was
|
||||
authorized only after the library candidate passed the complete validation
|
||||
ladder; the canonical-only guard and automatic fallback preserve that boundary.
|
||||
|
||||
- Do not alter production source, configuration, dispatch, or existing
|
||||
research files.
|
||||
- Do not expose this extension through `Nvfp4Linear` or any production operator.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
# FC2 NVFP4 Scheduling Results
|
||||
|
||||
Status: research retained; validated as a canonical-shape production-integration
|
||||
candidate, but not wired into production dispatch.
|
||||
Status: production accepted for the canonical descriptor shape behind
|
||||
`H3_NVFP4_FC2_LT_SPLITK1`; all other shapes retain the existing fallback.
|
||||
|
||||
## Decision
|
||||
|
||||
|
|
@ -70,13 +70,52 @@ traversal.
|
|||
| Canonical 12-step | `278.201 s` | `255.371 s` | `8.21%` | video/audio BF16 exact |
|
||||
|
||||
Trajectory values are single paired runs in baseline-then-candidate
|
||||
order, not repeated medians. Their approximately `38 ms` per-FC2 savings agree
|
||||
order, not repeated medians or AB/BA trials. Their approximately `38 ms` per-FC2 savings agree
|
||||
with the alternating isolated and complete-block gates, but the precise
|
||||
end-to-end percentages retain run-order uncertainty.
|
||||
|
||||
The 12-step saving of `22.829 s` matches approximately 600 FC2 invocations
|
||||
multiplied by the isolated roughly `38 ms` saving.
|
||||
|
||||
## Production Integration
|
||||
|
||||
The packaged path lives in `src/h3_blackwell_runtime/fc2_lt.py` and is called
|
||||
from `Nvfp4Linear.forward_swiglu` only after the accepted native producer. It
|
||||
requires the FC2 role, unsharded `K=14,336/N=5,376`, BF16 output, no bias or
|
||||
active LoRA, GB10/SM121, CUDA 13.0, cuBLAS 13.1 headers, cuBLASLt 13.0 runtime,
|
||||
logical `M=37,810`, and packed
|
||||
`M=37,824`. A shape-specific `cublasLtMatmulAlgoCheck` must pass with zero
|
||||
workspace. Load, check, or execution failure falls back through the already
|
||||
produced NVFP4 bytes without quantizing twice; strict mode is available through
|
||||
`H3_NVFP4_FC2_LT_STRICT=1`. Production startup prepares the extension and
|
||||
caches its measured runtime compatibility result before inference.
|
||||
|
||||
The 20-round production-method gate exercised `Nvfp4Linear.forward_swiglu`
|
||||
directly and recorded 23 successful dispatches with zero fallbacks per block:
|
||||
|
||||
| Block | Baseline p50 | Production p50 | Improvement |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 0 | `464.153 ms` | `420.714 ms` | `9.36%` |
|
||||
| 24 | `460.184 ms` | `422.656 ms` | `8.16%` |
|
||||
| 49 | `456.183 ms` | `420.335 ms` | `7.86%` |
|
||||
|
||||
Production-method trajectories remained exact:
|
||||
|
||||
| Trajectory | Baseline | Production | Improvement | Dispatch |
|
||||
| --- | ---: | ---: | ---: | --- |
|
||||
| Two-step | `47.301 s` | `43.728 s` | `7.55%` | `100/100`, zero fallback |
|
||||
| Canonical 12-step | `286.431 s` | `262.979 s` | `8.19%` | `600/600`, zero fallback |
|
||||
|
||||
The row-shape gate prevents unsafe generalization. Algorithm 70 was exact for
|
||||
the tested `M=37,774`, `37,809`, `37,810`, `37,838`, `37,966`, and `38,222`
|
||||
inputs, but differed in two BF16 elements at each of `M=37,711`, `37,725`, and
|
||||
`37,742` despite passing `AlgoCheck`. Production therefore intentionally keeps
|
||||
the canonical-only guard rather than inferring exactness from nearby shapes.
|
||||
The retained shape artifact records compile-time CUDA/cuBLAS versions but
|
||||
predates runtime-version fields in `build_info()`. The final extension separately
|
||||
measured CUDA runtime `13000` and cuBLASLt runtime `130000`; the shape artifact
|
||||
alone is not runtime-ABI evidence.
|
||||
|
||||
## Traffic Attribution
|
||||
|
||||
The one-pass distinct-data footprint is:
|
||||
|
|
@ -150,9 +189,12 @@ synchronization waiting, not occupancy or a reduction workspace.
|
|||
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.csv`
|
||||
- `benchmarks/gb10-fc2-nvfp4-baseline-20260825.ncu-rep`
|
||||
- `benchmarks/gb10-fc2-nvfp4-splitk1-20260825.ncu-rep`
|
||||
- `benchmarks/gb10-fc2-nvfp4-production-block-gate-20260825.json`
|
||||
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-2step-20260825.json`
|
||||
- `benchmarks/gb10-fc2-nvfp4-production-trajectory-12step-20260825.json`
|
||||
- `benchmarks/gb10-fc2-nvfp4-shape-gate-20260825.json`
|
||||
- `src/h3_blackwell_runtime/csrc/fc2_nvfp4_lt.cpp`
|
||||
|
||||
Production source and configuration were not changed by this study. The
|
||||
candidate is validated only for the canonical descriptor shape and current
|
||||
GB10/CUDA-library combination. Promotion still requires shape-specific
|
||||
`AlgoCheck` with a safe fallback, integration behind the existing
|
||||
`Nvfp4Linear` boundary, broader shape tests, and deployment validation.
|
||||
The Spark hot runtime remains stopped while performance work continues.
|
||||
Resident-service validation is intentionally deferred until the performance
|
||||
program is finished.
|
||||
|
|
|
|||
|
|
@ -23,13 +23,22 @@ namespace py = pybind11;
|
|||
|
||||
namespace {
|
||||
|
||||
thread_local cublasLtHandle_t handle = nullptr;
|
||||
struct LtHandle {
|
||||
cublasLtHandle_t value = nullptr;
|
||||
|
||||
LtHandle() { LT_CHECK(cublasLtCreate(&value)); }
|
||||
~LtHandle() {
|
||||
if (value) cublasLtDestroy(value);
|
||||
}
|
||||
};
|
||||
|
||||
cublasLtHandle_t get_handle() {
|
||||
if (!handle) LT_CHECK(cublasLtCreate(&handle));
|
||||
return handle;
|
||||
static LtHandle handle;
|
||||
return handle.value;
|
||||
}
|
||||
|
||||
void prepare() { get_handle(); }
|
||||
|
||||
void require_cuda_contiguous(const torch::Tensor& tensor, const char* name) {
|
||||
TORCH_CHECK(tensor.is_cuda(), name, " must be CUDA");
|
||||
TORCH_CHECK(tensor.is_contiguous(), name, " must be contiguous");
|
||||
|
|
@ -50,6 +59,19 @@ struct Problem {
|
|||
const torch::Tensor& activation_block_scale,
|
||||
const torch::Tensor& weight_qdata,
|
||||
const torch::Tensor& weight_block_scale) {
|
||||
try {
|
||||
initialize(activation_qdata, activation_block_scale, weight_qdata,
|
||||
weight_block_scale);
|
||||
} catch (...) {
|
||||
reset();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
void initialize(const torch::Tensor& activation_qdata,
|
||||
const torch::Tensor& activation_block_scale,
|
||||
const torch::Tensor& weight_qdata,
|
||||
const torch::Tensor& weight_block_scale) {
|
||||
require_cuda_contiguous(activation_qdata, "activation_qdata");
|
||||
require_cuda_contiguous(activation_block_scale, "activation_block_scale");
|
||||
require_cuda_contiguous(weight_qdata, "weight_qdata");
|
||||
|
|
@ -122,13 +144,17 @@ struct Problem {
|
|||
LT_CHECK(cublasLtMatrixLayoutCreate(&d, CUDA_R_16BF, m, n, m));
|
||||
}
|
||||
|
||||
~Problem() {
|
||||
void reset() noexcept {
|
||||
if (d) cublasLtMatrixLayoutDestroy(d);
|
||||
if (c) cublasLtMatrixLayoutDestroy(c);
|
||||
if (b) cublasLtMatrixLayoutDestroy(b);
|
||||
if (a) cublasLtMatrixLayoutDestroy(a);
|
||||
if (operation) cublasLtMatmulDescDestroy(operation);
|
||||
d = c = b = a = nullptr;
|
||||
operation = nullptr;
|
||||
}
|
||||
|
||||
~Problem() { reset(); }
|
||||
};
|
||||
|
||||
template <typename T>
|
||||
|
|
@ -367,8 +393,15 @@ void run(torch::Tensor activation_qdata,
|
|||
|
||||
py::dict build_info() {
|
||||
py::dict result;
|
||||
int cuda_runtime_version = 0;
|
||||
const cudaError_t cuda_status = cudaRuntimeGetVersion(&cuda_runtime_version);
|
||||
TORCH_CHECK(cuda_status == cudaSuccess, "cudaRuntimeGetVersion failed: ",
|
||||
cudaGetErrorString(cuda_status));
|
||||
result["cuda_version"] = CUDA_VERSION;
|
||||
result["cublas_version"] = CUBLAS_VERSION;
|
||||
result["cuda_runtime_version"] = cuda_runtime_version;
|
||||
result["cublaslt_runtime_version"] =
|
||||
static_cast<int64_t>(cublasLtGetVersion());
|
||||
result["stream_k_public_control"] = false;
|
||||
result["stream_k_note"] =
|
||||
"CUDA 13 cuBLASLt exposes no documented MatmulAlgoConfig attribute "
|
||||
|
|
@ -381,6 +414,7 @@ py::dict build_info() {
|
|||
} // namespace
|
||||
|
||||
PYBIND11_MODULE(TORCH_EXTENSION_NAME, module) {
|
||||
module.def("prepare", &prepare);
|
||||
module.def("enumerate", &enumerate);
|
||||
module.def("check", &check);
|
||||
module.def("run", &run);
|
||||
197
src/h3_blackwell_runtime/fc2_lt.py
Normal file
197
src/h3_blackwell_runtime/fc2_lt.py
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
"""Guarded production dispatch for the canonical GB10 FC2 cuBLASLt schedule."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from functools import lru_cache
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
|
||||
|
||||
_TRUTHY = {"1", "true", "yes", "on"}
|
||||
_CANONICAL_LOGICAL_SHAPE = (37_810, 14_336, 5_376)
|
||||
_CANONICAL_PACKED_ROWS = 37_824
|
||||
_CONFIG = {
|
||||
"algorithm_id": 70,
|
||||
"tile_id": 20,
|
||||
"stages_id": 37,
|
||||
"split_k": 1,
|
||||
"reduction_scheme": 0,
|
||||
"custom_option": 0,
|
||||
"cta_swizzle": 0,
|
||||
}
|
||||
_VALIDATED: set[tuple[int, tuple[int, ...], tuple[int, ...]]] = set()
|
||||
_STATS = {"attempts": 0, "successes": 0, "fallbacks": 0}
|
||||
|
||||
|
||||
class _Fc2LtStrictError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def fc2_lt_enabled() -> bool:
|
||||
return os.getenv("H3_NVFP4_FC2_LT_SPLITK1", "").lower() in _TRUTHY
|
||||
|
||||
|
||||
def _fc2_lt_strict() -> bool:
|
||||
return os.getenv("H3_NVFP4_FC2_LT_STRICT", "").lower() in _TRUTHY
|
||||
|
||||
|
||||
def fc2_lt_status() -> dict:
|
||||
return {"enabled": fc2_lt_enabled(), **_STATS}
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _fc2_lt_extension():
|
||||
from torch.utils.cpp_extension import CUDA_HOME, load
|
||||
|
||||
if CUDA_HOME is None:
|
||||
raise RuntimeError("CUDA_HOME is unavailable")
|
||||
source = Path(__file__).resolve().parent / "csrc" / "fc2_nvfp4_lt.cpp"
|
||||
return load(
|
||||
name="h3_fc2_nvfp4_lt",
|
||||
sources=[str(source)],
|
||||
extra_include_paths=[str(Path(CUDA_HOME) / "include")],
|
||||
extra_cflags=["-O2", "-std=c++17"],
|
||||
extra_ldflags=[
|
||||
"-L" + str(Path(CUDA_HOME) / "lib64"),
|
||||
"-lcublasLt",
|
||||
"-lcublas",
|
||||
"-lcudart",
|
||||
],
|
||||
verbose=os.getenv("H3_NVFP4_FC2_LT_VERBOSE", "").lower() in _TRUTHY,
|
||||
)
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _fc2_lt_extension_result():
|
||||
try:
|
||||
return _fc2_lt_extension(), None
|
||||
except Exception as error:
|
||||
return None, error
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def _fc2_lt_compatible_result():
|
||||
extension, load_error = _fc2_lt_extension_result()
|
||||
if extension is None:
|
||||
return None, load_error
|
||||
try:
|
||||
info = dict(extension.build_info())
|
||||
compatible = (
|
||||
info.get("cuda_version") == 13_000
|
||||
and info.get("cublas_version") == 130_100
|
||||
and info.get("cuda_runtime_version") == 13_000
|
||||
and info.get("cublaslt_runtime_version") == 130_000
|
||||
)
|
||||
if not compatible:
|
||||
return None, RuntimeError(f"unsupported FC2 cuBLASLt environment: {info}")
|
||||
extension.prepare()
|
||||
return extension, None
|
||||
except Exception as error:
|
||||
return None, error
|
||||
|
||||
|
||||
def _canonical_fc2_supported(linear, gate_up: torch.Tensor, qdata: torch.Tensor) -> bool:
|
||||
logical_rows, in_features, out_features = _CANONICAL_LOGICAL_SHAPE
|
||||
return (
|
||||
linear.role == "h3_mlp_fc2"
|
||||
and linear.in_features == in_features
|
||||
and linear.out_features == out_features
|
||||
and linear.output_dtype == torch.bfloat16
|
||||
and linear.bias is None
|
||||
and tuple(gate_up.shape) == (logical_rows, in_features * 2)
|
||||
and tuple(qdata.shape) == (_CANONICAL_PACKED_ROWS, in_features // 2)
|
||||
and torch.cuda.get_device_capability(gate_up.device) == (12, 1)
|
||||
)
|
||||
|
||||
|
||||
def _fallback_or_raise(message: str, error: Exception | None = None):
|
||||
_STATS["fallbacks"] += 1
|
||||
if _fc2_lt_strict():
|
||||
if error is None:
|
||||
raise _Fc2LtStrictError(message)
|
||||
raise _Fc2LtStrictError(message) from error
|
||||
return None
|
||||
|
||||
|
||||
def prepare_fc2_lt() -> bool:
|
||||
"""Load and validate the extension before latency-sensitive inference."""
|
||||
if not fc2_lt_enabled():
|
||||
return False
|
||||
extension, error = _fc2_lt_compatible_result()
|
||||
if extension is None:
|
||||
_fallback_or_raise("FC2 cuBLASLt extension is incompatible", error)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def fc2_lt_linear(
|
||||
linear,
|
||||
gate_up: torch.Tensor,
|
||||
tensor_scale: torch.Tensor,
|
||||
qdata: torch.Tensor,
|
||||
block_scale: torch.Tensor,
|
||||
) -> torch.Tensor | None:
|
||||
"""Run the validated canonical schedule, or return ``None`` for fallback."""
|
||||
if not fc2_lt_enabled():
|
||||
return None
|
||||
try:
|
||||
supported = _canonical_fc2_supported(linear, gate_up, qdata)
|
||||
except Exception as error:
|
||||
return _fallback_or_raise("FC2 cuBLASLt eligibility check failed", error)
|
||||
if not supported:
|
||||
_STATS["fallbacks"] += 1
|
||||
return None
|
||||
_STATS["attempts"] += 1
|
||||
|
||||
extension, load_error = _fc2_lt_compatible_result()
|
||||
if extension is None:
|
||||
return _fallback_or_raise(
|
||||
"FC2 cuBLASLt extension failed to load or is incompatible", load_error
|
||||
)
|
||||
|
||||
try:
|
||||
device = gate_up.device.index if gate_up.device.index is not None else torch.cuda.current_device()
|
||||
validation_key = (device, tuple(qdata.shape), tuple(linear.weight.shape))
|
||||
if validation_key not in _VALIDATED:
|
||||
checked = dict(
|
||||
extension.check(
|
||||
qdata,
|
||||
block_scale,
|
||||
linear.weight,
|
||||
linear.weight_scale,
|
||||
_CONFIG,
|
||||
)
|
||||
)
|
||||
if not checked.get("valid") or checked.get("required_workspace_bytes") != 0:
|
||||
return _fallback_or_raise(
|
||||
f"FC2 cuBLASLt algorithm check rejected the candidate: {checked}"
|
||||
)
|
||||
_VALIDATED.add(validation_key)
|
||||
|
||||
alpha = (tensor_scale.float() * linear.weight_scale_2.float()).reshape(1).contiguous()
|
||||
beta = torch.zeros_like(alpha)
|
||||
output = torch.empty(
|
||||
(qdata.shape[0], linear.out_features),
|
||||
device=qdata.device,
|
||||
dtype=torch.bfloat16,
|
||||
)
|
||||
workspace = torch.empty(0, device=qdata.device, dtype=torch.uint8)
|
||||
extension.run(
|
||||
qdata,
|
||||
block_scale,
|
||||
linear.weight,
|
||||
linear.weight_scale,
|
||||
alpha,
|
||||
beta,
|
||||
output,
|
||||
workspace,
|
||||
_CONFIG,
|
||||
)
|
||||
_STATS["successes"] += 1
|
||||
return output[: gate_up.shape[0], : linear.out_features]
|
||||
except _Fc2LtStrictError:
|
||||
raise
|
||||
except Exception as error:
|
||||
return _fallback_or_raise("FC2 cuBLASLt candidate execution failed", error)
|
||||
|
|
@ -164,9 +164,16 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module):
|
|||
if gate_up.dtype != torch.bfloat16 or not gate_up.is_cuda or not gate_up.is_contiguous():
|
||||
raise ValueError("SwiGLU NVFP4 dispatch requires contiguous CUDA BF16 input")
|
||||
|
||||
from .nvfp4_quant import vortex_quantize_swiglu_nvfp4
|
||||
from .nvfp4_quant import vortex_native_quantize_swiglu_nvfp4, wrap_native_swiglu_nvfp4
|
||||
|
||||
packed_x = vortex_quantize_swiglu_nvfp4(gate_up)
|
||||
packed = vortex_native_quantize_swiglu_nvfp4(gate_up)
|
||||
if os.getenv("H3_NVFP4_FC2_LT_SPLITK1", "").lower() in {"1", "true", "yes", "on"}:
|
||||
from .fc2_lt import fc2_lt_linear
|
||||
|
||||
scheduled = fc2_lt_linear(self, gate_up, *packed)
|
||||
if scheduled is not None:
|
||||
return scheduled
|
||||
packed_x = wrap_native_swiglu_nvfp4(gate_up, packed)
|
||||
bias = self.bias.to(gate_up) if self.bias is not None else None
|
||||
output = functional.linear(packed_x, self._packed_weight(), bias)[
|
||||
: gate_up.shape[0], : self.out_features
|
||||
|
|
|
|||
|
|
@ -328,9 +328,16 @@ def vortex_native_quantize_swiglu_nvfp4(tensor: torch.Tensor):
|
|||
|
||||
def vortex_quantize_swiglu_nvfp4(tensor: torch.Tensor):
|
||||
"""Wrap fused SwiGLU output as a Comfy TensorCoreNVFP4Layout tensor."""
|
||||
return wrap_native_swiglu_nvfp4(
|
||||
tensor, vortex_native_quantize_swiglu_nvfp4(tensor),
|
||||
)
|
||||
|
||||
|
||||
def wrap_native_swiglu_nvfp4(tensor: torch.Tensor, packed):
|
||||
"""Wrap already-produced fused SwiGLU bytes for the Comfy fallback."""
|
||||
from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout
|
||||
|
||||
tensor_scale, qdata, block_scale = vortex_native_quantize_swiglu_nvfp4(tensor)
|
||||
tensor_scale, qdata, block_scale = packed
|
||||
output_shape = (tensor.shape[0], tensor.shape[1] // 2)
|
||||
return QuantizedTensor(
|
||||
qdata,
|
||||
|
|
|
|||
|
|
@ -154,9 +154,14 @@ class H3HotRuntime:
|
|||
def _load_h3(self) -> H3PackedDenoiser:
|
||||
model = H3PackedDenoiser.from_checkpoint(self.checkpoint, attention_backend=self.config.attention).eval()
|
||||
configure_mlp_chunking(model, self.config.mlp_chunks, self.config.mlp_chunk_threshold)
|
||||
from .fc2_lt import prepare_fc2_lt
|
||||
|
||||
prepare_fc2_lt()
|
||||
return model
|
||||
|
||||
def status(self) -> dict:
|
||||
from .fc2_lt import fc2_lt_status
|
||||
|
||||
return {
|
||||
"ready": True,
|
||||
"initial_attention": self.config.attention,
|
||||
|
|
@ -168,6 +173,7 @@ class H3HotRuntime:
|
|||
"vae_tile_size": self.config.vae_tile_size,
|
||||
"mlp_chunks": self.config.mlp_chunks,
|
||||
"mlp_chunk_threshold": self.config.mlp_chunk_threshold,
|
||||
"fc2_lt": fc2_lt_status(),
|
||||
"loaded_at": self.loaded_at,
|
||||
"load_stages": self.load_stages,
|
||||
}
|
||||
|
|
|
|||
287
tests/test_fc2_lt.py
Normal file
287
tests/test_fc2_lt.py
Normal file
|
|
@ -0,0 +1,287 @@
|
|||
import os
|
||||
import unittest
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import torch
|
||||
|
||||
from h3_blackwell_runtime.fc2_lt import (
|
||||
_VALIDATED,
|
||||
_canonical_fc2_supported,
|
||||
_fc2_lt_compatible_result,
|
||||
_STATS,
|
||||
fc2_lt_enabled,
|
||||
fc2_lt_linear,
|
||||
prepare_fc2_lt,
|
||||
)
|
||||
from h3_blackwell_runtime.nvfp4 import Nvfp4Linear
|
||||
from h3_blackwell_runtime.runtime import H3HotRuntime
|
||||
|
||||
|
||||
class Fc2LtContracts(unittest.TestCase):
|
||||
def setUp(self):
|
||||
_VALIDATED.clear()
|
||||
_fc2_lt_compatible_result.cache_clear()
|
||||
_STATS.update(attempts=0, successes=0, fallbacks=0)
|
||||
|
||||
def test_flag_is_opt_in(self):
|
||||
with patch.dict(os.environ, {}, clear=True):
|
||||
self.assertFalse(fc2_lt_enabled())
|
||||
with patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "yes"}, clear=True):
|
||||
self.assertTrue(fc2_lt_enabled())
|
||||
|
||||
def test_only_canonical_unsharded_fc2_is_eligible(self):
|
||||
linear = SimpleNamespace(
|
||||
role="h3_mlp_fc2",
|
||||
in_features=14_336,
|
||||
out_features=5_376,
|
||||
output_dtype=torch.bfloat16,
|
||||
bias=None,
|
||||
)
|
||||
gate_up = SimpleNamespace(
|
||||
shape=(37_810, 28_672),
|
||||
device=torch.device("cuda", 0),
|
||||
)
|
||||
qdata = SimpleNamespace(shape=(37_824, 7_168))
|
||||
with patch("torch.cuda.get_device_capability", return_value=(12, 1)):
|
||||
self.assertTrue(_canonical_fc2_supported(linear, gate_up, qdata))
|
||||
linear.in_features //= 2
|
||||
self.assertFalse(_canonical_fc2_supported(linear, gate_up, qdata))
|
||||
|
||||
def test_prepare_loads_and_validates_extension(self):
|
||||
extension = MagicMock()
|
||||
extension.build_info.return_value = {
|
||||
"cuda_version": 13_000,
|
||||
"cublas_version": 130_100,
|
||||
"cuda_runtime_version": 13_000,
|
||||
"cublaslt_runtime_version": 130_000,
|
||||
}
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch(
|
||||
"h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result",
|
||||
return_value=(extension, None),
|
||||
),
|
||||
):
|
||||
self.assertTrue(prepare_fc2_lt())
|
||||
extension.prepare.assert_called_once_with()
|
||||
|
||||
def test_ineligible_shape_is_counted_as_fallback(self):
|
||||
args = (MagicMock(), MagicMock(), MagicMock(), MagicMock(), MagicMock())
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=False),
|
||||
):
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
self.assertEqual(_STATS["fallbacks"], 1)
|
||||
|
||||
def test_h3_model_load_prepares_extension(self):
|
||||
model = MagicMock()
|
||||
model.eval.return_value = model
|
||||
runtime = SimpleNamespace(
|
||||
checkpoint=MagicMock(),
|
||||
config=SimpleNamespace(attention="sage2", mlp_chunks=1, mlp_chunk_threshold=4096),
|
||||
)
|
||||
with (
|
||||
patch(
|
||||
"h3_blackwell_runtime.runtime.H3PackedDenoiser.from_checkpoint",
|
||||
return_value=model,
|
||||
),
|
||||
patch("h3_blackwell_runtime.runtime.configure_mlp_chunking") as configure,
|
||||
patch("h3_blackwell_runtime.fc2_lt.prepare_fc2_lt") as prepare,
|
||||
):
|
||||
self.assertIs(H3HotRuntime._load_h3(runtime), model)
|
||||
configure.assert_called_once_with(model, 1, 4096)
|
||||
prepare.assert_called_once_with()
|
||||
|
||||
def test_extension_load_failure_falls_back_unless_strict(self):
|
||||
args = (MagicMock(), MagicMock(), MagicMock(), MagicMock(), MagicMock())
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(None, RuntimeError("load"))),
|
||||
):
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1", "H3_NVFP4_FC2_LT_STRICT": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(None, RuntimeError("load"))),
|
||||
):
|
||||
with self.assertRaisesRegex(RuntimeError, "failed to load"):
|
||||
fc2_lt_linear(*args)
|
||||
|
||||
def test_build_info_and_runtime_version_failures_fall_back(self):
|
||||
extension = MagicMock()
|
||||
extension.build_info.side_effect = RuntimeError("metadata")
|
||||
args = (MagicMock(), MagicMock(), MagicMock(), MagicMock(), MagicMock())
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(extension, None)),
|
||||
):
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
|
||||
_fc2_lt_compatible_result.cache_clear()
|
||||
extension.build_info.side_effect = None
|
||||
extension.build_info.return_value = {
|
||||
"cuda_version": 13_000,
|
||||
"cublas_version": 130_100,
|
||||
"cuda_runtime_version": 12_900,
|
||||
"cublaslt_runtime_version": 130_000,
|
||||
}
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(extension, None)),
|
||||
):
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
|
||||
def test_algo_check_and_execution_failures_fall_back(self):
|
||||
extension = MagicMock()
|
||||
extension.build_info.return_value = {
|
||||
"cuda_version": 13_000,
|
||||
"cublas_version": 130_100,
|
||||
"cuda_runtime_version": 13_000,
|
||||
"cublaslt_runtime_version": 130_000,
|
||||
}
|
||||
linear = MagicMock()
|
||||
linear.weight.shape = (5_376, 7_168)
|
||||
gate_up = MagicMock()
|
||||
gate_up.device.index = 0
|
||||
qdata = MagicMock()
|
||||
qdata.shape = (37_824, 7_168)
|
||||
args = (linear, gate_up, MagicMock(), qdata, MagicMock())
|
||||
common = (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(extension, None)),
|
||||
)
|
||||
extension.check.return_value = {"valid": False, "required_workspace_bytes": 0}
|
||||
with common[0], common[1], common[2]:
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
|
||||
_VALIDATED.clear()
|
||||
extension.check.return_value = {"valid": True, "required_workspace_bytes": 0}
|
||||
extension.run.side_effect = RuntimeError("run")
|
||||
with (
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result", return_value=(extension, None)),
|
||||
patch("h3_blackwell_runtime.fc2_lt.torch.zeros_like", return_value=MagicMock()),
|
||||
patch("h3_blackwell_runtime.fc2_lt.torch.empty", return_value=MagicMock()),
|
||||
):
|
||||
self.assertIsNone(fc2_lt_linear(*args))
|
||||
|
||||
def test_strict_mode_raises_on_algo_check_and_execution_failures(self):
|
||||
extension = MagicMock()
|
||||
extension.build_info.return_value = {
|
||||
"cuda_version": 13_000,
|
||||
"cublas_version": 130_100,
|
||||
"cuda_runtime_version": 13_000,
|
||||
"cublaslt_runtime_version": 130_000,
|
||||
}
|
||||
linear = MagicMock()
|
||||
linear.weight.shape = (5_376, 7_168)
|
||||
gate_up = MagicMock()
|
||||
gate_up.device.index = 0
|
||||
qdata = MagicMock()
|
||||
qdata.shape = (37_824, 7_168)
|
||||
args = (linear, gate_up, MagicMock(), qdata, MagicMock())
|
||||
environment = {
|
||||
"H3_NVFP4_FC2_LT_SPLITK1": "1",
|
||||
"H3_NVFP4_FC2_LT_STRICT": "1",
|
||||
}
|
||||
extension.check.return_value = {"valid": False, "required_workspace_bytes": 0}
|
||||
with (
|
||||
patch.dict(os.environ, environment, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch(
|
||||
"h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result",
|
||||
return_value=(extension, None),
|
||||
),
|
||||
):
|
||||
with self.assertRaisesRegex(RuntimeError, "algorithm check rejected"):
|
||||
fc2_lt_linear(*args)
|
||||
|
||||
_VALIDATED.clear()
|
||||
_fc2_lt_compatible_result.cache_clear()
|
||||
extension.check.return_value = {"valid": True, "required_workspace_bytes": 0}
|
||||
extension.run.side_effect = RuntimeError("run")
|
||||
with (
|
||||
patch.dict(os.environ, environment, clear=True),
|
||||
patch("h3_blackwell_runtime.fc2_lt._canonical_fc2_supported", return_value=True),
|
||||
patch(
|
||||
"h3_blackwell_runtime.fc2_lt._fc2_lt_extension_result",
|
||||
return_value=(extension, None),
|
||||
),
|
||||
patch("h3_blackwell_runtime.fc2_lt.torch.zeros_like", return_value=MagicMock()),
|
||||
patch("h3_blackwell_runtime.fc2_lt.torch.empty", return_value=MagicMock()),
|
||||
):
|
||||
with self.assertRaisesRegex(RuntimeError, "execution failed"):
|
||||
fc2_lt_linear(*args)
|
||||
|
||||
def test_forward_swiglu_uses_schedule_without_repacking(self):
|
||||
linear = SimpleNamespace(
|
||||
role="h3_mlp_fc2",
|
||||
in_features=4,
|
||||
out_features=3,
|
||||
output_dtype=torch.bfloat16,
|
||||
bias=None,
|
||||
pre_quant_scale=None,
|
||||
active_lora=None,
|
||||
lora_strength=0.0,
|
||||
full_precision_matrix_mult=False,
|
||||
)
|
||||
gate_up = torch.randn(1, 8, dtype=torch.bfloat16)
|
||||
packed = (MagicMock(), MagicMock(), MagicMock())
|
||||
expected = torch.randn(1, 3, dtype=torch.bfloat16)
|
||||
with (
|
||||
torch.inference_mode(),
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch.object(torch.Tensor, "is_cuda", new_callable=unittest.mock.PropertyMock, return_value=True),
|
||||
patch("h3_blackwell_runtime.nvfp4_quant.vortex_native_quantize_swiglu_nvfp4", return_value=packed) as quantize,
|
||||
patch("h3_blackwell_runtime.nvfp4_quant.wrap_native_swiglu_nvfp4") as wrap,
|
||||
patch("h3_blackwell_runtime.fc2_lt.fc2_lt_linear", return_value=expected) as scheduled,
|
||||
):
|
||||
actual = Nvfp4Linear.forward_swiglu(linear, gate_up)
|
||||
self.assertTrue(torch.equal(actual, expected))
|
||||
quantize.assert_called_once_with(gate_up)
|
||||
scheduled.assert_called_once_with(linear, gate_up, *packed)
|
||||
wrap.assert_not_called()
|
||||
|
||||
def test_forward_swiglu_reuses_native_bytes_on_fallback(self):
|
||||
packed_weight = MagicMock()
|
||||
linear = SimpleNamespace(
|
||||
role="h3_mlp_fc2",
|
||||
in_features=4,
|
||||
out_features=3,
|
||||
output_dtype=torch.bfloat16,
|
||||
bias=None,
|
||||
pre_quant_scale=None,
|
||||
active_lora=None,
|
||||
lora_strength=0.0,
|
||||
full_precision_matrix_mult=False,
|
||||
_packed_weight=lambda: packed_weight,
|
||||
)
|
||||
gate_up = torch.randn(1, 8, dtype=torch.bfloat16)
|
||||
packed = (MagicMock(), MagicMock(), MagicMock())
|
||||
wrapped = MagicMock()
|
||||
expected = torch.randn(1, 3, dtype=torch.bfloat16)
|
||||
with (
|
||||
torch.inference_mode(),
|
||||
patch.dict(os.environ, {"H3_NVFP4_FC2_LT_SPLITK1": "1"}, clear=True),
|
||||
patch.object(torch.Tensor, "is_cuda", new_callable=unittest.mock.PropertyMock, return_value=True),
|
||||
patch("h3_blackwell_runtime.nvfp4_quant.vortex_native_quantize_swiglu_nvfp4", return_value=packed) as quantize,
|
||||
patch("h3_blackwell_runtime.nvfp4_quant.wrap_native_swiglu_nvfp4", return_value=wrapped) as wrap,
|
||||
patch("h3_blackwell_runtime.fc2_lt.fc2_lt_linear", return_value=None),
|
||||
patch("h3_blackwell_runtime.nvfp4.functional.linear", return_value=expected) as gemm,
|
||||
):
|
||||
actual = Nvfp4Linear.forward_swiglu(linear, gate_up)
|
||||
self.assertTrue(torch.equal(actual, expected))
|
||||
quantize.assert_called_once_with(gate_up)
|
||||
wrap.assert_called_once_with(gate_up, packed)
|
||||
gemm.assert_called_once_with(wrapped, packed_weight, None)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
|
@ -27,7 +27,7 @@ from h3_blackwell_runtime.t2v import random_av_latents
|
|||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SOURCE = ROOT / "research" / "fc2_nvfp4_scheduling" / "fc2_nvfp4_lt.cpp"
|
||||
SOURCE = ROOT / "src" / "h3_blackwell_runtime" / "csrc" / "fc2_nvfp4_lt.cpp"
|
||||
DEFAULT_BUDGETS = (0, 4 << 20, 8 << 20, 16 << 20, 32 << 20, 64 << 20)
|
||||
DEFAULT_BLOCKS = (0, 24, 49)
|
||||
ENV_PREFIXES = ("H3_", "COMFY_KITCHEN_", "CUDA_", "TORCH_")
|
||||
|
|
@ -35,7 +35,15 @@ ENV_PREFIXES = ("H3_", "COMFY_KITCHEN_", "CUDA_", "TORCH_")
|
|||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--mode", choices=("compile", "characterize", "sweep", "selected", "profile", "block-gate", "trajectory"), default="characterize")
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=(
|
||||
"compile", "characterize", "sweep", "selected", "profile",
|
||||
"block-gate", "trajectory", "production-block-gate",
|
||||
"production-trajectory", "shape-gate",
|
||||
),
|
||||
default="characterize",
|
||||
)
|
||||
parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors")
|
||||
parser.add_argument("--capture", type=Path, help="Directory containing input.pt, or an input .pt file")
|
||||
parser.add_argument("--output", type=Path, default=Path("/output/h3-blackwell-runtime/benchmarks/fc2-nvfp4-scheduling.json"))
|
||||
|
|
@ -48,6 +56,10 @@ def parse_args() -> argparse.Namespace:
|
|||
parser.add_argument("--sampler-step", type=int, default=1)
|
||||
parser.add_argument("--seed", type=int, default=440420)
|
||||
parser.add_argument("--text-tokens", type=int, default=100)
|
||||
parser.add_argument(
|
||||
"--shape-text-tokens", type=int, nargs="+",
|
||||
default=[1, 15, 32, 64, 99, 100, 128, 256, 512],
|
||||
)
|
||||
parser.add_argument("--blocks", type=int, nargs="+", default=list(DEFAULT_BLOCKS))
|
||||
parser.add_argument("--probe-block", type=int, default=24)
|
||||
parser.add_argument("--workspace-budgets", type=int, nargs="+", default=list(DEFAULT_BUDGETS))
|
||||
|
|
@ -553,7 +565,109 @@ def block_gate(extension, model, block_inputs, block_outputs, gate_up, timesteps
|
|||
return rows
|
||||
|
||||
|
||||
def trajectory_gate(extension, checkpoint, model, config, args):
|
||||
def production_block_gate(model, block_inputs, block_outputs, timesteps, rotation, segments, args):
|
||||
from h3_blackwell_runtime.fc2_lt import fc2_lt_status
|
||||
|
||||
flag = "H3_NVFP4_FC2_LT_SPLITK1"
|
||||
previous = os.environ.get(flag)
|
||||
rows = []
|
||||
|
||||
def select(enabled: bool) -> None:
|
||||
if enabled:
|
||||
os.environ[flag] = "1"
|
||||
else:
|
||||
os.environ.pop(flag, None)
|
||||
|
||||
try:
|
||||
for index in DEFAULT_BLOCKS:
|
||||
block = model.backbone.blocks[index]
|
||||
adaln_values = tuple(value.detach().clone() for value in model.backbone.adaln[index](timesteps))
|
||||
|
||||
def baseline():
|
||||
select(False)
|
||||
return block(block_inputs[index].detach().clone(), rotation, *adaln_values, segments)
|
||||
|
||||
def candidate():
|
||||
select(True)
|
||||
return block(block_inputs[index].detach().clone(), rotation, *adaln_values, segments)
|
||||
|
||||
before = fc2_lt_status()
|
||||
timing = benchmark_pair(
|
||||
baseline, candidate, args.rounds, args.warmup,
|
||||
block_outputs[index].shape[0], block_outputs[index].shape[1], 1,
|
||||
report_dense=False,
|
||||
)
|
||||
with torch.inference_mode():
|
||||
baseline_value = baseline().detach().clone()
|
||||
candidate_value = candidate().detach().clone()
|
||||
after = fc2_lt_status()
|
||||
dispatch_delta = {
|
||||
name: after[name] - before[name]
|
||||
for name in ("attempts", "successes", "fallbacks")
|
||||
}
|
||||
row = {
|
||||
"block": index,
|
||||
"production_method": "block.mlp.fc2.forward_swiglu",
|
||||
"candidate_vs_baseline": compare(candidate_value, baseline_value),
|
||||
"baseline_vs_traversal": compare(baseline_value, block_outputs[index]),
|
||||
"candidate_vs_traversal": compare(candidate_value, block_outputs[index]),
|
||||
"timing": timing,
|
||||
"dispatch_delta": dispatch_delta,
|
||||
}
|
||||
if not all(
|
||||
row[name]["bf16_exact"]
|
||||
for name in ("candidate_vs_baseline", "baseline_vs_traversal", "candidate_vs_traversal")
|
||||
):
|
||||
raise RuntimeError(f"production FC2 schedule is not byte-exact in block {index}")
|
||||
if dispatch_delta["successes"] == 0 or dispatch_delta["fallbacks"] != 0:
|
||||
raise RuntimeError(f"production FC2 schedule did not dispatch cleanly in block {index}")
|
||||
rows.append(row)
|
||||
finally:
|
||||
if previous is None:
|
||||
os.environ.pop(flag, None)
|
||||
else:
|
||||
os.environ[flag] = previous
|
||||
return rows
|
||||
|
||||
|
||||
def shape_gate(extension, gate_up, fc2, config, args):
|
||||
base_rows = gate_up.shape[0] - args.text_tokens
|
||||
rows = []
|
||||
for text_tokens in args.shape_text_tokens:
|
||||
logical_rows = base_rows + text_tokens
|
||||
if logical_rows <= gate_up.shape[0]:
|
||||
actual_gate_up = gate_up[:logical_rows].contiguous()
|
||||
else:
|
||||
extra = logical_rows - gate_up.shape[0]
|
||||
actual_gate_up = torch.cat((gate_up, gate_up[:extra]), dim=0).contiguous()
|
||||
packed = packed_boundary(actual_gate_up, fc2)
|
||||
run_candidate, _output, workspace = candidate_runner(
|
||||
extension, packed, fc2, config, args.workspace_bytes,
|
||||
)
|
||||
timing = benchmark_pair(
|
||||
comfy_packed_runner(packed, fc2, logical_rows), run_candidate,
|
||||
args.rounds, args.warmup, logical_rows, fc2.out_features, fc2.in_features,
|
||||
)
|
||||
checked = dict(
|
||||
extension.check(
|
||||
packed[1], packed[2], fc2.weight, fc2.weight_scale, config,
|
||||
)
|
||||
)
|
||||
row = {
|
||||
"text_tokens": text_tokens,
|
||||
"logical_rows": logical_rows,
|
||||
"packed_rows": packed[1].shape[0],
|
||||
"parity": timing["parity"],
|
||||
"timing": timing,
|
||||
"checked": checked,
|
||||
"supplied_workspace_bytes": workspace.numel(),
|
||||
}
|
||||
row["passed"] = row["parity"]["bf16_exact"] and bool(checked.get("valid"))
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def trajectory_gate(extension, checkpoint, model, config, args, *, production_dispatch=False):
|
||||
packer = H3PromptPacker(checkpoint)
|
||||
torch.manual_seed(args.seed)
|
||||
video, audio, aligned_frames = random_av_latents(
|
||||
|
|
@ -583,11 +697,31 @@ def trajectory_gate(extension, checkpoint, model, config, args):
|
|||
return output[:actual_gate_up.shape[0], :fc2.out_features]
|
||||
return types.MethodType(replacement, fc2)
|
||||
|
||||
candidates = [candidate_method(block.mlp.fc2) for block in model.backbone.blocks]
|
||||
candidates = None if production_dispatch else [
|
||||
candidate_method(block.mlp.fc2) for block in model.backbone.blocks
|
||||
]
|
||||
flag = "H3_NVFP4_FC2_LT_SPLITK1"
|
||||
previous_flag = os.environ.get(flag)
|
||||
if production_dispatch:
|
||||
os.environ[flag] = "1"
|
||||
from h3_blackwell_runtime.fc2_lt import prepare_fc2_lt
|
||||
|
||||
if not prepare_fc2_lt():
|
||||
raise RuntimeError("production FC2 extension preparation failed")
|
||||
if previous_flag is None:
|
||||
os.environ.pop(flag, None)
|
||||
else:
|
||||
os.environ[flag] = previous_flag
|
||||
|
||||
def run(candidate: bool):
|
||||
for index, block in enumerate(model.backbone.blocks):
|
||||
block.mlp.fc2.forward_swiglu = candidates[index] if candidate else originals[index]
|
||||
if production_dispatch:
|
||||
if candidate:
|
||||
os.environ[flag] = "1"
|
||||
else:
|
||||
os.environ.pop(flag, None)
|
||||
else:
|
||||
for index, block in enumerate(model.backbone.blocks):
|
||||
block.mlp.fc2.forward_swiglu = candidates[index] if candidate else originals[index]
|
||||
torch.cuda.synchronize()
|
||||
started = time.perf_counter()
|
||||
result = sample_video_res_multistep(
|
||||
|
|
@ -607,10 +741,19 @@ def trajectory_gate(extension, checkpoint, model, config, args):
|
|||
try:
|
||||
with torch.inference_mode():
|
||||
(reference_video, reference_audio), baseline_seconds = run(False)
|
||||
if production_dispatch:
|
||||
from h3_blackwell_runtime.fc2_lt import fc2_lt_status
|
||||
dispatch_before = fc2_lt_status()
|
||||
(candidate_video, candidate_audio), candidate_seconds = run(True)
|
||||
if production_dispatch:
|
||||
dispatch_after = fc2_lt_status()
|
||||
finally:
|
||||
for block, original in zip(model.backbone.blocks, originals, strict=True):
|
||||
block.mlp.fc2.forward_swiglu = original
|
||||
if previous_flag is None:
|
||||
os.environ.pop(flag, None)
|
||||
else:
|
||||
os.environ[flag] = previous_flag
|
||||
|
||||
video_parity = compare(candidate_video, reference_video)
|
||||
audio_parity = compare(candidate_audio, reference_audio)
|
||||
|
|
@ -631,6 +774,17 @@ def trajectory_gate(extension, checkpoint, model, config, args):
|
|||
"accepted_swiglu_producer_preserved": True,
|
||||
"accepted_gate_and_residual_path_preserved": True,
|
||||
}
|
||||
if production_dispatch:
|
||||
result["production_dispatch"] = True
|
||||
result["dispatch_delta"] = {
|
||||
name: dispatch_after[name] - dispatch_before[name]
|
||||
for name in ("attempts", "successes", "fallbacks")
|
||||
}
|
||||
if result["dispatch_delta"]["successes"] == 0 or result["dispatch_delta"]["fallbacks"] != 0:
|
||||
raise RuntimeError(
|
||||
f"production FC2 schedule did not dispatch cleanly in trajectory: "
|
||||
f"{result['dispatch_delta']}"
|
||||
)
|
||||
if not result["bf16_exact"]:
|
||||
raise RuntimeError("FC2 library candidate trajectory is not byte-exact")
|
||||
return result
|
||||
|
|
@ -684,7 +838,11 @@ def main() -> None:
|
|||
"explicit_split_k_checks": explicit,
|
||||
}
|
||||
|
||||
if args.mode in {"characterize", "sweep", "selected", "profile", "block-gate", "trajectory"}:
|
||||
if args.mode in {
|
||||
"characterize", "sweep", "selected", "profile", "block-gate",
|
||||
"trajectory", "production-block-gate", "production-trajectory",
|
||||
"shape-gate",
|
||||
}:
|
||||
config = resolve_candidate(args, enumerated, explicit)
|
||||
result["selected"] = config
|
||||
if args.mode == "characterize":
|
||||
|
|
@ -712,8 +870,20 @@ def main() -> None:
|
|||
result["profile"] = profile_one(extension, packed, fc2, config, gate_ups[probe].shape[0], args)
|
||||
elif args.mode == "block-gate":
|
||||
result["block_gate"] = block_gate(extension, model, block_inputs, block_outputs, gate_ups, timesteps, rotation, segments, config, args)
|
||||
elif args.mode == "production-block-gate":
|
||||
result["production_block_gate"] = production_block_gate(
|
||||
model, block_inputs, block_outputs, timesteps, rotation, segments, args,
|
||||
)
|
||||
elif args.mode == "shape-gate":
|
||||
result["shape_gate"] = shape_gate(
|
||||
extension, gate_ups[probe], fc2, config, args,
|
||||
)
|
||||
elif args.mode == "trajectory":
|
||||
result["trajectory"] = trajectory_gate(extension, checkpoint, model, config, args)
|
||||
elif args.mode == "production-trajectory":
|
||||
result["trajectory"] = trajectory_gate(
|
||||
extension, checkpoint, model, config, args, production_dispatch=True,
|
||||
)
|
||||
|
||||
result["errors_and_unsupported"] = errors + [{
|
||||
"feature": "Stream-K",
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue