diff --git a/CURRENT_STATE.md b/CURRENT_STATE.md index 0eda2c2..d91b1fe 100644 --- a/CURRENT_STATE.md +++ b/CURRENT_STATE.md @@ -1,6 +1,6 @@ # H3 Runtime Current State -Status date: 2026-08-22 +Status date: 2026-08-24 This document is the canonical snapshot of implemented scope and remaining work. Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states. @@ -26,6 +26,10 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states. - Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests. - Sequence-sharded 50-block execution and distributed final projection. - True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2. +- Bit-exact fused H3 modulation and residual gates on GB10: `3.0%` faster over + the canonical 12-step 1344x768/124-frame sampling run with identical video + and audio checksums, `3.6-3.9%` faster individual blocks, approximately 390 + MiB lower peak allocated memory, and 32 passing deployed tests. - Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling. - Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames, and 12 steps, including exact video and audio latent equality. @@ -66,7 +70,216 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states. - H3-specific attention backend optimized for real GB10 tensor shapes. - Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs. - CUDA graph capture and shape buckets. -- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path. +- A four-GEMM NVFP4 roofline study for QKV, attention output, FC1, and FC2. + +The active NVFP4 fusion profile is now measured on one canonical GB10 block: +32 of 53 launches belong to the four scale/pack/GEMM paths. Native packed data, +block scales, and linear outputs are exact at every H3 projection width after +generalizing the block-scale swizzle. The next implementation must remove +intermediate traffic across these exact boundaries rather than deploy the +standalone packer, which is not consistently faster than Comfy Kitchen. + +The active subcomponent is direct QKV projection into Sage2's required layout, +followed by direct attention output into the NVFP4 output projection's +token-major layout. The existing `scaled_mm_nvfp4` wrapper exposes only a +contiguous BF16 output, so true copy elimination requires either a supported +strided epilogue from its underlying CUTLASS kernel or an owned projection +collective. A post-GEMM copy kernel is useful only as a diagnostic and does not +satisfy this boundary-removal target. For single-GPU Sage2, the supported +strided NHD interface made that post-GEMM kernel unnecessary: Q/K/V remain +strided views of the interleaved projection output and Sage emits contiguous +token-major-compatible NHD output. The bit-exact path reduces canonical sampling +from `301.05 s` to `290.23 s` and is enabled for Spark deployments with +`H3_SAGE_QKV_LAYOUT=strided_nhd`. + +NVFP4 streaming feasibility is confirmed but not yet deployable. Comfy Kitchen's +cuBLAS interface requires complete activation and scale pointers, while CUTLASS +DSL 4.6.2 runs block-scaled FP4 on SM121 and accepts the same logical H3 data. +The experimental alpha-before-BF16 epilogue is bit-exact for QKV, attention +output, and FC1 on 128-row real tiles. FC2 still differs because its reference +uses a different reduction policy, so no streamed producer is enabled. + +The explicit implementation policy is to retain FC2 on cuBLAS and develop the +streamed CuTe path only for QKV, attention output, and FC1. This is a numerical +fallback, not a silent compatibility path: FC2's reference reduction order is +part of the exactness contract. + +The fixed 128-row P1 producer-consumer checkpoint is complete. The existing +32-thread DMA warp now produces four BF16 activation rows per lane directly into +the owned GEMM's staged E2M1 A and E4M3 SFA shared-memory layouts. B/SFB remain +on TMA, and their completion publishes the stage to the unchanged MMA consumer. +No complete global activation QDATA or SFA tensor is passed to the streamed +kernel. Every real 128-K tile and the complete BF16 GEMM output are bit-exact for +QKV, attention output, and FC1. Matching Comfy requires its `--use_fast_math` +`rcp.approx.ftz.f32` encode-scale operation. FC2 streaming is explicitly +rejected and retains the cuBLAS fallback. The prototype is validator-only and +still needs canonical M/padding support and runtime packaging. +See `benchmarks/gb10-cute-p1-stream-a-summary.json`. + +The first timing gate rejects direct per-CTA streaming. For 128 rows, the exact +streamed kernel is `9.14-12.98x` slower than the complete Vortex-scale plus +Comfy-pack/GEMM reference because every output-N CTA rereads and repacks A. +Measured producer cost is approximately `0.56-0.60 us` per `(N,K)` tile, and +break-even would require reusing A across 28-102 N tiles. Duplicating that many +accumulators is not viable. The active design is now a bounded global packed-tile +ring or persistent work queue that produces each `(M,K)` tile once, shares it +across N consumers, and recycles the slot without materializing the complete +activation. See `benchmarks/gb10-cute-p1-stream-a-timing-summary.json`. + +The bounded-ring implementation uses caller-owned native QDATA/SFA buffers and +an allocation-free `_into` producer. A capacity sweep with one full-activation +scale selected 2048 rows: 6.19 MB for QKV/FC1 and 8.26 MB for attention output. +At that capacity, complete 37,810-row projection parity is exact for QKV, +attention output, and FC1 in blocks 0, 24, and 49, including the final 946-row +chunk. The subsequent single-model alternating gate rejects runtime QKV dispatch: +all three block outputs are bit-exact, but median block time regresses by `0.52%` +to `0.81%`. QKV capacity checks at 3072, 4096, 8192, and 37888 rows also fail to +produce a block-level gain; 4096 is closest at `0.24%` slower. Attention output +and FC1 remain experimental, and FC2 remains on Comfy/cuBLAS. Do not run +trajectory validation or enable the backend until launch fusion or a different +persistent scheduler passes this block gate. See +`benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`. + +The fresh post-optimization canonical run is `288.93 s` with unchanged video +and audio checksums. Block 24 is `468.22 ms` median, of which production NHD +Sage2 attention consumes `258.47 ms`. Internal attribution places `238.81 ms` +in the SM89 attention mainloop, versus `7.68 ms` Q/K quantization and `10.65 ms` +V quantization. The deployed SM121 path therefore still spends most of the +block in an Ada-style MMA kernel. Recompiling SageAttention's Hopper WGMMA +mainloop for SM121 is not possible: CUDA 13 ptxas rejects WGMMA instructions for +`sm_121a`. CUTLASS SM120/121 UMMA supports F8/F6/F4, not the INT8 QK operation +required for exact Sage2 parity, so a native exact attention rewrite is paused. +The next practical boundary was the two approximately 10.5 ms AdaLN modulation +passes. Their exact BF16 values now feed NVFP4 scale/pack for QKV and FC1 without +materializing the modulated values, while retaining the current Comfy GEMMs. +The producer is byte-exact for complete block 0, 24, and 49 inputs. Integrated +block medians improve by `0.28-0.79%`; warmed two-step and canonical 12-step +trajectories improve by `0.52%` and `0.56%`, respectively, with bit-identical +video and audio tensors. Spark enables the path with +`H3_NVFP4_MODULATE_FUSION=1`. See `NVFP4_MODULATE_FUSION_DESIGN.md`, +`SAGE2_BLACKWELL_DESIGN.md`, and +`benchmarks/gb10-post-optimization-profile-summary.json`. + +The Spark hot runtime was rebuilt and recreated with image +`sha256:5f879c43374bcedb89745971d7d95d82afc8fcf9c41d30f11b257c2863b9fe28`. +Health and startup warmup pass with modulation fusion enabled. A resident real +generation smoke completed in `2.14 s` (`0.227 s` sampling) and produced a valid +22-frame 320x192 H.264 file. The first startup warmup includes one-time CUDA +extension compilation. See +`benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json`. + +Exact SwiGLU-to-FC2 NVFP4 producer fusion is also complete. It preserves the +two reference BF16 boundaries, emits byte-identical tensor scale/QDATA/SFA, and +retains the exact Comfy FC2 GEMM. Blocks 0, 24, and 49 improve by `2.14-2.26%`. +The warmed canonical 12-step run improves from `289.14 s` to `277.36 s` +(`4.07%`) with bit-identical video and audio tensors. Spark enables it with +`H3_NVFP4_SWIGLU_FUSION=1`. See `NVFP4_SWIGLU_FUSION_DESIGN.md` and +`benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`. + +Optional BF16 materialization inside both fused producers was tested for active +Turbo LoRA requests. The isolated canonical Turbo-4 trajectory was bit-exact, +but regressed from `131.11 s` to `135.14 s` (`3.08%`), so the prototype was +rejected. Active LoRA retains the exact materialized fallback instead of using +either producer fusion. See +`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`. + +The active Spark image is now +`sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83` +with both accepted NVFP4 producer fusions enabled for eligible base requests. +Health, startup warmup, three deployed CUDA parity tests, and a resident real +generation smoke pass. The final smoke completed in `2.321 s` with `0.217 s` +sampling and produced a valid 22-frame 320x192 H.264 file. See +`benchmarks/gb10-post-lora-revert-deployment-smoke.json`. + +A fresh fully fused Nsight recapture now supersedes the old approximately +515 ms block profile. Block 24 is `458.78 ms` median uninstrumented and +`465.78 ms` across the Nsight GPU span, with 41 kernels and only `0.084 ms` of +inter-kernel idle time. Sage2 is `57.60%` of kernel time, the four NVFP4 GEMMs +are `26.87%`, packing is `8.92%`, norm/RoPE is `4.27%`, and the two remaining +gate/add kernels are `2.35%`. A complete warmed step takes `23.708 s` and shows +the same distribution. Hardware counters attribute `56.25%` of the warm-cache +off-chip request proxy to the NVFP4 GEMMs even though Sage2 remains the time +bottleneck. See `benchmarks/gb10-fully-fused-fresh-nsight-summary.json`. + +The real block-24 Sage2 scheduler study and exact SM89 P0 retune are complete. +Manual preparation plus the +unchanged prequantized SM89 mainloop is byte-exact against public SageAttention +2.2.0. Uninstrumented medians are `2.37 ms` for K mean/smoothing, `3.80 ms` for +Q quantization, `3.83 ms` for K subtract-mean quantization, `5.08 ms` for V +transpose/pad/permute, `5.53 ms` for V scale/FP8 quantization, and `237.09 ms` +for the fused mainloop. Nsight Compute reports 255 registers/thread, 32 KiB +dynamic shared memory/CTA, `16.83%` achieved occupancy, and no eligible warp in +`63.53%` of scheduler cycles. INT8 QK and FP8 PV each use `37.77%` of elapsed +tensor-pipe capacity; combined tensor activity is `75.54%`. The kernel is +scheduler/compute limited rather than off-chip-bandwidth limited: L2 hit rate is +`98.84%`, while fixed-latency dependency and math-pipe stalls dominate. Tail +CTAs add less than 1 ms. + +The P0 mapped the exact register cliff: caps from 255 through 170 registers +remain at two CTAs and `16.67%` theoretical occupancy; only 168 registers reaches +three CTAs and `25%`, while generating `4.95 billion` local spill requests and +worsening no-eligible cycles to `78.79%`. Narrowed scopes reduced static spills +from 44/44 to 12/12 bytes and dynamic spill requests from `1.46 million` to +`0.40 million`, but changed interleaved latency by only `+0.06%` and worsened +no-eligible cycles. In-place score reuse, early K prefetch, and independent +softmax-chain interleaving were also byte-exact and neutral or slower. + +The `7.68%` shared excess maps entirely to repeated V-staging `LDGSTS.128` +instructions. Padding V to a 128-byte shared stride increased shared memory to +40 KiB but left all `626,970,624` excessive wavefronts unchanged and changed +latency by `-0.04%`. No variant crossed the 3% complete-block gate, so none was +integrated or deployed. See +`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json` and the associated +P0 latency JSON and NCU reports. + +The exact Sage2 entry-fusion P1 is also complete and rejected. A single CUDA +kernel fused strided-NHD Q/K RMSNorm, split-half RoPE, and Sage2 Q INT8 +quantization while leaving K mean/quantization, V preparation, and the SM89 +mainloop unchanged. Randomized edge lengths and canonical block 0/24/49 tensors +were bit-exact through prepared Q/K, Q/K quantization, scales, K mean, Sage2 +output, and complete block output. Entry-only median latency improved by +`20.9-23.4%`, but canonical complete-block median improvement was only `0.73%`, +`0.86%`, and `0.53%` for blocks 0, 24, and 49. The candidate removed one launch +(`87` to `86`), did not change peak memory, and reduced complete-block L2 traffic +by only `0.136-0.155%`. It therefore failed the required 1% gate; the opt-in +runtime branch was removed and two-step/12-step validation was skipped. See +`benchmarks/gb10-sage2-p1-entry-fusion-analysis.json` and its referenced parity, +timing, and Nsight reports. + +The exact Sage2 V-preparation P2 is complete and rejected at its isolated gate. +An owned three-stage CUDA path consumes projection-strided NHD BF16 V and emits +Sage2's padded/permuted E4M3 V plus FP32 per-channel scales without materializing +the approximately 517 MiB BF16 transpose tensor. FP8 bytes and scales are exact +for 13 boundary lengths from 1 through 37,810 tokens with 56 heads. On the +canonical shape, median V preparation improves from `10.56 ms` to `6.39 ms` +(`39.48%`), but the `4.17 ms` absolute saving projects to only `0.91%` of the +`458.78 ms` complete block and misses the required `6.0 ms` isolated go gate. +Complete-block and trajectory validation were therefore skipped, and production +dispatch remains unchanged. See +`benchmarks/gb10-sage2-p2-vprep-analysis.json`. + +The exact Sage2 mainloop P3 temporal-pair experiment is also complete and +rejected. Two warp pairs alternated QK/online-softmax and prior-tile PV while +retaining private per-warp scores, softmax state, and output accumulators. The +isolated extension is sanitizer-clean and byte-exact over 13 adversarial short +shapes plus the real 37,810-token block-24 SHA. In a 50-sample alternating run, +mainloop median changed from `245.44 ms` to `245.20 ms`, only `0.10%`, and +missed the absolute `<220 ms` gate. Ptxas reports 254 registers/thread and +32/24-byte static store/load spills versus baseline 255 registers and 24/24-byte +spills. NCU, block integration, and trajectory validation were skipped. +Production remains unchanged. See +`benchmarks/gb10-sage2-p3-temporal-pair-analysis.json` and +`research/sage2_temporal_pair/`. + +The follow-on four-GEMM NVFP4 roofline study selects MLP FC2 as the next exact +kernel target. QKV, attention output, and FC1 sustain `315-321` dense-equivalent +TFLOP/s and sit above the practical GB10 compute/memory ridge point. FC2 instead +uses the stream-K kernel, sustains only `106.02` TFLOP/s, and generates a +`12.16 GB` warm-cache off-chip request proxy, `16.11x` its ideal unique bytes. +Its measured arithmetic intensity is `479` FLOP/byte versus a practical ridge +near `1,177` FLOP/byte. The next experiment should reduce FC2 K-split rereads and +reuse fused SwiGLU producer tiles; the other three GEMMs are not first targets. +See `benchmarks/gb10-four-gemm-nvfp4-roofline.json`. ## Quality Work Remaining @@ -108,13 +321,19 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states. ## Recommended Execution Order -1. Validate full quoted-dialogue video lip-sync and close the audio prompt change. -2. Correct the inactive fused Sol path. -3. Add explicit task schemas and automated single-GPU mode tests. -4. Implement full Ref2VA, including reference-audio encoding. -5. Build the H3-specific attention backend and CUDA graph buckets. -6. Harden the service API and operational deployment. -7. Complete RunPod NCCL validation and distributed scaling benchmarks. +1. Run the four-GEMM NVFP4 roofline study on QKV, attention output, FC1, and FC2. +2. Capture matched SM120 and SM100 component profiles and package Sage2 on SM120. +3. Resume NVFP4 GEMM/epilogue work only with a design that preserves the + accepted producer fusions and exact BF16 boundaries. +4. Validate full quoted-dialogue video lip-sync and close the audio prompt change. +5. Add explicit task schemas and automated single-GPU mode tests. +6. Implement full Ref2VA, including reference-audio encoding. +7. Add CUDA graph buckets after the kernel and shape policies stabilize. +8. Harden the service API and operational deployment. +9. Complete RunPod NCCL validation and distributed scaling benchmarks. + +See `PERFORMANCE_ROADMAP.md` for measured component costs, architecture-specific +targets, quality gates, and the rationale for this ordering. The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still diff --git a/DISTRIBUTED.md b/DISTRIBUTED.md index 35ecbbf..24a943d 100644 --- a/DISTRIBUTED.md +++ b/DISTRIBUTED.md @@ -93,6 +93,46 @@ replace multi-GPU NCCL parity testing. ## RunPod +### Regional volume bootstrap + +Prepare the same benchmark assets in any RunPod data center with one guarded +command. The tool creates or reuses a network volume, attaches it to a temporary +two-vCPU pod, downloads the two pinned Hugging Face artifacts, verifies their +SHA-256 hashes, uploads a `git archive` of the current commit, writes a manifest, +and terminates the CPU pod in a `finally` block. + +Set credentials in the process environment; do not pass secrets on the command +line or commit them: + +```powershell +$env:RUNPOD_API_KEY = "..." +$env:HF_TOKEN = "..." +python .\tools\prepare_runpod_region.py --region US-MO-2 --dry-run +python .\tools\prepare_runpod_region.py --region US-MO-2 ` + --volume-name h3-vortex-us-mo2 --yes +``` + +Resume an interrupted preparation without creating another volume: + +```powershell +python .\tools\prepare_runpod_region.py --region US-MO-2 ` + --volume-id VOLUME_ID --yes +``` + +Matching files are skipped by checksum. Partial `aria2` downloads resume in +place. The resulting volume contains: + +- `ComfyUI/models/diffusion_models/minimax_h3_fl2va_pruned_nvfp4.safetensors` +- `ComfyUI/models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors` +- `h3-runtime/h3-blackwell-runtime-.tar.gz` +- `h3-runtime/regional-bootstrap-manifest.json` +- `h3-runtime/artifacts-ready` + +The archive intentionally contains committed files only. The manifest records a +dirty-worktree flag so an operator can see when local uncommitted changes were +excluded. CPU compute is automatically terminated on success, failure, or +Ctrl+C; the network volume is retained for GPU pods. + The target is one eight-GPU RTX PRO 6000 Blackwell machine. Query current stock using the guarded API v2 client: @@ -149,6 +189,13 @@ and NCCL 2.27.7 selected `P2P/CUMEM` with GDR enabled and zero NVLS channels. RunPod allocations varied between cross-NUMA `SYS` and same-NUMA `NODE` topologies, so topology must be recorded with every scaling result. +One B200 in US-NC-2 completed the same 1344x768 SDPA tensor benchmarks in +`79.72s` at 124 frames and `257.86s` at 243 frames. Relative to one RTX PRO +6000, these are `1.23x` and `1.21x` speedups, but sampling cost is `2.64x` and +`2.69x` higher at the measured `$6.79/hour` B200 rate. The full timings, +checksums, memory peaks, and comparison are in +`benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`. + ## Remaining GPU Gates - Real NCCL identity at 4, 6, and 8 GPUs; two-GPU transport is complete. diff --git a/Dockerfile.spark b/Dockerfile.spark index 28e9c34..904191c 100644 --- a/Dockerfile.spark +++ b/Dockerfile.spark @@ -31,6 +31,10 @@ RUN git clone https://github.com/Saganaki22/ComfyUI-sol-attn.git /opt/ComfyUI-so RUN python -m pip install --no-cache-dir --no-deps -e . \ && python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); assert hasattr(comfy_kitchen, 'int8_attention'); assert hasattr(comfy_kitchen, 'int8_attention_is_available'); print(torch.__version__, torch.version.cuda)" +ENV TORCH_EXTENSIONS_DIR=/opt/h3-blackwell-runtime/.torch_extensions +RUN TORCH_CUDA_ARCH_LIST=12.1 python -c \ + "from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension; print(_vortex_scale_extension().__file__)" + ENV H3_MODEL_PATH=/models/minimax_h3_ref2va_pruned_nvfp4.safetensors ENV PYTHONPATH=/opt/ComfyUI-sol-attn ENV TORCH_COMPILE_DISABLE=0 TORCHDYNAMO_DISABLE=0 diff --git a/NVFP4_MODULATE_FUSION_DESIGN.md b/NVFP4_MODULATE_FUSION_DESIGN.md new file mode 100644 index 0000000..e6424ae --- /dev/null +++ b/NVFP4_MODULATE_FUSION_DESIGN.md @@ -0,0 +1,54 @@ +# NVFP4 Modulation Producer Fusion + +## Target + +Fuse the two per-block AdaLN modulation operations into the exact NVFP4 +activation scale and packing producer for attention QKV and MLP FC1. Retain the +current Comfy Kitchen GEMMs and all BF16 numerical boundaries. + +At the canonical 37,810-token shape, block 24 spends a median 10.47 ms on MSA +modulation and 10.60 ms on MLP modulation. Production already mutates the two +RMSNorm outputs in place, so this fusion targets memory traffic rather than a +separate allocation. + +## Dataflow + +Current: + +1. Read normalized BF16 input and write modulated BF16 output. +2. Read modulated output to compute the global NVFP4 activation scale. +3. Read modulated output again to produce packed E2M1 data and E4M3 scales. +4. Execute the unchanged Comfy NVFP4 GEMM. + +Prototype: + +1. Read normalized input, reproduce the reference modulation BF16 boundary, and + reduce its absolute maximum without materializing the modulated tensor. +2. Read normalized input again, reproduce the same BF16 values, and write exact + QDATA/SFA directly into caller-owned buffers. +3. Execute the unchanged Comfy NVFP4 GEMM. + +## Exactness Contract + +- Preserve segment-specific shift and scale selection. +- Preserve `(x * (scale + 1)) + shift` operation order and BF16 rounding. +- Preserve Vortex's approximate-reciprocal NVFP4 encoding behavior. +- Produce byte-identical QDATA and E4M3 block scales. +- Produce bit-identical QKV/FC1 outputs before changing block dispatch. + +## Gate + +1. Adversarial modulation and quantization tests. +2. Exact packed bytes and linear outputs for blocks 0, 24, and 49. +3. Alternating module-forward benchmark in one loaded block. +4. Full 50-block step and two-step trajectory only after all three blocks gain. +5. Canonical 12-step trajectory before enabling the path by default. + +## Result + +The producer is byte-exact for complete 37,810-row block 0, 24, and 49 inputs. +It takes 6.02 ms for block 24 QKV input. Integrated block medians improve by +0.28-0.79%, a warmed two-step trajectory improves by 0.52%, and the warmed +canonical 12-step run improves from 285.996 s to 284.408 s (0.56%). Video and +audio tensors are bit-identical. Spark deployments enable the path with +`H3_NVFP4_MODULATE_FUSION=1`. diff --git a/NVFP4_STREAMING_DESIGN.md b/NVFP4_STREAMING_DESIGN.md new file mode 100644 index 0000000..d65482c --- /dev/null +++ b/NVFP4_STREAMING_DESIGN.md @@ -0,0 +1,224 @@ +# NVFP4 Streaming Design + +## Decision + +Comfy Kitchen 0.2.31 cannot consume incrementally produced NVFP4 tiles. Its +Python path allocates complete activation QDATA, complete swizzled block scales, +and a complete BF16 output before calling `cublas_gemm_blockwise_fp4` with fixed +full-tensor pointers. The registered operator has no tile callback, producer +interface, output stride, user output, or custom epilogue argument. + +True activation-materialization removal therefore requires an owned CUTLASS or +CuTe DSL mainloop. Wrapping `scaled_mm_nvfp4` cannot provide it. + +## Available Foundation + +The Spark image contains: + +- CUDA 13.0.88. +- NVIDIA CUTLASS DSL 4.6.2. +- CUDA Python bindings. +- Working SM121 block-scaled E2M1/E4M3 MMA support. + +The official CUTLASS 4.6 SM120/SM121 persistent cooperative block-scaled GEMM +was compiled and executed on GB10 with E2M1 A/B, E4M3 scales, FP32 accumulation, +and BF16 output. A 1024x1024x1024 case passed its reference check at `16.9664 us` +and approximately `126.57 TFLOP/s`. + +The public kernel still takes complete global A, B, SFA, and SFB tensors and +uses separate TMA descriptors for all four. CUTLASS does not expose a ready +BF16-to-NVFP4 input producer callback. + +## Numerical Constraint + +H3's activation quantization uses: + +1. One exact global BF16 absolute maximum. +2. One FP32 tensor scale derived from that maximum. +3. One E4M3 block scale per 16 activation values. +4. E2M1 values quantized using both scales. + +No activation tile can be packed reference-exactly until the full activation's +global reduction is complete. The minimum exact design therefore has two +phases: + +1. Reduce the complete BF16 activation to one FP32 scale. +2. Produce packed tiles and consume them in block-scaled MMA. + +A future cooperative mega-kernel could combine the phases around a grid-wide +barrier, but that is not the first prototype. Programmatic dependent launch can +overlap independent weight preparation but cannot transfer shared-memory tiles +between kernels. + +## Reuse Constraint + +The SM121 block-scaled kernel uses cluster shape `1x1x1`; there is no TMA +multicast path for sharing one packed activation tile among output-N CTAs. +Replacing global packed A with an ordinary per-CTA software producer would make +every output-N CTA reread BF16 A and repeat scale/pack work. BF16 A is materially +larger than packed E2M1 plus E4M3 scales, so this can lose despite removing the +initial packed-tensor write. + +The owned kernel must therefore test an N-group schedule where one CTA retains +one or more packed A K-tiles while accumulating multiple output-N tiles. The +number of simultaneous N tiles is constrained by accumulator registers and +shared memory. This reuse factor is a first-class tuning parameter. + +## Prototype Sequence + +### P0: Owned Prepacked Baseline + +Port the official persistent cooperative SM120/SM121 CuTe DSL kernel behind an +experimental Vortex entry point. Match Comfy Kitchen for the four H3 projection +shapes using existing packed activation and weight tensors. + +Acceptance: + +- Exact BF16 output versus `scaled_mm_nvfp4`. +- H3 widths 5376, 7168, and 14336 plus QKV output width 21504. +- No regression beyond measurement noise before changing the A producer. + +Current P0 result: real 128-row H3 tiles execute through the CuTe kernel. Raw +block-scaled BF16 output is bit-exact for QKV, attention output, and FC1. P0 is +not complete: + +- The experimental epilogue now applies H3's FP32 global-scale product before + BF16 conversion. QKV, attention output, and FC1 are bit-exact. +- FC2 at K=14336 differs even before global scaling (`max_abs=8192`, mean + `0.015625`). The correct alpha epilogue reduces final error to mean + `2.42e-5`, but bit equality still fails with `max_abs=16`. Its cuBLAS path + uses a different reduction/Stream-K policy; CUTLASS K tiles 128 and 256 + produce the same non-reference result. +- The stock CuTe Float32 output mode fails the official example's own GB10 + reference check and cannot be used as an accumulation oracle. + +Do not begin P1 until the FC2 reduction contract is exact or FC2 is deliberately +kept on the existing cuBLAS fallback. See +`benchmarks/gb10-cute-p0-h3-summary.json`. + +Decision: FC2 remains explicitly on the existing cuBLAS path. P1 and later +streaming work target QKV, attention output, and FC1 only. This preserves the +reference FC2 Stream-K reduction order and therefore full-model bit parity while +the owned producer-consumer path is developed independently. + +### P1: Single-Tile Software A Producer + +Replace the A and SFA TMA loads for one fixed output tile with a software warp +that loads BF16 A, generates E4M3 scales and E2M1 values into the existing staged +shared-memory layouts, and commits a producer-consumer pipeline stage. Keep +prepacked B/SFB and the existing MMA consumer and BF16 epilogue. + +Acceptance: + +- Packed bits and scale bytes agree with Comfy for every consumed tile. +- GEMM output is bit-exact for adversarial and randomized tile inputs. +- No global QDATA or SFA allocation in the captured kernel. + +Current P1 checkpoint: the fixed 128-row producer-consumer path is exact. The +existing 32-thread DMA warp loads four BF16 rows per lane, writes E2M1 and E4M3 +directly into the staged `sA`/`sSFA` layouts, and lets retained B/SFB TMA +completion publish each stage to the unchanged MMA consumer. The kernel takes +BF16 A and one tensor scale; it does not take or allocate global activation +QDATA or SFA. + +Every 128-K tile has zero differing packed or scale bytes for real QKV, +attention-output, and FC1 activations. Their complete 128-row GEMM outputs are +also bit-exact after the alpha-before-BF16 epilogue. Exact Comfy compatibility +requires PTX `rcp.approx.ftz.f32`, because Comfy Kitchen builds the quantizer +with `--use_fast_math`; ordinary FP32 division changes E2M1 decisions at +midpoints. CUTLASS DSL 4.6.2 also requires a static contiguous view for vector +FP4 stores, as lowering the same store through a rank-2 dynamic layout aborts +MLIR construction. Evidence is in +`benchmarks/gb10-cute-p1-stream-a-summary.json`. + +P1 is not deployment-complete. The next work is canonical row-count and padding +support and an owned runtime module rather than validator-time patching of the +CUTLASS example. FC2 continues to reject streaming and remains on Comfy/cuBLAS. + +### P2: N-Group Reuse + +Accumulate multiple output-N tiles per produced A tile. Sweep N-group size and +mainloop stages while recording registers, occupancy, achieved tensor +throughput, BF16 bytes read, and total projection latency. + +Reject the no-materialization design if repeated BF16 reads or reduced occupancy +make it slower than the prepacked baseline. In that case, test a bounded global +ring buffer of packed tiles as the producer-consumer compromise. + +Current P2 result: reject the one-producer-per-output-N-CTA schedule. At 128 +rows, streamed execution is `9.14x`, `12.98x`, and `12.44x` slower than the +complete scale+pack+GEMM reference for QKV, attention output, and FC1. Producer +overhead scales at approximately `0.56-0.60 us` per `(N tile, K tile)`. Merely +matching the removed activation quantization cost would require theoretical N +reuse factors of 78, 28, and 102 respectively. Those factors are not practical +through duplicate accumulator state, especially with the existing 232-register +consumer requirement. + +The next prototype is therefore a bounded global packed-tile ring or persistent +work queue: produce each `(M,K)` A tile once, let all required N consumers reuse +it, then recycle the slot. This deliberately restores bounded packed global +traffic while avoiding complete activation materialization. See +`benchmarks/gb10-cute-p1-stream-a-timing-summary.json`. + +The bounded-ring capacity sweep is complete. A caller-owned `_into` CUDA +producer writes exact QDATA and SFA into reusable buffers, and one full-activation +scale is reused by every chunk. Capacities 128, 512, 1024, 2048, and 4096 were +measured; 2048 rows is the selected point. It uses 6.19 MB for QKV/FC1 and 8.26 +MB for attention output. Smaller chunks leave launch overhead, while 4096 rows +regresses. + +At 2048 rows, measured chunk latency improves by `10.7%` for QKV, `14.4%` for +attention output, and `11.3%` for FC1. Modeled canonical totals improve by +`10.1-12.5%` after charging one full-activation scale reduction. All packed +bytes, scale bytes, and BF16 outputs are exact. QKV, attention output, and FC1 +therefore advance to complete real-projection validation; FC2 remains on +Comfy/cuBLAS. See `benchmarks/gb10-cute-p2-ring-capacity-summary.json`. + +Complete 37,810-row projection parity now passes for QKV, attention output, and +FC1 in blocks 0, 24, and 49. All 19 chunks, including the final 946-row chunk, +are bit-exact with equal full checksums. QKV has a stable approximately 32 ms +ring time and improves in every tested block, so it advances to opt-in resident +runtime validation. Attention output timing is mixed. FC1 timing is unstable +under duplicate-model unified-memory pressure, including one container stop, +despite exact outputs. Do not approve those two roles from this harness. See +`benchmarks/gb10-cute-p2-ring-full-projection-summary.json`. + +The opt-in QKV-only integration passes numerical validation but fails the +single-model performance gate. Alternating baseline/ring calls in one loaded +block are bit-exact for blocks 0, 24, and 49, while the 2048-row schedule is +`0.52-0.81%` slower at block level. Capacity checks from 3072 through a full +37888-row workspace do not recover a gain; 4096 rows is closest at `0.24%` +slower. The reusable output also requires serialized model requests because the +next projection reuses its storage after GPU work is enqueued. Keep this backend +disabled. The next valid prototype must fuse chunk launches through a persistent +work queue or replace the scheduler before repeating this block gate; trajectory +validation is intentionally skipped. See +`benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`. + +### P3: Full H3 Projection + +Support canonical H3 M/N/K shapes, padding, current weight/block-scale layout, +and token-row slicing. Integrate through `Nvfp4Linear` behind an opt-in feature +flag. + +### P4: Exact Epilogue + +After the owned GEMM is exact, add the explicit BF16 conversion, gate multiply, +residual add, and final BF16 rounding sequence. Validate this independently from +input streaming. + +## Validation Ladder + +Every phase must pass, in order: + +1. Adversarial scale, E4M3, E2M1, signed-zero, midpoint, NaN, and infinity cases. +2. Randomized packed-tile and GEMM comparisons. +3. Real QKV, output, FC1, and FC2 projections at blocks 0, 24, and 49. +4. Complete block equality. +5. All 50 blocks in a denoiser step. +6. Two-step video/audio trajectory equality. +7. Canonical 12-step video/audio checksum equality. +8. Full block Nsight recapture and peak-memory comparison. + +No streamed path becomes a deployment default before all applicable exactness +gates pass. diff --git a/NVFP4_SWIGLU_FUSION_DESIGN.md b/NVFP4_SWIGLU_FUSION_DESIGN.md new file mode 100644 index 0000000..64f0611 --- /dev/null +++ b/NVFP4_SWIGLU_FUSION_DESIGN.md @@ -0,0 +1,30 @@ +# NVFP4 SwiGLU Producer Fusion + +## Implementation + +The H3 MLP now recomputes its exact BF16 SwiGLU values inside the Vortex global +scale and NVFP4 pack passes instead of materializing the 37,810x14,336 BF16 +activation. The FC2 Comfy Kitchen NVFP4 GEMM and its reduction order are +unchanged. + +The fused producer preserves both BF16 boundaries: + +1. `silu(gate)` rounded to BF16. +2. The BF16 SiLU result multiplied by BF16 `up` and rounded to BF16. + +It then preserves the established Vortex tensor-scale compatibility cast, +E4M3 block scales, E2M1 encoding, approximate reciprocal, and Comfy physical +scale layout. + +## Validation + +- Randomized BF16 tensor scale, QDATA, and SFA are byte-exact. +- Real block-24 FC1 output is byte-exact; producer median is 21.93 ms. +- Blocks 0, 24, and 49 are bit-exact and improve by 2.14-2.26%. +- Warmed two-step video/audio tensors are bit-exact. +- Warmed canonical 12-step tensors are bit-exact and improve from 289.14 s to + 277.36 s, or 4.07%. + +Spark single-GPU deployments enable the path with +`H3_NVFP4_SWIGLU_FUSION=1`. Active FC2 LoRA, full-precision, pre-scale, +autograd, and unsupported dtype/layout cases use the materialized fallback. diff --git a/PERFORMANCE_ROADMAP.md b/PERFORMANCE_ROADMAP.md new file mode 100644 index 0000000..9390832 --- /dev/null +++ b/PERFORMANCE_ROADMAP.md @@ -0,0 +1,565 @@ +# H3 Performance Roadmap + +This document is the canonical performance plan for the direct MiniMax H3 +runtime on GB10/SM121, RTX PRO 6000/SM120, and B200/SM100. It separates measured +results from projections and requires numerical and visual gates before an +approximate optimization becomes a default. + +## Executive Decision + +The first component, bit-exact H3 modulation and residual-gate fusion on GB10, +is complete. + +This is the best first engineering target because: + +1. The current path spends about `39 ms` per representative block in modulation + and residual gates, before counting the separate `25.8 ms` SwiGLU operation. +2. A Triton reference with explicit BF16 rounding already exists and has been + validated bit-exact against eager H3 block behavior. +3. The optimization does not alter attention semantics, weights, precision, or + scheduler behavior. +4. The same elementwise design applies to SM100, SM120, and SM121. +5. It provides a low-risk test of the profiling, fallback, parity, and deployment + machinery required by later custom kernels. + +The next component is NVFP4 scale, activation packing, GEMM, and epilogue +fusion. Component 1 established the numerical contracts and validation +infrastructure required to attempt it safely. A custom attention target follows +this component and should preserve SageAttention 2.2.0's validated numerical +policy while adopting zero-copy strided QKV input and SM-specific fused +execution. Sol sparsity must remain experimental until full latent, audio, and +visual quality gates pass. + +Completed result: blocks 0, 24, and 49 are bit-exact and `3.6-3.9%` faster; +the canonical 12-step sampling run is checksum-identical and improves from +`310.11 s` to `301.05 s` (`3.0%`). Peak allocated memory falls by +`408,825,856` bytes. Evidence is in +`benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`. + +Internal closeout summary: + +> Vortex Component 1 is complete. We implemented bit-exact H3 modulation and +> residual-gate fusion on GB10, reducing canonical 12-step sampling from 310.11 +> seconds to 301.05 seconds while retaining identical video and audio checksums. +> The fused runtime improves individual transformer blocks by 3.6-3.9%, reduces +> peak allocated memory by approximately 390 MiB, and passes 32 deployed +> numerical and integration tests. This establishes that Vortex can remove +> meaningful H3 execution overhead without changing the model's numerical +> program or generated output. + +This is measured engineering evidence, not a projected optimization. + +## Measured GB10 Baseline + +The representative workload is 1344x768, 124 frames, 100 synthetic refined-text +tokens, and 37,810 packed tokens. Measurements use block 24 unless noted. + +Warmed top-level sampling step: + +| Backend | Step time | Relative to Sage2 | +| --- | ---: | ---: | +| SageAttention 2.2.0 | `25.414 s` | baseline | +| Sol BF16, tau 1.3 | `21.573 s` | `1.18x` | +| Sol residual-INT8 Q/K, tau 1.3 | `19.774 s` | `1.29x` | + +The denoiser consumes effectively all warmed-step time. Packing takes about +`15 ms`; output transformation and the RES update together take about `1.4 ms`. + +Representative Sage2 block: + +| Component | P50 | Approximate block share | +| --- | ---: | ---: | +| Attention | `259.9 ms` | `50.5%` | +| NVFP4 QKV/out/FC1/FC2 projections | `151.1 ms` | `29.3%` | +| Modulation, residual gates, and SwiGLU | `64.9 ms` | `12.6%` | +| QKV and output layout materialization | `20.3 ms` | `4.0%` | +| Norm and RoPE | `19.7 ms` | `3.8%` | +| Full module forward | `514.9 ms` | baseline | + +SageAttention 2.2.0 internals: + +| Component | P50 | +| --- | ---: | +| Q/K INT8 quantization | `6.45 ms` | +| V FP8 quantization | `10.19 ms` | +| Attention forward | `239.00 ms` | +| K smoothing and framework estimate | `2.98 ms` | + +Sol residual-INT8 Q/K internals at tau 1.3 with exact conditioning KV: + +| Component | P50 | +| --- | ---: | +| K summary and residual quantization | `4.45 ms` | +| V summary | `2.45 ms` | +| Q quantization and routing threshold | `4.27 ms` | +| Total preparation | `11.53 ms` | +| Routed forward estimate | `114.88 ms` | +| Total | `126.42 ms` | + +INT8 P-by-V is not useful on GB10: it increases total Sol attention time to +`138.65 ms`. + +## Quality Boundary + +Current Sol routing is a performance bound, not an accepted Sage2 replacement. + +| Sol policy | Relative L2 versus Sage2 | Performance | +| --- | ---: | ---: | +| Fully dense, tau -100 | `0.0171` | `3.2x` slower than Sage2 | +| Exact conditioning KV, tau 0.8 | about `0.471` | about `1.12x` faster | +| Exact conditioning KV, tau 1.3 | about `0.626` | about `1.87x` faster in isolated attention | + +No measured Sol setting is both faster than Sage2 and numerically Sage-like. +Exact conditioning sinks protect prompt/audio rows but do not remove sparse +video-attention error. A timestep or block hybrid therefore needs complete +latent, audio, lip-sync, and visual evaluation; attention-output timing alone +cannot approve it. + +## Opportunity Ranking + +The recoverable ranges below overlap and must not be added directly. + +| Priority | Target | Measured cost | Plausible saving | Risk | Portability | +| ---: | --- | ---: | ---: | --- | --- | +| Complete | Fuse modulation and residual gates | `39.1 ms/block` projected scope | `3.6-3.9%` measured block gain | Low | Validated on SM121 | +| 1 | Fuse NVFP4 scale, pack, GEMM, and epilogue | `151.1 ms/block` projections | `30-75 ms/block` | Medium | Architecture-specific | +| 2 | Retain Vortex scale discovery | up to `21 ms/block` versus Comfy scale path | `3-9%` block gain already measured | Low | SM100/120/121, retune launch | +| 4 | Sage-compatible zero-copy dense attention | `259.9 ms/block` attention plus `20.3 ms` layout | `70-140 ms/block` | High | Separate SM100/120/121 policy | +| 5 | Fuse SwiGLU with FC1/FC2 boundaries | `25.8 ms/block` plus projection traffic | `5-15 ms/block` | Medium | SM100/120/121 | +| 6 | CUDA graphs or persistent denoiser execution | Not isolated yet | likely `1-5%` sampling | Medium | Shape-bucket-specific | +| 7 | Quality-gated sparse scheduling | up to `140 ms/block` attention bound | Unknown accepted saving | Very high | Backend-specific | + +Native Sage NHD was tested and rejected as a standalone optimization. It was +numerically exact but did not materially improve median path time and had worse +tail latency. Eliminating layout cost requires fusion with QKV production, +RMSNorm/RoPE, or the attention kernel rather than changing Sage's layout flag. + +## Implementation Phases + +### Phase 1: Exact H3 Elementwise Fusion + +Status: complete on GB10/SM121. Spark deployments enable +`H3_FUSED_ELEMENTWISE=1`; other deployments retain the eager default until +validated on their architecture. + +Implement an opt-in direct-runtime path for: + +1. RMSNorm output plus segmented AdaLN scale/shift modulation. +2. Attention residual gate/add. +3. MLP segmented scale/shift modulation. +4. MLP residual gate/add. +5. A request-layout segment-index cache reused by all 50 blocks and sampling + steps. + +Preserve explicit BF16 rounding boundaries. Keep the eager implementation as a +fallback until all gates pass. + +The achieved `3.0%` sampling gain is materially below the original theoretical +`20-35 ms/block` opportunity. Before extending this fusion, profile: + +1. Which modulation, gate, or add launches remain outside the fused path. +2. Whether tensor conversions or intermediate allocations remain. +3. Whether the fused kernels are launch-bound or bandwidth-bound. +4. Whether generated kernels perform unnecessary loads or stores. +5. Which remaining elementwise operations can move into exact NVFP4 GEMM + epilogues. + +The GB10 Nsight follow-up is complete. One warmed block has 53 kernel launches. +The fused path contains exactly two modulation and two residual gate/add +launches; no modulation, gate, or add launch remains outside it. Those four +kernels still consume about `20.48 ms`, showing that the residual is tensor +traffic rather than launch overhead. The four NVFP4 projections account for 32 +launches across scale reduction/finalization, scalar conversions, block-scale +initialization, activation packing, and GEMM. Their current intermediate tensor +sizes sum to approximately `7.73 GB` per block when counted at each projection +boundary. See +`benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json`. + +Success criteria: + +- Bit-exact outputs at blocks 0, 24, and 49. +- Bit-exact one-step full-denoiser video and audio outputs. +- No change to a complete 12-step latent checksum. +- At least `5%` median block improvement or `3%` warmed sampling improvement. +- No peak-memory regression. + +### Phase 2: Architecture Baselines + +Capture the same component report on each architecture before writing a shared +kernel abstraction: + +| Architecture | Required baseline | +| --- | --- | +| SM121 GB10 | Sage2 2.2, current Vortex scale path, 38K and 74K tokens | +| SM120 RTX PRO 6000 | SDPA, packaged Sage2 2.2, 38K and 74K tokens | +| SM100 B200 | SDPA, forced cuDNN SDPA, Sage2 if supported, 38K and 74K tokens | + +Record actual kernel names, clocks, power, SM utilization, achieved bandwidth, +and achieved tensor throughput. Do not infer one architecture's policy from +another's result. + +### Phase 3: NVFP4 Fused Projection Prototype + +Status: active next component on GB10/SM121. + +The first profile also exposed and fixed a native activation-packer layout bug: +the previous width-specific block-scale swizzle failed at the attention output's +7168-feature width. The replacement general 128-row by 4-scale-column mapping is +packed-bit exact at H3 widths 5376, 7168, and 14336, and all four linear outputs +now match Comfy exactly. Standalone native packing is not consistently faster, +so it remains prototype infrastructure rather than a deployed backend. + +Component 2 execution order, revised from the measured Nsight profile: + +1. Make QKV projection land directly in the attention backend's required + layout, with Q/K normalization and RoPE operating there. +2. Make attention output land directly in the token-major layout consumed by + the output projection. +3. Eliminate complete NVFP4 packed-activation and block-scale materialization + through a CUTLASS/CuTe producer-consumer or persistent design. +4. Integrate reference-exact residual gates into output-projection and FC2 + epilogues. +5. Optimize the standalone packer only after the boundary-removing paths exist. +6. Recapture the complete block profile after every accepted boundary removal. + +The first two layout items are complete for single-GPU Sage2 on GB10. Sage2 +accepts the projection-strided NHD Q/K/V views directly, including in-place Q/K +normalization and RoPE, and emits contiguous NHD output that reshapes to +token-major rows without a copy. This removes three QKV copies and the attention +output copy without replacing the projection GEMM. Blocks 0, 24, and 49 and the +two-step and canonical 12-step trajectories are bit-exact. Canonical sampling +improves from the Component 1 baseline of `301.05 s` to `290.23 s` (`3.6%`), +with cumulative improvement of `6.4%` from the original `310.11 s` baseline. +The accepted Nsight recapture falls from 53 to 49 launches and contains none of +the four large BF16 layout-copy kernels. +Evidence is in +`benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json`. + +The target NVFP4 dataflow is not another wrapper around +`scaled_mm_nvfp4`. Tensor-core GEMMs require exact packed-data and block-scale +layouts, so avoiding durable HBM intermediates likely requires an owned +CUTLASS/CuTe collective with tile-level producer-consumer execution. + +Interface investigation confirms this boundary. Comfy Kitchen 0.2.31 allocates +complete QDATA, block-scale, and BF16 output tensors before invoking a cuBLAS +blockwise FP4 GEMM with fixed pointers. It cannot consume incremental tiles. +CUTLASS DSL 4.6.2's stock SM121 E2M1/E4M3 persistent kernel passes on GB10, but +its public mainloop also TMA-loads prebuilt A and SFA. The owned implementation +will follow `NVFP4_STREAMING_DESIGN.md`: preserve the mandatory global-scale +reduction, replace the A/SFA producer in one fixed tile, then measure multi-N +reuse before attempting full H3 shapes. + +P0 is complete for the selected streamed roles. The owned CuTe bridge and +alpha-before-BF16 epilogue are bit-exact +for QKV, attention output, and FC1, proving E2M1, E4M3, and global-scale +interoperability. FC2 remains non-exact because its reference uses a different +reduction policy (`max_abs=16` after the correct epilogue). Resolve that policy +or retain an explicit FC2 cuBLAS fallback before the streaming A producer begins. + +FC2 fallback is selected. The streamed backend will reject FC2 dispatch and use +the existing Comfy/cuBLAS implementation there; QKV, attention output, and FC1 +advance to the P1 software A/SFA producer. + +The fixed 128-row P1 producer-consumer checkpoint is complete. The stock +kernel's 32-thread DMA warp produces four rows per lane directly into staged +E2M1 A and E4M3 SFA shared memory; only B and SFB retain TMA loads. Every 128-K +tile has zero packed-data and scale-byte differences for real QKV, +attention-output, and FC1 activations, and all three complete GEMM outputs are +bit-exact after the fused alpha epilogue. The streamed kernel receives BF16 A +plus its tensor scale and has no global activation QDATA or SFA input. + +The final arithmetic contract includes Comfy's `--use_fast_math` behavior: +encode scale must use PTX `rcp.approx.ftz.f32`. Correctly rounded division changes +FP4 midpoint decisions for real H3 values. CUTLASS DSL 4.6.2 also requires a +static contiguous destination view for vector FP4 stores. Evidence is in +`benchmarks/gb10-cute-p1-stream-a-summary.json`. + +The P1 timing gate rejects direct per-output-N-CTA streaming. At 128 rows, the +exact streamed kernel is `9.14x` slower than the complete reference projection +for QKV, `12.98x` slower for attention output, and `12.44x` slower for FC1. +Producer overhead is linear at approximately `0.56-0.60 us` per `(N,K)` tile. +Break-even against the removed quantizer would require theoretical N reuse of +78, 28, and 102 tiles respectively, which cannot be implemented by retaining +duplicate accumulator state under the current register budget. + +The next P2 prototype is a bounded global packed-tile ring or persistent work +queue. It must produce each `(M,K)` A tile once, expose it to all N consumers, +and recycle the slot, trading bounded global packed traffic for elimination of +the measured repeated BF16 conversion. Do not add canonical-shape runtime +dispatch for the rejected schedule. FC2 remains an explicit Comfy/cuBLAS +fallback. Evidence is in +`benchmarks/gb10-cute-p1-stream-a-timing-summary.json`. + +The bounded-ring P2 checkpoint now uses caller-owned native QDATA/SFA buffers and +an allocation-free `_into` producer. A capacity sweep with one full-activation +scale selected 2048 rows. The ring occupies 6.19 MB for QKV/FC1 and 8.26 MB for +attention output. Measured chunk latency improves QKV by `10.7%`, attention +output by `14.4%`, and FC1 by `11.3%`; modeled canonical totals improve by +`10.1-12.5%`. Every packed byte, scale byte, and BF16 output matches the +reference. The 4096-row point regresses, confirming that larger buffers are not +monotonically better. Evidence is in +`benchmarks/gb10-cute-p2-ring-capacity-summary.json`. + +Complete 37,810-row projection parity now passes for QKV, attention output, and +FC1 in blocks 0, 24, and 49. Every one of the 19 chunks, including the final +946-row tail, is bit-exact. QKV has a stable approximately 32 ms ring time and a +positive result across all three blocks. Attention-output timing is mixed. FC1 +timing is not usable from the duplicate-model harness because multi-gigabyte +output pressure caused large variance and one container stop. + +The opt-in QKV runtime gate is complete and rejects the current bounded-ring +schedule. An alternating baseline/ring benchmark inside one loaded block removes +the clock, allocator, input, and model-copy bias present in separate-process +profiles. Blocks 0, 24, and 49 remain bit-exact, but the 2048-row ring regresses +median block time by `0.81%`, `0.52%`, and `0.52%`, respectively. On block 24, +3072 rows regress by `1.00%`, 4096 by `0.24%`, 8192 by `4.80%`, and a 37888-row +full workspace by `14.49%`. The isolated projection win does not survive the +chunk launch and scheduler overhead. Keep the implementation opt-in and disabled; +do not spend trajectory-validation compute until a launch-fused work queue or a +different persistent scheduler passes this same block gate. Attention output and +FC1 remain experimental, and FC2 remains the explicit Comfy/cuBLAS fallback. +Evidence is in `benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`. + +The post-optimization profile identifies Sage2's mainloop as the largest cost. +Canonical sampling is `288.93 s`; block 24 is +`468.22 ms`, and its production NHD Sage2 call is `258.47 ms`. The existing +SM89 mainloop alone takes `238.81 ms`, while Q/K and V quantization together take +`18.34 ms`. A direct Hopper WGMMA retarget was rejected by SM121 ptxas, and +CUTLASS SM120/121 UMMA does not support the INT8 QK operation needed for exact +Sage2 parity. Do not substitute Sage3's different FP4 algorithm under the exact +optimization contract. + +The next implementable exact target was AdaLN modulation into NVFP4 production. +The MSA and MLP modulation passes cost `10.47 ms` and `10.60 ms` and materialize +their values in the RMSNorm buffers solely for QKV and FC1. The deployed fusion +recomputes the exact BF16 values inside Vortex scale/pack and retains Comfy +GEMMs. Complete producer bytes match in blocks 0, 24, and 49. Alternating block +medians improve by `0.28-0.79%`; warmed two-step and 12-step runs improve by +`0.52%` and `0.56%`, with bit-identical outputs. Spark deployments enable +`H3_NVFP4_MODULATE_FUSION=1`. Evidence is in +`benchmarks/gb10-nvfp4-modulate-fusion-summary.json`. + +SwiGLU-to-FC2 producer fusion is complete. Recomputing the exact BF16 SiLU and +multiply boundaries inside Vortex scale/pack removes the complete intermediate +activation while retaining the reference Comfy FC2 GEMM. The real producer is +byte-exact and takes `21.93 ms`. Alternating blocks 0, 24, and 49 improve by +`2.14-2.26%`; the warmed canonical 12-step trajectory improves by `4.07%` with +identical video and audio tensors. Enable `H3_NVFP4_SWIGLU_FUSION=1` for Spark +single-GPU deployments. Evidence is in +`benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`. + +The mandatory post-fusion profile supersedes the earlier approximately 515 ms +block distribution. With all production flags enabled, fresh block-24 timing is +`458.78 ms` median without Nsight and `465.78 ms` across the Nsight GPU span. +The 41 kernels have only `0.084 ms` total inter-kernel idle time, so CPU launch +gaps are not a block-level bottleneck. A complete warmed sampling step takes +`23.708 s`, contains 2,694 kernels, and has `12.995 ms` total inter-kernel idle +time. + +The fresh block kernel-time distribution is: + +| Component | Time | Share | +| --- | ---: | ---: | +| Sage2 preparation and mainloop | `268.22 ms` | `57.60%` | +| Four NVFP4 GEMMs | `125.11 ms` | `26.87%` | +| NVFP4 scale and packing | `41.53 ms` | `8.92%` | +| Norm and RoPE | `19.86 ms` | `4.27%` | +| Two residual gate/add kernels | `10.94 ms` | `2.35%` | + +The one-step distribution independently matches these shares within 0.5 +percentage points. The standalone modulation and SwiGLU kernels are gone. +Their remaining producer work is `13.04 ms` for modulated QKV/FC1 packing and +`23.04 ms` for SwiGLU-to-FC2 packing. The block still records 8,009,578,496 +positive self-allocated bytes, but there are no large activation-layout copy +kernels or explicit CUDA memcpy operations. Remaining conversions are small +AdaLN table and scalar operations. + +GB10 exposes no direct physical DRAM-byte counter. Nsight Compute measures +`320.999 GB` of L2-request traffic for the block. With cache control disabled, +L2 read/write miss sectors imply a `42.117 GB` off-chip request proxy: NVFP4 +GEMMs account for `56.25%`, packing `18.25%`, Sage2 `15.09%`, norm/RoPE +`6.54%`, and gates `3.87%`. This separates the time bottleneck, Sage2, from the +off-chip traffic bottleneck, the NVFP4 GEMMs. Do not select a new kernel from +the old profile. See +`benchmarks/gb10-fully-fused-fresh-nsight-summary.json`. + +The follow-up real block-24 Sage2 decomposition now selects the next exact +kernel experiment. Manual preparation plus the existing prequantized mainloop +is byte-exact against public SageAttention 2.2.0. Uninstrumented median phase +times are `2.37 ms` K smoothing, `7.63 ms` combined Q/K quantization, `10.61 ms` +combined V transpose/quantization, and `237.09 ms` for the fused mainloop. + +The full-counter mainloop capture is profiler-perturbed to `258.92 ms`, so its +duration is not used as the baseline. Its ratios establish the bottleneck: + +| Mainloop metric | Result | +| --- | ---: | +| Registers per thread | `255` | +| Dynamic shared memory per CTA | `32 KiB` | +| Achieved occupancy | `16.83%` / `8.08` warps per SM | +| Scheduler cycles with no eligible warp | `63.53%` | +| INT8 QK tensor-pipe utilization | `37.77%` of elapsed cycles | +| FP8 PV tensor-pipe utilization | `37.77%` of elapsed cycles | +| Combined tensor-pipe utilization | `75.54%` | +| Memory throughput | `31.61%` | +| L2 hit rate | `98.84%` | +| L2 request traffic | `161.50 GB` | +| Off-chip request proxy | `1.85 GB` | +| Excess shared-memory wavefronts | `7.68%` | + +This explains why high SM activity did not imply a saturated tensor pipeline. +QK and PV alternate on separate INT and FP tensor sub-pipelines, while online +softmax, scaling, conversion, and synchronization occupy scalar pipelines. +Only `0.46` warps per scheduler are eligible on average. Fixed-latency +dependencies consume `2.01` of the `5.48` cycles between issued instructions, +and math-pipe throttle consumes another `1.24`; memory scoreboards are much +smaller. The 255-register footprint limits the kernel to two CTAs per SM, so it +cannot hide these dependencies. Q and KV tail sweeps each add less than 1 ms and +exclude tail scheduling as the primary target. + +The source-identical classic SM89 P0 is complete and rejected. The exact +three-CTA cliff is 168 registers/thread, not approximately 170: 170 still has +two CTAs and `16.67%` theoretical occupancy, while 168 reaches `25%` at the cost +of `4.95 billion` local spill requests and `78.79%` no-eligible cycles. Caps from +240 through 170 never change residency and progressively worsen scheduler +eligibility. + +Byte-exact source variants also missed the 3% gate. Narrowed scopes reduced +static spills from 44/44 to 12/12 bytes and dynamic spill requests by `73%`, but +improved interleaved latency by only `0.06%`. In-place score storage was +`+0.01%`; early K prefetch was `-0.30%`; independent softmax-chain interleaving +was `-0.02%`. The persistent 128-register FP32 output fragment plus exact QK or +instantaneous PV fragments prevents a source-only lifetime cleanup from +reaching the next residency tier. + +The shared-memory follow-up is also closed. Source attribution maps all +`626,970,624` excessive wavefronts to four repeated V-staging `LDGSTS.128` +instructions. A 128-byte padded V stride raised dynamic shared memory from 32 +to 40 KiB but left the excess count exactly unchanged and changed latency by +`-0.04%`. No complete-block benchmark or deployment was run because no mainloop +variant crossed 3%. Evidence is in +`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`, the per-variant +latency JSON files, and the P0 NCU reports. + +Materializing the optional BF16 LoRA input while these fused producers pack +NVFP4 was also tested against the official Turbo-4 adapter. The isolated +1344x768, 124-frame trajectory was bit-exact but regressed from `131.11 s` to +`135.14 s` (`3.08%`). Reject this implementation: its extra BF16 writes cost +more than the removed standalone producer. Active LoRA must retain the existing +materialized fallback unless a future design consumes the producer values +without global-memory materialization. Evidence is in +`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`. + +An exact residual-gate epilogue must preserve this numerical program: + +1. Complete NVFP4 GEMM accumulation. +2. Convert to the reference BF16 GEMM output. +3. Apply the reference-equivalent gate multiplication and its BF16 boundary. +4. Add the BF16 residual using the reference operation order. +5. Round the final output to BF16. + +Bit-exact GEMM output alone is insufficient. Validate randomized epilogues, +representative real blocks, all 50 blocks in one denoiser step, a two-step +trajectory, and the canonical 12-step trajectory before enabling an epilogue. + +The direct-layout target extends naturally to distributed execution: QKV tiles +must be able to land in rank-destined Ulysses buffers, and attention output must +land in the output projection's local token-major layout without an intervening +transpose or contiguous copy. + +Prototype behind `Nvfp4Linear` so model code remains unchanged. Compare: + +1. Comfy Kitchen baseline. +2. Vortex scale plus Comfy pack/GEMM. +3. Fused scale and activation pack. +4. Fused scale, pack, GEMM, bias, and output epilogue. +5. Transformer Engine or CUTLASS reference where supported. + +Gate every variant on packed activation identity, linear output error, block +output error, full denoiser-step error, and final media quality. + +### Phase 4: Dense Sage-Compatible Attention + +The first owned attention kernel should remain dense and target Sage2 behavior: + +1. Consume strided BSHD views directly from fused QKV output. +2. Preserve Sage2 per-warp INT8 Q/K quantization and FP8 V policy. +3. Preserve its accumulation and K-smoothing behavior. +4. Fuse Q/K/V preparation with the attention launch where profitable. +5. Emit token-major output suitable for the NVFP4 output projection without an + intermediate transpose/contiguous allocation. +6. Tune independently for SM100, SM120, and SM121. + +The initial goal is `1.5x` attention speedup with bit-exact Sage2 output where +the operation order permits it; any nonzero tolerance must be declared and +quality-gated before implementation. A `2x` attention target is stretch. + +### Phase 5: Sparse Hybrid Research + +Only after the dense backend passes: + +1. Keep early quality-sensitive sampling steps dense. +2. Keep first/last sensitive transformer blocks dense. +3. Force text, reference, and audio conditioning KV blocks exact. +4. Sweep tau by timestep using complete generated media, not random attention + tensors alone. +5. Reject any policy that degrades prompt adherence, identity, lip sync, audio + onset, or temporal stability. + +## Performance Targets + +### GB10 / SM121 + +The current two-step profile implies roughly `305 s` for a warmed 12-step +1344x768/124-frame sampling run. + +| Target | Estimated sampling | Speedup | +| --- | ---: | ---: | +| Conservative | `215-240 s` | `1.3-1.4x` | +| Engineering | `170-195 s` | `1.6-1.8x` | +| Stretch | `130-145 s` | `2.1-2.3x` | +| Aggressive ceiling | about `120 s` | about `2.5x` | + +Use `1.5x` as the commitment target, `1.8x` as the engineering target, and +`2.3x` as stretch. Do not plan around a `5x` end-to-end gain. + +### RTX PRO 6000 / SM120 + +These projections have lower confidence because the measured RTX baseline uses +PyTorch SDPA rather than packaged Sage2. + +| Workload | Current SDPA | Engineering target | Stretch target | +| --- | ---: | ---: | ---: | +| 1344x768, 124 frames | `97.94 s` | `45-60 s` | `35-45 s` | +| 1344x768, 243 frames | `311.02 s` | `120-170 s` | `90-120 s` | + +Packaging and profiling Sage2 2.2 on SM120 is the first RTX action. Expected +overall improvement is `1.6-2.2x` at 124 frames and `1.8-2.6x` at 243 frames; +stretch ranges are `2.2-2.8x` and `2.6-3.5x` respectively. These are planning +ranges, not commitments. + +### B200 / SM100 + +Measured B200 SDPA sampling is `79.72 s` at 124 frames and `257.86 s` at 243 +frames, only `1.23x` and `1.21x` faster than RTX PRO 6000 despite much larger +theoretical FP4 and memory-bandwidth capability. No optimized B200 target is +assigned until component profiling confirms actual SDPA dispatch, NVFP4 kernel +selection, clocks, power, utilization, and achieved throughput. + +The current B200 result measures this runtime path, not B200's hardware ceiling. + +## Evidence + +- `benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json` +- `benchmarks/gb10-37810-token-optimization-profile-seed440420.json` +- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json` +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `tools/profile_h3_block.py` +- `tools/profile_attention_components.py` +- `tools/profile_hybrid_attention.py` +- `tools/profile_sampling_stages.py` + +Component timings use synchronized probes for attribution. End-to-end claims +must use separate uninstrumented runs. Resident services remained loaded during +GB10 profiling, so medians are preferred over means and final claims require an +isolated rerun. diff --git a/PLAN.md b/PLAN.md index 556cc51..ad5f260 100644 --- a/PLAN.md +++ b/PLAN.md @@ -93,6 +93,33 @@ Current NVFP4 activation quantization status: - `vortex_native` BF16->NVFP4 pack is parity-safe for real H3 FC1/FC2 activation shapes, including qdata, cuBLAS tiled block-scale layout, signed zero, tie-to-even FP4 rounding, and downstream `scaled_mm_nvfp4` linear output. - Keep Comfy Kitchen as the default production packer. Pack-only sweep `8d90cec` shows native 512-thread pack is essentially tied on FC1 (`1.090 ms` vs CK `1.098 ms`) but still slower on FC2 (`2.977 ms` vs CK `2.886 ms`). Use `vortex_native` only as an experimental benchmark path until the wide-FC2 pack kernel is redesigned. +Major cross-architecture performance opportunities: + +1. Better attention kernels, especially for 38K-74K-token sequences. +2. Eliminating Q/K/V transpose and contiguous copies. +3. Fusing NVFP4 activation scaling, packing, GEMM, bias, and epilogues. +4. Replacing or tuning Comfy Kitchen GEMMs independently for SM100, SM120, and SM121. +5. CUDA graphs or persistent execution to reduce thousands of kernel launches per sample. + +GB10 profiling at 1344x768/124 frames produced 37,810 packed tokens. Sage2 +attention remains the parity baseline; native NHD layout was exact but did not +improve median path time, and Sol-Attn was faster but materially changed the +attention output. The first retained optimization routes H3 activation-scale +discovery through the parity-safe Vortex kernel when +`H3_NVFP4_SCALE_BACKEND=vortex`, while retaining Comfy Kitchen's packer and +GEMM. Representative blocks 0, 24, and 49 improved by `3.1%`, `8.8%`, and +`3.3%` respectively with identical output checksums. Details are recorded in +`benchmarks/gb10-37810-token-optimization-profile-seed440420.json`. + +The expanded sampling profile confirms that the denoiser consumes effectively +all warmed-step latency. At block 24, Sage2 spends about `50.5%` in attention, +`29.3%` in NVFP4 projections, `12.6%` in modulation/gates/SwiGLU, `4.0%` in +layout materialization, and `3.8%` in norm/RoPE. Sol's sparse timing is a useful +performance bound, but not an accepted backend: its first speed-positive +settings differ substantially from Sage2, while fully dense Sol is `3.2x` +slower. See +`benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json`. + Validation and profiling sequence: 1. Use `tools/profile_h3_block.py` to measure one representative H3 block before writing kernels. Record QKV, RoPE/RMS, attention kernel, output projection, MLP fc1, activation, MLP fc2, modulation/gating, and total block time. diff --git a/README.md b/README.md index 51bf678..5eb0eb4 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,8 @@ Implemented and validated: - Opt-in EasyCache and H3-Cache experiments. - Ragged Ulysses sequence parallelism and true NVFP4 tensor parallelism with automatic 1/2/4/6/8-GPU launch tooling. +- Opt-in bit-exact Triton fusion for H3 segmented modulation and residual gates; + enabled by default in the validated Spark deployment. The main remaining feature gap is full arbitrary Ref2VA, especially reference video/audio, identity, and voice conditioning. Multi-GPU code is CPU- and @@ -148,6 +150,12 @@ read/write is available and NCCL selected `P2P/CUMEM`; topology varied between `SYS` and `NODE` across RunPod allocations. See `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`. +A matched one-GPU B200 SDPA run took `79.72s` at 1344x768/124 frames and +`257.86s` at 1344x768/243 frames. That is only `1.23x` and `1.21x` faster than +one RTX PRO 6000, while the measured sampling cost at RunPod rates was `2.64x` +and `2.69x` higher. See +`benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`. + Matched GB10 backend results and parity evidence are recorded in [`PLAN.md`](PLAN.md) and [`PARITY.md`](PARITY.md). Standalone `tools/compare_*`, `tools/trace_*`, `tools/inspect_*`, and `tools/patch_comfy_*` commands are diagnostic utilities, @@ -165,6 +173,12 @@ not runtime startup requirements. - [`FLASH4.md`](FLASH4.md): FlashAttention-4 integration and benchmark. - [`PARITY.md`](PARITY.md): direct-versus-Comfy evidence ledger. - [`PLAN.md`](PLAN.md): historical investigation and future kernel plan. +- [`PERFORMANCE_ROADMAP.md`](PERFORMANCE_ROADMAP.md): measured bottlenecks, + architecture targets, quality gates, and the ordered optimization plan. +- [`VORTEX_RESEARCH_AGENDA.md`](VORTEX_RESEARCH_AGENDA.md): flagship research + thesis, novel systems contributions, evaluation targets, and project ranking. +- [`NVFP4_STREAMING_DESIGN.md`](NVFP4_STREAMING_DESIGN.md): measured GEMM + interface boundary and the exact producer-consumer kernel prototype sequence. ## Known Gaps diff --git a/SAGE2_BLACKWELL_DESIGN.md b/SAGE2_BLACKWELL_DESIGN.md new file mode 100644 index 0000000..03ca6b6 --- /dev/null +++ b/SAGE2_BLACKWELL_DESIGN.md @@ -0,0 +1,152 @@ +# Sage2 Blackwell Mainloop + +## Decision + +The adjacent NVFP4 producer fusions are complete. The next exact kernel +experiment is now a retune of SageAttention 2.2.0's existing classic SM89 +mainloop. Do not replace the algorithm with UMMA, FP8 Q/K, or FP4 attention. + +The canonical GB10 profile is dominated by the attention kernel: + +- Complete block 24: 468.22 ms median. +- Sage2 NHD path: 260.02 ms median. +- SM89 attention mainloop: 238.81 ms median. +- Q/K quantization: 7.68 ms median. +- V quantization: 10.65 ms median. +- Remaining framework overhead: 2.87 ms estimated. + +The fresh real block-24 decomposition supersedes those internal timings: + +- Fused mainloop: 237.09 ms uninstrumented median. +- K mean/smoothing: 2.37 ms median. +- Q quantization: 3.80 ms median. +- K subtract-mean quantization: 3.83 ms median. +- V transpose/pad/permute: 5.08 ms median. +- V scale/FP8 quantization: 5.53 ms median. +- Manual decomposition versus public Sage2: byte-exact. + +Q/K RMSNorm plus RoPE is 12.20 ms and is not the first target. NVFP4 ring +dispatch is also excluded because it failed the controlled block gate. + +## Existing Contract + +For the 37,810-token H3 shape, preserve these Sage2 inputs and outputs: + +- Batch 1, 56 heads, head dimension 128, non-causal attention. +- Projection-strided BF16 NHD Q/K/V input. +- Per-warp INT8 Q with 128-row blocks and 32-row warp groups. +- Per-block INT8 K with 64-row blocks. +- FP8 V and the existing per-channel scale layout. +- Sage2's `fp32+fp16` instantaneous-buffer accumulation order. +- Contiguous BF16 NHD output with exact equality to Sage2 2.2.0. + +The current mainloop is +`sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf`. +It receives Q/K INT8 tensors, FP8 V, Q/K/V scales, and emits the final BF16 +attention output. + +## Rejected Shortcut + +SageAttention's Hopper SM90 WGMMA source was compiled experimentally for +`sm_121a`. CUDA 13 ptxas rejects `wgmma.fence`, `wgmma.mma_async`, +`wgmma.commit_group`, and `wgmma.wait_group` for SM121, including both INT8 and +FP8 forms. A Hopper binary or source retarget is therefore not viable. + +CUTLASS 4.6 SM120/121 exposes `tcgen05`/UMMA for F8/F6/F4 but not INT8. It +therefore cannot preserve Sage2's INT8 QK operation. Changing Q/K to FP8 or FP4 +would define a different attention algorithm and require a quality gate rather +than exact parity. + +## Scheduler Diagnosis + +The full-counter capture reports 255 registers/thread and 32 KiB dynamic shared +memory for each 128-thread CTA. Registers limit residency to two CTAs, producing +`16.83%` achieved occupancy and only `0.46` eligible warps per scheduler. No +warp is eligible in `63.53%` of scheduler cycles. Fixed-latency dependency +stalls are `2.01` cycles and math-pipe throttle is `1.24` cycles out of the +`5.48` cycles between issued instructions. + +The fused phase attribution is counter-based rather than separate wall time: + +- INT8 QK uses `37.77%` of elapsed tensor-pipe capacity. +- FP8 PV uses `37.77%` of elapsed tensor-pipe capacity. +- Scalar FMA and ALU pipelines use `15.01%` and `8.81%`, covering scale, + online-softmax, conversion, normalization, and output work. +- Combined tensor activity is `75.54%`; the INT and FP sub-pipelines alternate, + so neither individually exceeds `37.77%`. +- Memory throughput is `31.61%`, L2 hit rate is `98.84%`, and the warm + off-chip request proxy is only `1.85 GB`. This is not a DRAM bottleneck. +- Shared accesses create `626,970,624` excessive wavefronts, `7.68%` of all + shared wavefronts. +- Real Q and KV tail CTAs each cost less than 1 ms. + +## Completed P0 + +The source-identical classic `mma.sync` retune was completed without an accepted +variant. The exact three-CTA threshold is 168 registers/thread; 170 remains at +two CTAs. The 168-register cap causes `4.95 billion` local spill requests and +worsens no-eligible cycles from `63.13%` to `78.79%`. Narrowed scopes reduce +static spills from 44/44 to 12/12 bytes but improve real-input latency by only +`0.06%`. In-place score reuse, early K prefetch, and independent softmax-chain +interleaving are also neutral or slower. + +The shared excess is produced by repeated V-staging `LDGSTS.128` instructions. +A 128-byte padded V layout leaves all `626,970,624` excessive wavefronts +unchanged and provides no latency gain. This counter is therefore not an +actionable layout target for the exact kernel. + +The P0 gate was: + +1. Adversarial short-shape comparison against the Sage2 mainloop. +2. Exact real block-24 attention output at 37,810 tokens. +3. Mainloop median below 220 ms before block integration. +4. Blocks 0, 24, and 49 exact after integration. +5. Alternating block benchmark must improve all three blocks before trajectory + validation. + +Use the Blackwell SageAttention3/CUTLASS code only as a scheduler reference. Its +FP4 algorithm and output are not a correctness replacement for Sage2. + +No variant exceeded the 3% mainloop gate, so block integration and deployment +were intentionally skipped. Detailed evidence is in +`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`. + +## Completed P1 And P2 + +P1 fused strided-NHD Q/K RMSNorm, split-half RoPE, and Q INT8 quantization. It +was byte-exact, but complete-block gains were only `0.53-0.86%`, below the 1% +gate, so its runtime branch was removed. + +P2 replaced Sage2's separate BF16 V transpose and FP8 quantization with an owned +three-stage direct-NHD producer. It emits byte-identical FP8 V and FP32 scales +for 13 boundary lengths. Canonical V preparation improves from `10.56 ms` to +`6.39 ms` (`39.48%`), but the `4.17 ms` saving is only `0.91%` of the complete +block and misses the `6.0 ms` isolated go threshold. No block integration or +trajectory validation was run. The validator-only implementation remains useful +for a future owned attention backend; production Sage2 remains unchanged. + +These results close the remaining preparation-only Sage2 opportunities. The +next Sage-class optimization with plausible multi-percent block impact is a new +warp-specialized QK/PV mainloop or a quality-gated Blackwell-native attention +algorithm, not another launch-boundary fusion. + +## Completed P3 + +P3 tested the smallest exact temporal QK/PV specialization. Warp pairs A and B +alternated INT8 QK plus online softmax against the other pair's prior-tile FP8 +PV. Every warp retained its original query rows and private `RS`, `RS_f8`, `RO`, +`m`, and `d`; no numerical state crossed warp boundaries. The candidate is +compute-sanitizer clean and byte-exact across 13 short boundary cases and the +real block-24 output SHA. + +The schedule does not produce a material latency gain. In a rotating 50-sample +comparison, baseline median is `245.44 ms` and the candidate is `245.20 ms`, a +`0.10%` improvement. It remains far above the `<220 ms` gate. Ptxas reports 254 +registers/thread and 32/24-byte static store/load spills for the candidate, +versus 255 registers and 24/24-byte spills for the clean baseline build. NCU and +block integration were skipped because the latency gate failed. + +This closes temporal warp separation as an exact Sage2 optimization on the +current SM89 instruction stream. A multi-percent attention gain now requires a +substantially new mainloop or a quality-gated Blackwell-native attention +algorithm rather than another exact schedule rearrangement. diff --git a/TURBO.md b/TURBO.md index 2c25a30..5195b1a 100644 --- a/TURBO.md +++ b/TURBO.md @@ -30,6 +30,12 @@ output = NVFP4_base(x) + strength * (alpha / rank) * B(A(x)) The BF16 branch receives the original activation. It does not receive the base checkpoint's `pre_quant_scale`-modified activation, and the LoRA delta is never merged into packed NVFP4 weights. Fused QKV uses rank/alpha 384; the other released projections use rank/alpha 128. +Active LoRA intentionally uses the materialized projection fallback. Emitting +the exact BF16 LoRA input while the NVFP4 modulation and SwiGLU producers pack +was bit-exact, but slowed an isolated canonical Turbo-4 trajectory by `3.08%`. +The rejected result is recorded in +`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`. + Turbo sampling maintains independent native video and audio states. The 4-step and 8-step variants use their own shifted sigma grids and positive data-ward training-Euler updates. The original beta/RES sampler remains unchanged when Turbo is disabled. ## Hot API diff --git a/VORTEX_RESEARCH_AGENDA.md b/VORTEX_RESEARCH_AGENDA.md new file mode 100644 index 0000000..ad266dc --- /dev/null +++ b/VORTEX_RESEARCH_AGENDA.md @@ -0,0 +1,185 @@ +# Vortex Research Agenda + +## Flagship Research Project + +**Vortex: Reference-Exact, Communication-Avoiding Video Diffusion on Commodity PCIe GPUs** + +The core research question: + +> Can a large multimodal video diffusion model achieve near-NVLink scaling on +> ordinary PCIe hardware while preserving reference-exact output? + +The existing `86.1%` two-GPU efficiency is the starting evidence, not the final +contribution. + +## Novel Architecture + +Build an end-to-end H3 block where: + +- NVFP4 QKV projections write directly into rank-destined communication buffers. +- Q, K, and V are never separately materialized or repacked. +- Ulysses communication begins at tile granularity while later projection tiles + are still computing. +- Video, audio, text, and conditioning tokens use different sharding policies. +- Small conditioning segments are replicated when replication is cheaper than + communication. +- Received attention output writes directly into the output-projection layout. +- Reference BF16 rounding boundaries are preserved exactly. +- The scheduler automatically accounts for PIX, PXB, PHB, NODE, and SYS topology. + +The target pipeline is: + +```text +NVFP4 projection + | + v +rank-destined QKV tiles + | + v overlapping +PCIe all-to-all + | + v overlapping +attention + | + v +output projection layout +``` + +There is no intermediate QKV pack, full synchronization point, or redundant +memory round trip. + +Communication and computation overlap has already been explored by systems such +as Ulysses Unbound and topology-aware systems such as SwiftFusion. The +contribution therefore cannot simply be "we overlapped an all-to-all." + +The novel combination is: + +- Direct NVFP4 GEMM-to-collective dataflow. +- Ragged multimodal sequence parallelism. +- Segment-aware replication and sharding. +- Reference-exact numerical behavior. +- Automatic PCIe topology adaptation. +- Scaling across inexpensive non-NVLink hardware. + +That combination appears meaningfully ahead of public H3 implementations. + +## Rounding-Boundary-Aware Exact Fusion + +Create a compiler or kernel-generation framework that understands where the +original model performs BF16 rounding. + +Ordinary fusion can silently change output because fused multiply-add performs +one rounding where separate operations perform two. NVIDIA documents this +numerical distinction in its CUDA floating-point guidance. + +The system would: + +- Trace the reference H3 graph. +- Mark mandatory numerical materialization boundaries. +- Fuse everything between safe boundaries. +- Generate Triton or CUDA kernels. +- Prove equivalence with adversarial and randomized testing. +- Preserve bitwise behavior across SM100, SM120, and SM121 where possible. + +Apply it to: + +- AdaLN modulation. +- Residual gates. +- SwiGLU. +- QK normalization and RoPE. +- NVFP4 scaling and packing. +- GEMM epilogues. + +This could be a broader contribution than Vortex itself. It would answer: + +> How much of a mixed-precision transformer can be fused without altering its +> numerical program? + +That is a serious systems and numerical-computing paper. + +## Trajectory-Bounded Sparse Attention + +This is the highest-risk project. + +The Sol results show why simple threshold sparsity is not enough: + +- It becomes fast at higher thresholds. +- Attention-level relative error rises to roughly `0.5-0.6`. +- Local attention error does not reveal what happens to the final video. + +Instead of using one global threshold, allocate an error budget across: + +- Denoising timestep. +- Transformer block. +- Attention head. +- Video, audio, text, and conditioning segments. +- Spatial and temporal regions. + +The runtime would choose dense or sparse attention dynamically, with exact +fallback when its predicted error exceeds the remaining trajectory budget. + +Existing research already covers distributed sparse attention, including DSA, +and feature reuse such as FasterCache. The potential Vortex contribution is +different: + +> Optimize sparsity against final multimodal trajectory error, not isolated +> attention-tensor error. + +That requires proper evaluation of image quality, temporal consistency, audio +quality, lip synchronization, and prompt adherence. + +## Practical Ranking + +| Research direction | Novelty | Technical risk | Commercial value | Recommendation | +| --- | --- | --- | --- | --- | +| PCIe-native ragged Ulysses | Very high | Medium | Exceptional | First | +| Rounding-aware exact fusion | Very high | Medium | Exceptional | Build alongside | +| Trajectory-bounded sparsity | Very high | Very high | Potentially exceptional | Later | +| NVFP4 block superkernel | Medium-high | Medium | Very high | Core component | +| Topology-autonomous planner | Medium | Low-medium | High | Supporting work | +| Conventional caching | Medium-low | Medium | High | Only with a novel error guarantee | + +## Landmark Result + +A compelling target is: + +| Scale | Target efficiency | Interconnect | +| --- | ---: | --- | +| 2 GPUs | Above `90%` | PCIe, no NVLink | +| 4 GPUs | Above `80%` | PCIe | +| 8 GPUs | Above `70%` | PCIe or cross-NUMA | +| Output | Reference-exact or formally bounded | All scales | +| Portability | SM100, SM120, SM121 | Same runtime | + +Also measure: + +- End-to-end latency, not sampling alone. +- Exposed communication time. +- Bytes transferred per block. +- HBM traffic and achieved bandwidth. +- Tensor-core utilization. +- Cost per generated second. +- Energy per generated second. +- PIX/NODE/PHB/SYS topology sensitivity. +- Comparison against SGLang, xDiT, and LightX2V. + +## Recommendation + +Do not lead with sparse attention yet. Lead with: + +> Reference-exact, communication-avoiding ragged Ulysses with direct NVFP4 +> projection-to-collective dataflow. + +The first fused modulation kernel establishes the exact-fusion infrastructure. +Use that infrastructure next to build the direct QKV-to-Ulysses path. + +The measured single-GPU implementation order is direct QKV-to-attention layout, +direct attention-to-output-projection layout, streamed NVFP4 scale/pack/GEMM, +and exact residual-gate epilogues. This order attacks the measured `105.83 ms` +layout boundary first while preserving the larger projection-to-collective +architecture as the distributed destination. + +If strong four-GPU and eight-GPU scaling over PCIe can be demonstrated while +retaining exact output, this stops being merely an excellent H3 runtime. It +becomes credible new research into how large generative models should be +executed without proprietary high-bandwidth interconnects. diff --git a/benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json b/benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json new file mode 100644 index 0000000..94e93c9 --- /dev/null +++ b/benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json @@ -0,0 +1,107 @@ +{ + "device": "NVIDIA B200", + "comparison_device": "NVIDIA RTX PRO 6000 Blackwell Server Edition", + "attention": "sdpa", + "mode": "tensor", + "world_size": 1, + "steps": 12, + "seed": 440420, + "torch": "2.9.1+cu130", + "hourly_usd": { + "b200": 6.79, + "rtx_pro_6000": 2.09 + }, + "b200_host": { + "data_center": "US-NC-2", + "driver": "580.105.08", + "memory_mib": 183359, + "power_limit_w": 1000 + }, + "results": [ + { + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "video_seconds_at_24fps": 5.166666666666667, + "b200": { + "model_load_seconds": 11.831760542932898, + "conditioning_seconds": 45.36778333503753, + "sampling_seconds": 79.72125827614218, + "sampling_cost_usd": 0.15036315102639039, + "sampling_peak_allocated_bytes": 17246854656, + "checksums": [ + -83553.09375, + -397.89813232421875 + ], + "console_step_seconds_rounded": [ + 6.4, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1, + 6.1 + ] + }, + "rtx_pro_6000": { + "sampling_seconds": 97.94385590963066, + "sampling_cost_usd": 0.05686184968086891 + }, + "b200_speedup": 1.2285789013812125, + "b200_sampling_cost_ratio": 2.6443591242685138 + }, + { + "resolution": [ + 1344, + 768 + ], + "frames": 243, + "video_seconds_at_24fps": 10.125, + "b200": { + "model_load_seconds": 10.712746233213693, + "conditioning_seconds": 38.57332478091121, + "sampling_seconds": 257.8619639207609, + "sampling_cost_usd": 0.48635631528387957, + "sampling_peak_allocated_bytes": 22955644416, + "checksums": [ + 41606.85546875, + 629.4112548828125 + ], + "console_step_seconds_rounded": [ + 21.4, + 21.1, + 21.1, + 21.1, + 21.0, + 21.0, + 21.0, + 21.0, + 21.0, + 21.0, + 21.0, + 21.0 + ] + }, + "rtx_pro_6000": { + "sampling_seconds": 311.01632468774915, + "sampling_cost_usd": 0.18056225516594326 + }, + "b200_speedup": 1.2061349411862938, + "b200_sampling_cost_ratio": 2.6935657999890426 + } + ], + "cost_scope": "Sampling wall time only. Model/Qwen loading, text conditioning, VAE decode, media encoding, muxing, and pod startup are excluded.", + "notes": [ + "The 124-frame run preceded the 243-frame run on the same host.", + "Both successful runs followed one pre-sampling dependency failure, so host and network-volume file caches were warm.", + "Console step durations are rounded to one decimal place; sampling_seconds is the authoritative synchronized benchmark timer.", + "The RTX PRO 6000 measurements are copied from rtxpro6000-server-1v2-sdpa-scaling-seed440420.json." + ] +} diff --git a/benchmarks/gb10-37810-token-optimization-profile-seed440420.json b/benchmarks/gb10-37810-token-optimization-profile-seed440420.json new file mode 100644 index 0000000..fdb32cf --- /dev/null +++ b/benchmarks/gb10-37810-token-optimization-profile-seed440420.json @@ -0,0 +1,75 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "torch": "2.9.1+cu130", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "packed_tokens": 37810, + "seed": 440420, + "attention": "sage2", + "representative_block_results": [ + { + "block": 0, + "comfy_scale_p50_seconds": 0.5251280819647945, + "vortex_scale_p50_seconds": 0.5088961640140042, + "speedup": 1.031896326006387, + "latency_reduction": 0.0309103978786619, + "checksum": -8279139.0 + }, + { + "block": 24, + "comfy_scale_p50_seconds": 0.5646577654988505, + "vortex_scale_p50_seconds": 0.5149167295021471, + "speedup": 1.0966001552227602, + "latency_reduction": 0.08809059050619694, + "checksum": 89071408.0 + }, + { + "block": 49, + "comfy_scale_p50_seconds": 0.524194426019676, + "vortex_scale_p50_seconds": 0.5067900969879702, + "speedup": 1.034342283196033, + "latency_reduction": 0.03320204902570367, + "checksum": -40617144320.0 + } + ], + "block_24_quantization_p50_milliseconds": { + "attn_qkv_proj": { + "comfy": 9.078, + "vortex": 3.908 + }, + "attn_out_proj": { + "comfy": 10.035, + "vortex": 5.151 + }, + "mlp_fc1": { + "comfy": 7.27, + "vortex": 5.182 + }, + "mlp_fc2": { + "comfy": 19.187, + "vortex": 10.163 + } + }, + "attention_findings": [ + "Sage2 was substantially faster than PyTorch SDPA, forced cuDNN SDPA, and FlashAttention-4 at 37,810 tokens.", + "Sage2 NHD produced an exact output but did not materially improve median total path time and had worse tail latency.", + "Strict Sol-Attn reached about 157 ms kernel time versus about 260 ms for Sage2, but its output differed materially from Sage2 and is not a correctness-preserving replacement.", + "Explicit Sage2 kernel variants did not provide an exact speed improvement over the Sage2 dispatcher." + ], + "validation": { + "linear_reference_diff": "zero for all four Vortex-scale plus Comfy-pack H3 linears", + "block_checksums": "identical for Comfy and Vortex paths at blocks 0, 24, and 49", + "remote_contracts": "28 passed", + "full_model_smoke": "not run: the deployed image predates the distributed runner; direct preview did not produce an artifact while resident services remained active" + }, + "notes": [ + "The optimization changes activation-scale discovery only; Comfy Kitchen still packs activations and executes scaled_mm_nvfp4.", + "The default H3_NVFP4_SCALE_BACKEND=torch path is unchanged.", + "The native Vortex packer is not enabled because the attention output projection failed parity in this profile.", + "Resident H3 and Qwen services remained running during profiling, so medians are preferred over means and tail latency requires an isolated rerun." + ] +} diff --git a/benchmarks/gb10-component2-layout-hnd-2step.json b/benchmarks/gb10-component2-layout-hnd-2step.json new file mode 100644 index 0000000..a0bce8e --- /dev/null +++ b/benchmarks/gb10-component2-layout-hnd-2step.json @@ -0,0 +1,22 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 2, + "seed": 440420, + "text_tokens": 100, + "elapsed_seconds": 52.49690314201871, + "stage_trace": [], + "checksums": [ + -276716.375, + 727.1075439453125 + ], + "peak_allocated_bytes": 18867826176, + "peak_reserved_bytes": 21846032384, + "measurement_policy": "uninstrumented sampling wall time" +} diff --git a/benchmarks/gb10-component2-layout-strided-nhd-12step.json b/benchmarks/gb10-component2-layout-strided-nhd-12step.json new file mode 100644 index 0000000..bfebf7d --- /dev/null +++ b/benchmarks/gb10-component2-layout-strided-nhd-12step.json @@ -0,0 +1,22 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 12, + "seed": 440420, + "text_tokens": 100, + "elapsed_seconds": 290.2269145210157, + "stage_trace": [], + "checksums": [ + -91481.7890625, + 678.401611328125 + ], + "peak_allocated_bytes": 17680283648, + "peak_reserved_bytes": 19809697792, + "measurement_policy": "uninstrumented sampling wall time" +} diff --git a/benchmarks/gb10-component2-layout-strided-nhd-2step.json b/benchmarks/gb10-component2-layout-strided-nhd-2step.json new file mode 100644 index 0000000..6d3fe16 --- /dev/null +++ b/benchmarks/gb10-component2-layout-strided-nhd-2step.json @@ -0,0 +1,22 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 2, + "seed": 440420, + "text_tokens": 100, + "elapsed_seconds": 50.95522483601235, + "stage_trace": [], + "checksums": [ + -276716.375, + 727.1075439453125 + ], + "peak_allocated_bytes": 17681987584, + "peak_reserved_bytes": 19809697792, + "measurement_policy": "uninstrumented sampling wall time" +} diff --git a/benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json b/benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json new file mode 100644 index 0000000..9196e44 --- /dev/null +++ b/benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json @@ -0,0 +1,69 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "block": 24, + "attention": "sage2", + "seed": 440420 + }, + "capture": { + "tool": "Nsight Systems 2025.3.2", + "scope": "one warmed H3 block between cudaProfilerStart and cudaProfilerStop", + "fused_elementwise": true, + "runtime_kernel_launches": 53, + "cudaLaunchKernel_calls": 45, + "cuLaunchKernelEx_calls": 8 + }, + "component1_residual": { + "modulation_launches": 2, + "modulation_total_ms": 9.5432, + "gate_add_launches": 2, + "gate_add_total_ms": 10.936032, + "unfused_modulation_gate_or_add_launches": 0, + "conclusion": "The four fused kernels perform substantial tensor traffic and are not launch-bound. Eliminate that traffic only at adjacent NVFP4 boundaries." + }, + "nvfp4": { + "launches": 32, + "gemm_launches": 4, + "gemm_total_ms": 162.339136, + "activation_pack_launches": 4, + "activation_pack_total_ms": 13.121248, + "absmax_partial_launches": 4, + "absmax_partial_total_ms": 10.767616, + "scale_finalize_launches": 4, + "scale_finalize_total_ms": 0.013536, + "block_scale_zero_launches": 4, + "block_scale_zero_total_ms": 0.292992, + "small_scale_conversion_launches": 12, + "intermediate_materialization_bytes_across_four_projections": 7732981760, + "conclusion": "Scale discovery, activation packing, buffer initialization, GEMM, and output allocation remain separate. The global scale dependency requires an explicit synchronization design in any exact fused implementation." + }, + "other_boundaries": { + "qkv_and_attention_layout_copy_launches": 4, + "qkv_and_attention_layout_copy_total_ms": 105.83168, + "swiglu_silu_total_ms": 9.390784, + "swiglu_multiply_total_ms": 14.38736 + }, + "native_pack_validation": { + "initial_finding": "The previous width-specific block-scale swizzle failed for the 7168-feature attention output projection while QDATA remained exact.", + "fix": "Replaced width-specific indexing with the general 128-row by 4-scale-column tiled offset.", + "validated_feature_widths": [5376, 7168, 14336], + "qdata_difference_count": 0, + "block_scale_difference_count": 0, + "linear_output_max_abs_difference": 0.0, + "deployment_decision": "Do not enable the standalone native packer for performance; it is not consistently faster than Comfy packing. Use it as exact infrastructure for the fused prototype." + }, + "validation": { + "deployed_tests": "33 passed", + "new_regression": "Comfy packed QDATA and block-scale identity at feature widths 5376, 7168, and 14336" + }, + "source_artifacts": [ + "benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json", + "benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json", + "benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json", + "benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json" + ] +} diff --git a/benchmarks/gb10-cute-nvfp4-conversion-contract.json b/benchmarks/gb10-cute-nvfp4-conversion-contract.json new file mode 100644 index 0000000..faa988e --- /dev/null +++ b/benchmarks/gb10-cute-nvfp4-conversion-contract.json @@ -0,0 +1,88 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "value_count": 41, + "padded_value_count": 48, + "fp4_torch_shape": [ + 48 + ], + "fp4_bytes": [ + 255, + 239, + 222, + 205, + 188, + 171, + 154, + 137, + 8, + 16, + 17, + 34, + 50, + 51, + 68, + 84, + 85, + 102, + 118, + 119, + 7, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0, + 0 + ], + "expected_low_first_bytes": [ + 255, + 239, + 222, + 205, + 188, + 171, + 154, + 137, + 8, + 16, + 17, + 34, + 50, + 51, + 68, + 84, + 85, + 102, + 118, + 119, + 7, + 0, + 0, + 0 + ], + "fp4_prefix_equal": true, + "fp8_equal": true, + "fp8_difference_count": 0 +} diff --git a/benchmarks/gb10-cute-nvfp4-tile-producer.json b/benchmarks/gb10-cute-nvfp4-tile-producer.json new file mode 100644 index 0000000..d26301d --- /dev/null +++ b/benchmarks/gb10-cute-nvfp4-tile-producer.json @@ -0,0 +1,92 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "tile": [ + 128, + 128 + ], + "all_equal": true, + "cases": [ + { + "name": "random", + "tensor_scale": 0.0013885498046875, + "fp4_difference_count": 0, + "fp4_equal": true, + "block_scale_difference_count": 0, + "block_scales_equal": true, + "scalar_block_scales_equal": true, + "actual_block_scale_bytes": [ + 107, + 110, + 111, + 112, + 113, + 114, + 115, + 116, + 117, + 118, + 119, + 120, + 121, + 122, + 123, + 124, + 125, + 126 + ], + "expected_block_scale_bytes": [ + 107, + 110, + 111, + 112, + 113, + 114, + 115, + 116, + 117, + 118, + 119, + 120, + 121, + 122, + 123, + 124, + 125, + 126 + ] + }, + { + "name": "zeros", + "tensor_scale": 0.0, + "fp4_difference_count": 0, + "fp4_equal": true, + "block_scale_difference_count": 0, + "block_scales_equal": true, + "scalar_block_scales_equal": true, + "actual_block_scale_bytes": [ + 126 + ], + "expected_block_scale_bytes": [ + 126 + ] + }, + { + "name": "sparse_extremes", + "tensor_scale": 0.037109375, + "fp4_difference_count": 0, + "fp4_equal": true, + "block_scale_difference_count": 0, + "block_scales_equal": true, + "scalar_block_scales_equal": true, + "actual_block_scale_bytes": [ + 0, + 126 + ], + "expected_block_scale_bytes": [ + 0, + 126 + ] + } + ] +} diff --git a/benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json b/benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json new file mode 100644 index 0000000..0f7af80 --- /dev/null +++ b/benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json @@ -0,0 +1,85 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "rows": 128, + "mnk": [ + 128, + 5376, + 7168 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "cute_shapes": { + "a": "(128, 7168, 1)", + "b": "(5376, 7168, 1)", + "sfa": "(128, 448, 1)", + "sfb": "(5376, 448, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 0.0263671875, + "b": 0.00072479248046875, + "alpha": 1.911073923110962e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": 257945.5, + "max_abs": 1368.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": 13503609856.0, + "candidate_checksum": 257945.5, + "equal": false, + "max_abs": 71826088.0, + "mean_abs": 2754388.25 + }, + "reference_checksum": 257945.5, + "candidate_checksum": 257945.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-attn-out-128rows.json b/benchmarks/gb10-cute-p0-attn-out-128rows.json new file mode 100644 index 0000000..9b82fbe --- /dev/null +++ b/benchmarks/gb10-cute-p0-attn-out-128rows.json @@ -0,0 +1,83 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "rows": 128, + "mnk": [ + 128, + 5376, + 7168 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "cute_shapes": { + "a": "(128, 7168, 1)", + "b": "(5376, 7168, 1)", + "sfa": "(128, 448, 1)", + "sfb": "(5376, 448, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 0.0263671875, + "b": 0.00072479248046875, + "alpha": 1.911073923110962e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": 13503609856.0, + "max_abs": 71827456.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "reference_checksum": 13503609856.0, + "candidate_checksum": 13503609856.0, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "reference_checksum": 257945.5, + "candidate_checksum": 258224.125, + "equal": false, + "max_abs": 8.0, + "mean_abs": 0.07462421804666519, + "relative_l2": 0.0028051051776856184, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json b/benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json new file mode 100644 index 0000000..e3c13cf --- /dev/null +++ b/benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json @@ -0,0 +1,85 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "rows": 128, + "mnk": [ + 128, + 21504, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(21504, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(21504, 336, 1)", + "c": "(128, 21504, 1)" + }, + "tensor_scales": { + "a": 0.0205078125, + "b": 0.00141143798828125, + "alpha": 2.8945505619049072e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -18469.02734375, + "max_abs": 187.0, + "nonzero": 2752509, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -640210368.0, + "candidate_checksum": -18469.02734375, + "equal": false, + "max_abs": 6455109.0, + "mean_abs": 455697.25 + }, + "reference_checksum": -18469.02734375, + "candidate_checksum": -18469.02734375, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-attn-qkv-128rows.json b/benchmarks/gb10-cute-p0-attn-qkv-128rows.json new file mode 100644 index 0000000..63104d9 --- /dev/null +++ b/benchmarks/gb10-cute-p0-attn-qkv-128rows.json @@ -0,0 +1,83 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "rows": 128, + "mnk": [ + 128, + 21504, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(21504, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(21504, 336, 1)", + "c": "(128, 21504, 1)" + }, + "tensor_scales": { + "a": 0.0205078125, + "b": 0.00141143798828125, + "alpha": 2.8945505619049072e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -640210368.0, + "max_abs": 6455296.0, + "nonzero": 2752509, + "finite": true + }, + "raw_blockscaled_parity": { + "reference_checksum": -640210368.0, + "candidate_checksum": -640210368.0, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "reference_checksum": -18469.02734375, + "candidate_checksum": -18533.14453125, + "equal": false, + "max_abs": 1.0, + "mean_abs": 0.018117837607860565, + "relative_l2": 0.002810996025800705, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-h3-summary.json b/benchmarks/gb10-cute-p0-h3-summary.json new file mode 100644 index 0000000..aab0ab5 --- /dev/null +++ b/benchmarks/gb10-cute-p0-h3-summary.json @@ -0,0 +1,62 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "cutlass_dsl": "4.6.2", + "rows": 128, + "kernel": "official persistent cooperative SM120/SM121 block-scaled GEMM", + "results": [ + { + "linear": "attn_qkv_proj", + "mnk": [128, 21504, 5376], + "raw_blockscaled_equal": true, + "raw_max_abs": 0.0, + "fused_alpha_equal": true, + "fused_alpha_max_abs": 0.0 + }, + { + "linear": "attn_out_proj", + "mnk": [128, 5376, 7168], + "raw_blockscaled_equal": true, + "raw_max_abs": 0.0, + "fused_alpha_equal": true, + "fused_alpha_max_abs": 0.0 + }, + { + "linear": "mlp_fc1", + "mnk": [128, 28672, 5376], + "raw_blockscaled_equal": true, + "raw_max_abs": 0.0, + "fused_alpha_equal": true, + "fused_alpha_max_abs": 0.0 + }, + { + "linear": "mlp_fc2", + "mnk": [128, 5376, 14336], + "raw_blockscaled_equal": false, + "raw_max_abs": 8192.0, + "raw_mean_abs": 0.01562502235174179, + "fused_alpha_equal": false, + "fused_alpha_max_abs": 16.0, + "fused_alpha_mean_abs": 0.000024163342459360138, + "tile_k_128_and_256_identical": true + } + ], + "findings": [ + "Comfy E2M1 values and E4M3 scale layouts are interoperable with CUTLASS after logical FP4 re-encoding.", + "The raw block-scaled operation is bit-exact for QKV, attention output, and FC1.", + "The custom alpha-before-BF16 epilogue is bit-exact for QKV, attention output, and FC1.", + "FC2 still requires matching the reference GEMM reduction policy; changing the stock K tile from 128 to 256 does not restore equality.", + "The official stock Float32 epilogue failed its own GB10 reference check and is not a valid accumulation oracle." + ], + "status": "P0 remains active; do not implement the streaming A producer until the exact alpha epilogue and FC2 reduction contract pass.", + "source_artifacts": [ + "benchmarks/gb10-cute-p0-attn-qkv-128rows.json", + "benchmarks/gb10-cute-p0-attn-out-128rows.json", + "benchmarks/gb10-cute-p0-mlp-fc1-128rows.json", + "benchmarks/gb10-cute-p0-mlp-fc2-128rows.json", + "benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json", + "benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json", + "benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json", + "benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json" + ] +} diff --git a/benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json b/benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json new file mode 100644 index 0000000..9581deb --- /dev/null +++ b/benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json @@ -0,0 +1,85 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "rows": 128, + "mnk": [ + 128, + 28672, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(28672, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(28672, 336, 1)", + "c": "(128, 28672, 1)" + }, + "tensor_scales": { + "a": 0.00494384765625, + "b": 0.0023651123046875, + "alpha": 1.1692754924297333e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -3814965.5, + "max_abs": 95.0, + "nonzero": 3670016, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -326266716160.0, + "candidate_checksum": -3814965.5, + "equal": false, + "max_abs": 8126369.0, + "mean_abs": 646601.3125 + }, + "reference_checksum": -3814965.5, + "candidate_checksum": -3814965.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-mlp-fc1-128rows.json b/benchmarks/gb10-cute-p0-mlp-fc1-128rows.json new file mode 100644 index 0000000..bf4ae56 --- /dev/null +++ b/benchmarks/gb10-cute-p0-mlp-fc1-128rows.json @@ -0,0 +1,83 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "rows": 128, + "mnk": [ + 128, + 28672, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(28672, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(28672, 336, 1)", + "c": "(128, 28672, 1)" + }, + "tensor_scales": { + "a": 0.00494384765625, + "b": 0.0023651123046875, + "alpha": 1.1692754924297333e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -326266716160.0, + "max_abs": 8126464.0, + "nonzero": 3670016, + "finite": true + }, + "raw_blockscaled_parity": { + "reference_checksum": -326266716160.0, + "candidate_checksum": -326266716160.0, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "reference_checksum": -3814965.5, + "candidate_checksum": -3814898.0, + "equal": false, + "max_abs": 0.5, + "mean_abs": 0.010720730759203434, + "relative_l2": 0.0028073240537196398, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json b/benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json new file mode 100644 index 0000000..f23e59d --- /dev/null +++ b/benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json @@ -0,0 +1,85 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc2", + "rows": 128, + "mnk": [ + 128, + 5376, + 14336 + ], + "tile_shape_mnk": [ + 128, + 128, + 256 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "cute_shapes": { + "a": "(128, 14336, 1)", + "b": "(5376, 14336, 1)", + "sfa": "(128, 896, 1)", + "sfb": "(5376, 896, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 1.1953125, + "b": 0.001129150390625, + "alpha": 0.0013496875762939453 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -1426078.5, + "max_abs": 14208.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -1057548288.0, + "candidate_checksum": -1426078.5, + "equal": false, + "max_abs": 10537088.0, + "mean_abs": 838238.5 + }, + "reference_checksum": -1426062.125, + "candidate_checksum": -1426078.5, + "equal": false, + "max_abs": 16.0, + "mean_abs": 2.4163342459360138e-05, + "relative_l2": 1.2930971934110858e-05, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p0-mlp-fc2-128rows.json b/benchmarks/gb10-cute-p0-mlp-fc2-128rows.json new file mode 100644 index 0000000..615ec93 --- /dev/null +++ b/benchmarks/gb10-cute-p0-mlp-fc2-128rows.json @@ -0,0 +1,83 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc2", + "rows": 128, + "mnk": [ + 128, + 5376, + 14336 + ], + "tile_shape_mnk": [ + 128, + 128, + 256 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "cute_shapes": { + "a": "(128, 14336, 1)", + "b": "(5376, 14336, 1)", + "sfa": "(128, 896, 1)", + "sfb": "(5376, 896, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 1.1953125, + "b": 0.001129150390625, + "alpha": 0.0013496875762939453 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -1057541632.0, + "max_abs": 10551296.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "reference_checksum": -1057548288.0, + "candidate_checksum": -1057541632.0, + "equal": false, + "max_abs": 8192.0, + "mean_abs": 0.01562502235174179 + }, + "reference_checksum": -1426062.125, + "candidate_checksum": -1428981.125, + "equal": false, + "max_abs": 64.0, + "mean_abs": 1.5995242595672607, + "relative_l2": 0.002791827078908682, + "numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-real-tiles-attn-out.json b/benchmarks/gb10-cute-p1-real-tiles-attn-out.json new file mode 100644 index 0000000..95b5295 --- /dev/null +++ b/benchmarks/gb10-cute-p1-real-tiles-attn-out.json @@ -0,0 +1,46 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "activation_shape": [ + 128, + 7168 + ], + "tensor_scale": 0.0263671875, + "tile_count": 56, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "equal": true, + "differing_tiles": [], + "difference_examples": [] +} diff --git a/benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json b/benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json new file mode 100644 index 0000000..2461e68 --- /dev/null +++ b/benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json @@ -0,0 +1,46 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "activation_shape": [ + 128, + 5376 + ], + "tensor_scale": 0.0205078125, + "tile_count": 42, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "equal": true, + "differing_tiles": [], + "difference_examples": [] +} diff --git a/benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json b/benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json new file mode 100644 index 0000000..24ba8dd --- /dev/null +++ b/benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json @@ -0,0 +1,46 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "activation_shape": [ + 128, + 5376 + ], + "tensor_scale": 0.00494384765625, + "tile_count": 42, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "equal": true, + "differing_tiles": [], + "difference_examples": [] +} diff --git a/benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json b/benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json new file mode 100644 index 0000000..8392404 --- /dev/null +++ b/benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json @@ -0,0 +1,86 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "rows": 128, + "mnk": [ + 128, + 5376, + 7168 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "cute_shapes": { + "a": "(128, 7168, 1)", + "b": "(5376, 7168, 1)", + "sfa": "(1,)", + "sfb": "(5376, 448, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 0.0263671875, + "b": 0.00072479248046875, + "alpha": 1.911073923110962e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": 257945.5, + "max_abs": 1368.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": 13503609856.0, + "candidate_checksum": 257945.5, + "equal": false, + "max_abs": 71826088.0, + "mean_abs": 2754388.25 + }, + "reference_checksum": 257945.5, + "candidate_checksum": 257945.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json b/benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json new file mode 100644 index 0000000..3485f10 --- /dev/null +++ b/benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json @@ -0,0 +1,86 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "rows": 128, + "mnk": [ + 128, + 21504, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(21504, 5376, 1)", + "sfa": "(1,)", + "sfb": "(21504, 336, 1)", + "c": "(128, 21504, 1)" + }, + "tensor_scales": { + "a": 0.0205078125, + "b": 0.00141143798828125, + "alpha": 2.8945505619049072e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -18469.02734375, + "max_abs": 187.0, + "nonzero": 2752509, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -640210368.0, + "candidate_checksum": -18469.02734375, + "equal": false, + "max_abs": 6455109.0, + "mean_abs": 455697.25 + }, + "reference_checksum": -18469.02734375, + "candidate_checksum": -18469.02734375, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json b/benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json new file mode 100644 index 0000000..7cd7539 --- /dev/null +++ b/benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json @@ -0,0 +1,86 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "rows": 128, + "mnk": [ + 128, + 28672, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(28672, 5376, 1)", + "sfa": "(1,)", + "sfb": "(28672, 336, 1)", + "c": "(128, 28672, 1)" + }, + "tensor_scales": { + "a": 0.00494384765625, + "b": 0.0023651123046875, + "alpha": 1.1692754924297333e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -3814965.5, + "max_abs": 95.0, + "nonzero": 3670016, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -326266716160.0, + "candidate_checksum": -3814965.5, + "equal": false, + "max_abs": 8126369.0, + "mean_abs": 646601.3125 + }, + "reference_checksum": -3814965.5, + "candidate_checksum": -3814965.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-stream-a-summary.json b/benchmarks/gb10-cute-p1-stream-a-summary.json new file mode 100644 index 0000000..f6068b4 --- /dev/null +++ b/benchmarks/gb10-cute-p1-stream-a-summary.json @@ -0,0 +1,54 @@ +{ + "date": "2026-08-23", + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "tile_shape_mnk": [128, 128, 128], + "producer_threads": 32, + "rows_per_producer_thread": 4, + "activation_input": "BF16", + "global_activation_qdata": false, + "global_activation_sfa": false, + "retained_tma_inputs": ["B", "SFB"], + "streamed_roles": { + "attn_qkv_proj": { + "mnk": [128, 21504, 5376], + "packed_fp4_differences": 0, + "block_scale_differences": 0, + "bf16_output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0 + }, + "attn_out_proj": { + "mnk": [128, 5376, 7168], + "packed_fp4_differences": 0, + "block_scale_differences": 0, + "bf16_output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0 + }, + "mlp_fc1": { + "mnk": [128, 28672, 5376], + "packed_fp4_differences": 0, + "block_scale_differences": 0, + "bf16_output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0 + } + }, + "mlp_fc2": { + "streaming_enabled": false, + "fallback": "Comfy/cuBLAS", + "validator_rejects_streaming": true + }, + "numerical_contract": { + "block_scale_order": "(amax / 6.0f) / global_scale", + "block_scale_clamp": "fminf(value, 448.0f)", + "encode_scale": "fminf(rcp.approx.ftz.f32(rounded_scale * global_scale), FLT_MAX)", + "fp4_conversion": "CuTe E2M1 round-to-nearest ties-to-even", + "epilogue": "FP32 global alpha before BF16 conversion" + }, + "status": "Fixed 128-row P1 producer-consumer checkpoint is exact; not runtime-integrated or performance-approved." +} diff --git a/benchmarks/gb10-cute-p1-stream-a-timing-summary.json b/benchmarks/gb10-cute-p1-stream-a-timing-summary.json new file mode 100644 index 0000000..3ed7449 --- /dev/null +++ b/benchmarks/gb10-cute-p1-stream-a-timing-summary.json @@ -0,0 +1,50 @@ +{ + "date": "2026-08-23", + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "rows": 128, + "tile_shape_mnk": [128, 128, 128], + "warmup": 5, + "iterations": 20, + "results": { + "attn_qkv_proj": { + "n_tiles": 168, + "k_tiles": 42, + "prepacked_gemm_ms": 0.4769120216369629, + "activation_quantize_ms": 0.05241439938545227, + "reference_complete_projection_ms": 0.49579682350158694, + "streamed_gemm_ms": 4.529531097412109, + "streamed_vs_prepacked": 9.497624073020535, + "streamed_vs_reference_complete": 9.135861471281917, + "repeated_producer_overhead_ms": 4.052619075775146, + "minimum_theoretical_n_reuse_to_cover_quantize_cost": 78 + }, + "attn_out_proj": { + "n_tiles": 42, + "k_tiles": 56, + "prepacked_gemm_ms": 0.054211199283599854, + "activation_quantize_ms": 0.05055999755859375, + "reference_complete_projection_ms": 0.11322239637374878, + "streamed_gemm_ms": 1.4691216468811035, + "streamed_vs_prepacked": 27.09996580587633, + "streamed_vs_reference_complete": 12.975539238999248, + "repeated_producer_overhead_ms": 1.4149104475975036, + "minimum_theoretical_n_reuse_to_cover_quantize_cost": 28 + }, + "mlp_fc1": { + "n_tiles": 224, + "k_tiles": 42, + "prepacked_gemm_ms": 0.46705121994018556, + "activation_quantize_ms": 0.051630401611328126, + "reference_complete_projection_ms": 0.45939040184020996, + "streamed_gemm_ms": 5.716193771362304, + "streamed_vs_prepacked": 12.238901275312733, + "streamed_vs_reference_complete": 12.442997825954953, + "repeated_producer_overhead_ms": 5.249142551422119, + "minimum_theoretical_n_reuse_to_cover_quantize_cost": 102 + } + }, + "producer_overhead_per_n_k_tile_ms_range": [0.0005579445739181673, 0.000601577571257442], + "decision": "Reject one-A-producer-per-output-N-CTA scheduling. Required N reuse factors of 28-102 are not practical with duplicate accumulator state. Prototype a bounded global packed-tile ring or persistent work queue before canonical-shape runtime integration.", + "numerical_status": "All timed streamed outputs remain bit-exact." +} diff --git a/benchmarks/gb10-cute-p1-timing-p0-attn-out.json b/benchmarks/gb10-cute-p1-timing-p0-attn-out.json new file mode 100644 index 0000000..43ef5a0 --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-p0-attn-out.json @@ -0,0 +1,98 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "rows": 128, + "mnk": [ + 128, + 5376, + 7168 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": false, + "streamed_activation_materialization": null, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 1.084223985671997, + "mean_ms": 0.054211199283599854 + }, + "reference_timing": { + "backend": "vortex_scale_plus_comfy_pack_gemm", + "activation_quantize_mean_ms": 0.05055999755859375, + "complete_projection_mean_ms": 0.11322239637374878 + }, + "cute_shapes": { + "a": "(128, 7168, 1)", + "b": "(5376, 7168, 1)", + "sfa": "(128, 448, 1)", + "sfb": "(5376, 448, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 0.0263671875, + "b": 0.00072479248046875, + "alpha": 1.911073923110962e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": 257945.5, + "max_abs": 1368.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": 13503609856.0, + "candidate_checksum": 257945.5, + "equal": false, + "max_abs": 71826088.0, + "mean_abs": 2754388.25 + }, + "reference_checksum": 257945.5, + "candidate_checksum": 257945.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json b/benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json new file mode 100644 index 0000000..468be4a --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json @@ -0,0 +1,98 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "rows": 128, + "mnk": [ + 128, + 21504, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": false, + "streamed_activation_materialization": null, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 9.538240432739258, + "mean_ms": 0.4769120216369629 + }, + "reference_timing": { + "backend": "vortex_scale_plus_comfy_pack_gemm", + "activation_quantize_mean_ms": 0.05241439938545227, + "complete_projection_mean_ms": 0.49579682350158694 + }, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(21504, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(21504, 336, 1)", + "c": "(128, 21504, 1)" + }, + "tensor_scales": { + "a": 0.0205078125, + "b": 0.00141143798828125, + "alpha": 2.8945505619049072e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -18469.02734375, + "max_abs": 187.0, + "nonzero": 2752509, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -640210368.0, + "candidate_checksum": -18469.02734375, + "equal": false, + "max_abs": 6455109.0, + "mean_abs": 455697.25 + }, + "reference_checksum": -18469.02734375, + "candidate_checksum": -18469.02734375, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json b/benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json new file mode 100644 index 0000000..f469fe1 --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json @@ -0,0 +1,98 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "rows": 128, + "mnk": [ + 128, + 28672, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": false, + "streamed_activation_materialization": null, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 9.341024398803711, + "mean_ms": 0.46705121994018556 + }, + "reference_timing": { + "backend": "vortex_scale_plus_comfy_pack_gemm", + "activation_quantize_mean_ms": 0.051630401611328126, + "complete_projection_mean_ms": 0.45939040184020996 + }, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(28672, 5376, 1)", + "sfa": "(128, 336, 1)", + "sfb": "(28672, 336, 1)", + "c": "(128, 28672, 1)" + }, + "tensor_scales": { + "a": 0.00494384765625, + "b": 0.0023651123046875, + "alpha": 1.1692754924297333e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -3814965.5, + "max_abs": 95.0, + "nonzero": 3670016, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -326266716160.0, + "candidate_checksum": -3814965.5, + "equal": false, + "max_abs": 8126369.0, + "mean_abs": 646601.3125 + }, + "reference_checksum": -3814965.5, + "candidate_checksum": -3814965.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-timing-stream-attn-out.json b/benchmarks/gb10-cute-p1-timing-stream-attn-out.json new file mode 100644 index 0000000..17d4a5d --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-stream-attn-out.json @@ -0,0 +1,96 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "rows": 128, + "mnk": [ + 128, + 5376, + 7168 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "streamed_activation_materialization": { + "global_qdata": false, + "global_sfa": false + }, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 29.38243293762207, + "mean_ms": 1.4691216468811035 + }, + "cute_shapes": { + "a": "(128, 7168, 1)", + "b": "(5376, 7168, 1)", + "sfa": "(1,)", + "sfb": "(5376, 448, 1)", + "c": "(128, 5376, 1)" + }, + "tensor_scales": { + "a": 0.0263671875, + "b": 0.00072479248046875, + "alpha": 1.911073923110962e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": 257945.5, + "max_abs": 1368.0, + "nonzero": 688128, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": 13503609856.0, + "candidate_checksum": 257945.5, + "equal": false, + "max_abs": 71826088.0, + "mean_abs": 2754388.25 + }, + "reference_checksum": 257945.5, + "candidate_checksum": 257945.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json b/benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json new file mode 100644 index 0000000..aaadfe0 --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json @@ -0,0 +1,96 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "rows": 128, + "mnk": [ + 128, + 21504, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "streamed_activation_materialization": { + "global_qdata": false, + "global_sfa": false + }, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 90.59062194824219, + "mean_ms": 4.529531097412109 + }, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(21504, 5376, 1)", + "sfa": "(1,)", + "sfb": "(21504, 336, 1)", + "c": "(128, 21504, 1)" + }, + "tensor_scales": { + "a": 0.0205078125, + "b": 0.00141143798828125, + "alpha": 2.8945505619049072e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -18469.02734375, + "max_abs": 187.0, + "nonzero": 2752509, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -640210368.0, + "candidate_checksum": -18469.02734375, + "equal": false, + "max_abs": 6455109.0, + "mean_abs": 455697.25 + }, + "reference_checksum": -18469.02734375, + "candidate_checksum": -18469.02734375, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json b/benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json new file mode 100644 index 0000000..759d1bb --- /dev/null +++ b/benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json @@ -0,0 +1,96 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "rows": 128, + "mnk": [ + 128, + 28672, + 5376 + ], + "tile_shape_mnk": [ + 128, + 128, + 128 + ], + "swap_nibbles": false, + "reencode_fp4": true, + "zero_a": false, + "fuse_alpha": true, + "stream_a": true, + "streamed_activation_materialization": { + "global_qdata": false, + "global_sfa": false + }, + "timing": { + "warmup": 5, + "iterations": 20, + "total_ms": 114.3238754272461, + "mean_ms": 5.716193771362304 + }, + "cute_shapes": { + "a": "(128, 5376, 1)", + "b": "(28672, 5376, 1)", + "sfa": "(1,)", + "sfb": "(28672, 336, 1)", + "c": "(128, 28672, 1)" + }, + "tensor_scales": { + "a": 0.00494384765625, + "b": 0.0023651123046875, + "alpha": 1.1692754924297333e-05 + }, + "raw_output": { + "dtype": "torch.bfloat16", + "checksum": -3814965.5, + "max_abs": 95.0, + "nonzero": 3670016, + "finite": true + }, + "raw_blockscaled_parity": { + "applicable": false, + "reference_checksum": -326266716160.0, + "candidate_checksum": -3814965.5, + "equal": false, + "max_abs": 8126369.0, + "mean_abs": 646601.3125 + }, + "reference_checksum": -3814965.5, + "candidate_checksum": -3814965.5, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "relative_l2": 0.0, + "numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." +} diff --git a/benchmarks/gb10-cute-p2-ring-attn-out.json b/benchmarks/gb10-cute-p2-ring-attn-out.json new file mode 100644 index 0000000..751de7f --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-attn-out.json @@ -0,0 +1,67 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 128, + 5376, + 7168 + ], + "ring": { + "row_capacity": 128, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 458752, + "sfa_bytes": 57344, + "logical_bytes": 516096 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.010279680490493775, + "global_scale_ms": 0.01211359977722168, + "cute_consumer_ms": 0.03870527982711792, + "modeled_ring_ms": 0.06109856009483337, + "actual_into_ring_cute_gemm_ms": 0.09652607917785644, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.10898783683776855, + "modeled_ring_vs_reference": 0.5605998051487184, + "actual_ring_vs_reference": 0.8856591889381034 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-attn-qkv.json b/benchmarks/gb10-cute-p2-ring-attn-qkv.json new file mode 100644 index 0000000..411026c --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-attn-qkv.json @@ -0,0 +1,67 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "mnk": [ + 128, + 21504, + 5376 + ], + "ring": { + "row_capacity": 128, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 344064, + "sfa_bytes": 43008, + "logical_bytes": 387072 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.015116159915924072, + "global_scale_ms": 0.017698240280151368, + "cute_consumer_ms": 0.4326623916625977, + "modeled_ring_ms": 0.46547679185867313, + "actual_into_ring_cute_gemm_ms": 0.4637001419067383, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.47588897705078126, + "modeled_ring_vs_reference": 0.978120558167505, + "actual_ring_vs_reference": 0.9743872295181523 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json new file mode 100644 index 0000000..8827f20 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 1024, + 5376, + 7168 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 37, + "ring": { + "row_capacity": 1024, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 3670016, + "sfa_bytes": 458752, + "logical_bytes": 4128768 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 50, + "producer_ms": 0.08987903594970703, + "global_scale_ms": 4.179701232910157, + "cute_consumer_ms": 0.44698047637939453, + "modeled_cute_chunk_ms": 0.5368595123291016, + "modeled_comfy_chunk_ms": 0.5336019134521484, + "actual_into_ring_cute_gemm_ms": 0.5717164611816407, + "comfy_consumer_ms": 0.4437228775024414, + "actual_into_ring_comfy_gemm_ms": 0.6249113464355469, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.5978803253173828, + "modeled_canonical_reference_ms": 26.301273269653322, + "modeled_canonical_cute_ring_ms": 25.33321029663086, + "modeled_canonical_comfy_ring_ms": 27.301421051025393, + "actual_ring_vs_reference": 0.9562389611635855, + "comfy_ring_vs_reference": 1.0452114243829898, + "modeled_canonical_cute_ring_vs_reference": 0.9631933038717398, + "modeled_canonical_comfy_ring_vs_reference": 1.0380265917591927 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json new file mode 100644 index 0000000..47f45a8 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json @@ -0,0 +1,72 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 128, + 5376, + 7168 + ], + "full_activation_rows": 37810, + "ring": { + "row_capacity": 128, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 458752, + "sfa_bytes": 57344, + "logical_bytes": 516096 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.019654719829559325, + "global_scale_ms": 3.735771484375, + "cute_consumer_ms": 0.07179071903228759, + "modeled_ring_ms": 3.8272169232368465, + "actual_into_ring_cute_gemm_ms": 0.09406271934509278, + "comfy_consumer_ms": 0.05615776062011719, + "actual_into_ring_comfy_gemm_ms": 0.07875840187072754, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.08742112159729004, + "modeled_ring_vs_reference": 43.77908740255154, + "actual_ring_vs_reference": 1.0759724609619812, + "comfy_ring_vs_reference": 0.9009081607707142 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json new file mode 100644 index 0000000..32917bc --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 2048, + 5376, + 7168 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 19, + "ring": { + "row_capacity": 2048, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 7340032, + "sfa_bytes": 917504, + "logical_bytes": 8257536 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.22272863388061523, + "global_scale_ms": 3.323794250488281, + "cute_consumer_ms": 0.775920639038086, + "modeled_cute_chunk_ms": 0.9986492729187012, + "modeled_comfy_chunk_ms": 1.0811997032165528, + "actual_into_ring_cute_gemm_ms": 1.008663330078125, + "comfy_consumer_ms": 0.8584710693359375, + "actual_into_ring_comfy_gemm_ms": 1.1376866912841797, + "reference_vortex_scale_comfy_pack_gemm_ms": 1.178064956665039, + "modeled_canonical_reference_ms": 25.70702842712402, + "modeled_canonical_cute_ring_ms": 22.488397521972654, + "modeled_canonical_comfy_ring_ms": 24.939841384887693, + "actual_ring_vs_reference": 0.8562034923214508, + "comfy_ring_vs_reference": 0.9657249244598827, + "modeled_canonical_cute_ring_vs_reference": 0.8747956842123643, + "modeled_canonical_comfy_ring_vs_reference": 0.9701565256983631 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json new file mode 100644 index 0000000..81052ec --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 4096, + 5376, + 7168 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 10, + "ring": { + "row_capacity": 4096, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 14680064, + "sfa_bytes": 1835008, + "logical_bytes": 16515072 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 50, + "producer_ms": 0.3313638305664062, + "global_scale_ms": 2.197855987548828, + "cute_consumer_ms": 1.3521932983398437, + "modeled_cute_chunk_ms": 1.6835571289062499, + "modeled_comfy_chunk_ms": 1.7651193237304688, + "actual_into_ring_cute_gemm_ms": 1.8329510498046875, + "comfy_consumer_ms": 1.4337554931640626, + "actual_into_ring_comfy_gemm_ms": 1.6257913208007813, + "reference_vortex_scale_comfy_pack_gemm_ms": 1.596439666748047, + "modeled_canonical_reference_ms": 18.1622526550293, + "modeled_canonical_cute_ring_ms": 20.527366485595703, + "modeled_canonical_comfy_ring_ms": 18.45576919555664, + "actual_ring_vs_reference": 1.148149277409534, + "comfy_ring_vs_reference": 1.0183856957854998, + "modeled_canonical_cute_ring_vs_reference": 1.130221392438976, + "modeled_canonical_comfy_ring_vs_reference": 1.0161608004305602 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json new file mode 100644 index 0000000..19600bf --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_out_proj", + "mnk": [ + 512, + 5376, + 7168 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 74, + "ring": { + "row_capacity": 512, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 1835008, + "sfa_bytes": 229376, + "logical_bytes": 2064384 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 50, + "producer_ms": 0.029301760196685792, + "global_scale_ms": 2.2483973693847656, + "cute_consumer_ms": 0.14257984161376952, + "modeled_cute_chunk_ms": 0.1718816018104553, + "modeled_comfy_chunk_ms": 0.2101740860939026, + "actual_into_ring_cute_gemm_ms": 0.20412223815917968, + "comfy_consumer_ms": 0.1808723258972168, + "actual_into_ring_comfy_gemm_ms": 0.21875711441040038, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.21859840393066407, + "modeled_canonical_reference_ms": 18.424679260253907, + "modeled_canonical_cute_ring_ms": 17.353442993164062, + "modeled_canonical_comfy_ring_ms": 18.436423835754393, + "actual_ring_vs_reference": 0.93377734918835, + "comfy_ring_vs_reference": 1.0007260367728332, + "modeled_canonical_cute_ring_vs_reference": 0.9418586206056386, + "modeled_canonical_comfy_ring_vs_reference": 1.0006374371751383 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json b/benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json new file mode 100644 index 0000000..904ce11 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "attn_qkv_proj", + "mnk": [ + 2048, + 21504, + 5376 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 19, + "ring": { + "row_capacity": 2048, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 5505024, + "sfa_bytes": 688128, + "logical_bytes": 6193152 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.12289536476135254, + "global_scale_ms": 1.8337306213378906, + "cute_consumer_ms": 2.898988037109375, + "modeled_cute_chunk_ms": 3.0218834018707272, + "modeled_comfy_chunk_ms": 1.69343674659729, + "actual_into_ring_cute_gemm_ms": 1.5230323791503906, + "comfy_consumer_ms": 1.5705413818359375, + "actual_into_ring_comfy_gemm_ms": 1.7044099426269532, + "reference_vortex_scale_comfy_pack_gemm_ms": 1.7054486083984375, + "modeled_canonical_reference_ms": 34.2372541809082, + "modeled_canonical_cute_ring_ms": 30.77134582519531, + "modeled_canonical_comfy_ring_ms": 34.217519531250005, + "actual_ring_vs_reference": 0.8930391520742736, + "comfy_ring_vs_reference": 0.9993909721076499, + "modeled_canonical_cute_ring_vs_reference": 0.89876792287725, + "modeled_canonical_comfy_ring_vs_reference": 0.999423591344273 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json b/benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json new file mode 100644 index 0000000..c3b67e8 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json @@ -0,0 +1,78 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "mnk": [ + 2048, + 28672, + 5376 + ], + "full_activation_rows": 37810, + "modeled_chunk_count": 19, + "ring": { + "row_capacity": 2048, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 5505024, + "sfa_bytes": 688128, + "logical_bytes": 6193152 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "comfy_output_equal": true + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.13199872016906739, + "global_scale_ms": 2.422532806396484, + "cute_consumer_ms": 6.302136840820313, + "modeled_cute_chunk_ms": 6.434135560989381, + "modeled_comfy_chunk_ms": 2.247994508743286, + "actual_into_ring_cute_gemm_ms": 2.0069894409179687, + "comfy_consumer_ms": 2.1159957885742187, + "actual_into_ring_comfy_gemm_ms": 2.2663865661621094, + "reference_vortex_scale_comfy_pack_gemm_ms": 2.26166748046875, + "modeled_canonical_reference_ms": 45.394214935302735, + "modeled_canonical_cute_ring_ms": 40.55533218383789, + "modeled_canonical_comfy_ring_ms": 45.48387756347656, + "actual_ring_vs_reference": 0.8873936855218004, + "comfy_ring_vs_reference": 1.0020865515086157, + "modeled_canonical_cute_ring_vs_reference": 0.8934030964438668, + "modeled_canonical_comfy_ring_vs_reference": 1.0019751994456036 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-capacity-summary.json b/benchmarks/gb10-cute-p2-ring-capacity-summary.json new file mode 100644 index 0000000..6c9fe22 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-capacity-summary.json @@ -0,0 +1,58 @@ +{ + "date": "2026-08-23", + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "full_activation_rows": 37810, + "global_scale_policy": "one full-activation Vortex scale reused by every ring chunk", + "attention_output_capacity_sweep": { + "128": { + "ring_bytes": 516096, + "cute_ring_vs_reference": 1.0759724609619812 + }, + "512": { + "ring_bytes": 2064384, + "cute_ring_vs_reference": 0.93377734918835 + }, + "1024": { + "ring_bytes": 4128768, + "cute_ring_vs_reference": 0.9562389611635855 + }, + "2048": { + "ring_bytes": 8257536, + "cute_ring_vs_reference": 0.8562034923214508, + "modeled_canonical_vs_reference": 0.8747956842123643 + }, + "4096": { + "ring_bytes": 16515072, + "cute_ring_vs_reference": 1.148149277409534 + } + }, + "selected_row_capacity": 2048, + "selected_capacity_results": { + "attn_qkv_proj": { + "ring_bytes": 6193152, + "chunk_improvement_percent": 10.69608479257264, + "modeled_canonical_improvement_percent": 10.123207712275, + "output_equal": true + }, + "attn_out_proj": { + "ring_bytes": 8257536, + "chunk_improvement_percent": 14.37965076785492, + "modeled_canonical_improvement_percent": 12.52043157876357, + "output_equal": true + }, + "mlp_fc1": { + "ring_bytes": 6193152, + "chunk_improvement_percent": 11.26063144781996, + "modeled_canonical_improvement_percent": 10.65969035561332, + "output_equal": true + } + }, + "policy": { + "attn_qkv_proj": "advance to real-block validation", + "attn_out_proj": "advance to real-block validation", + "mlp_fc1": "advance to real-block validation", + "mlp_fc2": "retain Comfy/cuBLAS fallback" + }, + "next_gate": "Execute 2048-row chunks across complete real projections in blocks 0, 24, and 49, including the final partial chunk, and compare total projection/block latency and exact outputs." +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json b/benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json new file mode 100644 index 0000000..45648e2 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 0, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 25.982784271240234, + "projection_ring_total_ms": 14.904607772827148, + "results": [ + { + "name": "attn_out_proj", + "mnk": [ + 37810, + 5376, + 7168 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 8257536, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": -55638368.0, + "candidate_checksum": -55638368.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 14.904607772827148, + "reference_complete_ms": 25.982784271240234, + "ring_vs_reference": 0.5736339730659553, + "improvement_percent": 42.63660269340447 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 418381824 + } + ], + "projection_total_improvement_percent": 42.63660269340447 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block0-fc1.json b/benchmarks/gb10-cute-p2-ring-full-block0-fc1.json new file mode 100644 index 0000000..29a38eb --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block0-fc1.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 0, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 217.1246795654297, + "projection_ring_total_ms": 176.6940155029297, + "results": [ + { + "name": "mlp_fc1", + "mnk": [ + 37810, + 28672, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": -327274368.0, + "candidate_checksum": -327274368.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 176.6940155029297, + "reference_complete_ms": 217.1246795654297, + "ring_vs_reference": 0.8137905642812181, + "improvement_percent": 18.62094357187819 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 2231369728 + } + ], + "projection_total_improvement_percent": 18.62094357187819 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block0-qkv.json b/benchmarks/gb10-cute-p2-ring-full-block0-qkv.json new file mode 100644 index 0000000..e726156 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block0-qkv.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 0, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 79.74476623535156, + "projection_ring_total_ms": 32.21440124511719, + "results": [ + { + "name": "attn_qkv_proj", + "mnk": [ + 37810, + 21504, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": -17431596.0, + "candidate_checksum": -17431596.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 32.21440124511719, + "reference_complete_ms": 79.74476623535156, + "ring_vs_reference": 0.40396884668321037, + "improvement_percent": 59.60311533167897 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 1673527296 + } + ], + "projection_total_improvement_percent": 59.60311533167897 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json new file mode 100644 index 0000000..078c2a7 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json @@ -0,0 +1,330 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "capacity": 1024, + "all_equal": true, + "projection_reference_total_ms": 19.279925028483074, + "projection_ring_total_ms": 21.812459309895832, + "results": [ + { + "name": "attn_out_proj", + "mnk": [ + 37810, + 5376, + 7168 + ], + "capacity": 1024, + "chunk_count": 37, + "final_chunk_rows": 946, + "ring_bytes": 4128768, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 81086272.0, + "candidate_checksum": 81086267.875 + }, + "timing": { + "warmup": 1, + "iterations": 3, + "ring_complete_ms": 21.812459309895832, + "reference_complete_ms": 19.279925028483074, + "ring_vs_reference": 1.1313560233077324, + "improvement_percent": -13.135602330773244 + }, + "chunks": [ + { + "start": 0, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 1024, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 3072, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 5120, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 7168, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 9216, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 11264, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 13312, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 15360, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 17408, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 19456, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 21504, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 23552, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 25600, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 27648, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 29696, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 31744, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 33792, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 35840, + "rows": 1024, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ] + } + ], + "projection_total_improvement_percent": -13.135602330773244 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json new file mode 100644 index 0000000..0f62315 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json @@ -0,0 +1,141 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "capacity": 4096, + "all_equal": true, + "projection_reference_total_ms": 16.191519419352215, + "projection_ring_total_ms": 18.844778696695965, + "results": [ + { + "name": "attn_out_proj", + "mnk": [ + 37810, + 5376, + 7168 + ], + "capacity": 4096, + "chunk_count": 10, + "final_chunk_rows": 946, + "ring_bytes": 16515072, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 81086272.0, + "candidate_checksum": 81086266.0 + }, + "timing": { + "warmup": 1, + "iterations": 3, + "ring_complete_ms": 18.844778696695965, + "reference_complete_ms": 16.191519419352215, + "ring_vs_reference": 1.1638672201555438, + "improvement_percent": -16.386722015554376 + }, + "chunks": [ + { + "start": 0, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 4096, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ] + } + ], + "projection_total_improvement_percent": -16.386722015554376 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json new file mode 100644 index 0000000..663e2fd --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 23.377840042114258, + "projection_ring_total_ms": 24.269023895263672, + "results": [ + { + "name": "attn_out_proj", + "mnk": [ + 37810, + 5376, + 7168 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 8257536, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 81086272.0, + "candidate_checksum": 81086272.0 + }, + "timing": { + "warmup": 1, + "iterations": 2, + "ring_complete_ms": 24.269023895263672, + "reference_complete_ms": 23.377840042114258, + "ring_vs_reference": 1.0381208807804305, + "improvement_percent": -3.812088078043052 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 418381824 + } + ], + "projection_total_improvement_percent": -3.812088078043052 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block24-fc1.json b/benchmarks/gb10-cute-p2-ring-full-block24-fc1.json new file mode 100644 index 0000000..30e36db --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block24-fc1.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 73.7689437866211, + "projection_ring_total_ms": 43.649391174316406, + "results": [ + { + "name": "mlp_fc1", + "mnk": [ + 37810, + 28672, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": -678324672.0, + "candidate_checksum": -678324672.0 + }, + "timing": { + "warmup": 1, + "iterations": 2, + "ring_complete_ms": 43.649391174316406, + "reference_complete_ms": 73.7689437866211, + "ring_vs_reference": 0.5917041634833975, + "improvement_percent": 40.82958365166025 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 2231369728 + } + ], + "projection_total_improvement_percent": 40.82958365166025 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block24-qkv.json b/benchmarks/gb10-cute-p2-ring-full-block24-qkv.json new file mode 100644 index 0000000..97213c8 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block24-qkv.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 53.62971115112305, + "projection_ring_total_ms": 32.13934326171875, + "results": [ + { + "name": "attn_qkv_proj", + "mnk": [ + 37810, + 21504, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 44535508.0, + "candidate_checksum": 44535508.0 + }, + "timing": { + "warmup": 1, + "iterations": 2, + "ring_complete_ms": 32.13934326171875, + "reference_complete_ms": 53.62971115112305, + "ring_vs_reference": 0.5992824233409232, + "improvement_percent": 40.07175766590768 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 1673527296 + } + ], + "projection_total_improvement_percent": 40.07175766590768 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json b/benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json new file mode 100644 index 0000000..e1c0956 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 49, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 26.224416732788086, + "projection_ring_total_ms": 13.855199813842773, + "results": [ + { + "name": "attn_out_proj", + "mnk": [ + 37810, + 5376, + 7168 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 8257536, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 1557006080.0, + "candidate_checksum": 1557006080.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 13.855199813842773, + "reference_complete_ms": 26.224416732788086, + "ring_vs_reference": 0.5283320485263558, + "improvement_percent": 47.16679514736442 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 418381824 + } + ], + "projection_total_improvement_percent": 47.16679514736442 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block49-fc1.json b/benchmarks/gb10-cute-p2-ring-full-block49-fc1.json new file mode 100644 index 0000000..105740d --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block49-fc1.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 49, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 242.2157440185547, + "projection_ring_total_ms": 763.5274047851562, + "results": [ + { + "name": "mlp_fc1", + "mnk": [ + 37810, + 28672, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": 8475313152.0, + "candidate_checksum": 8475313152.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 763.5274047851562, + "reference_complete_ms": 242.2157440185547, + "ring_vs_reference": 3.1522616660569636, + "improvement_percent": -215.22616660569636 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 2231369728 + } + ], + "projection_total_improvement_percent": -215.22616660569636 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-block49-qkv.json b/benchmarks/gb10-cute-p2-ring-full-block49-qkv.json new file mode 100644 index 0000000..e503c30 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-block49-qkv.json @@ -0,0 +1,205 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 49, + "capacity": 2048, + "all_equal": true, + "projection_reference_total_ms": 84.6115493774414, + "projection_ring_total_ms": 31.96544075012207, + "results": [ + { + "name": "attn_qkv_proj", + "mnk": [ + 37810, + 21504, + 5376 + ], + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "ring_bytes": 6193152, + "parity": { + "all_chunks_equal": true, + "max_abs": 0.0, + "mean_abs_max": 0.0, + "reference_checksum": -40304712.0, + "candidate_checksum": -40304712.0 + }, + "timing": { + "warmup": 1, + "iterations": 1, + "ring_complete_ms": 31.96544075012207, + "reference_complete_ms": 84.6115493774414, + "ring_vs_reference": 0.37779051424207216, + "improvement_percent": 62.22094857579279 + }, + "chunks": [ + { + "start": 0, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 2048, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 4096, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 6144, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 8192, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 10240, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 12288, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 14336, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 16384, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 18432, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 20480, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 22528, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 24576, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 26624, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 28672, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 30720, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 32768, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 34816, + "rows": 2048, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + { + "start": 36864, + "rows": 946, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + } + ], + "output_bytes": 1673527296 + } + ], + "projection_total_improvement_percent": 62.22094857579279 +} diff --git a/benchmarks/gb10-cute-p2-ring-full-projection-summary.json b/benchmarks/gb10-cute-p2-ring-full-projection-summary.json new file mode 100644 index 0000000..daf10e6 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-full-projection-summary.json @@ -0,0 +1,46 @@ +{ + "date": "2026-08-23", + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "rows": 37810, + "capacity": 2048, + "chunk_count": 19, + "final_chunk_rows": 946, + "parity": { + "blocks": [0, 24, 49], + "roles": ["attn_qkv_proj", "attn_out_proj", "mlp_fc1"], + "all_chunks_equal": true, + "max_abs": 0.0, + "checksums_equal": true + }, + "timing_percent_improvement": { + "block_0": { + "attn_qkv_proj": 59.60311533167897, + "attn_out_proj": 42.63660269340447, + "mlp_fc1": 18.62094357187819 + }, + "block_24": { + "attn_qkv_proj": 40.07175766590768, + "attn_out_proj": -3.812088078043052, + "mlp_fc1": 40.82958365166025 + }, + "block_49": { + "attn_qkv_proj": 62.22094857579279, + "attn_out_proj": 47.16679514736442, + "mlp_fc1": -215.22616660569636 + } + }, + "timing_interpretation": { + "attn_qkv_proj": "Consistent 32 ms ring time and positive result across all blocks; advance to in-runtime validation.", + "attn_out_proj": "Exact but timing is mixed; retain as experimental until in-runtime measurement.", + "mlp_fc1": "Exact but duplicate-model unified-memory timing is unstable; no performance approval from this harness." + }, + "benchmark_caveat": "The resident service and validator hold separate model copies on unified memory. Multi-gigabyte FC1 outputs caused pressure and one combined validator run stopped the container. Timing must be repeated inside the single resident model before deployment.", + "policy": { + "attn_qkv_proj": "advance", + "attn_out_proj": "experimental", + "mlp_fc1": "parity-approved, performance-pending", + "mlp_fc2": "Comfy/cuBLAS fallback" + }, + "next_gate": "Integrate only QKV behind an opt-in runtime flag, benchmark blocks 0, 24, and 49 in the resident model, then decide whether attention output or FC1 should be added." +} diff --git a/benchmarks/gb10-cute-p2-ring-mlp-fc1.json b/benchmarks/gb10-cute-p2-ring-mlp-fc1.json new file mode 100644 index 0000000..5ffd6f4 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-mlp-fc1.json @@ -0,0 +1,67 @@ +{ + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "linear": "mlp_fc1", + "mnk": [ + 128, + 28672, + 5376 + ], + "ring": { + "row_capacity": 128, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": 344064, + "sfa_bytes": 43008, + "logical_bytes": 387072 + }, + "parity": { + "tensor_scale_equal": true, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "timing": { + "warmup": 10, + "iterations": 100, + "producer_ms": 0.015716160535812377, + "global_scale_ms": 0.014631999731063843, + "cute_consumer_ms": 0.6108745574951172, + "modeled_ring_ms": 0.6412227177619935, + "actual_into_ring_cute_gemm_ms": 0.6573414611816406, + "reference_vortex_scale_comfy_pack_gemm_ms": 0.5960015869140625, + "modeled_ring_vs_reference": 1.0758741785941779, + "actual_ring_vs_reference": 1.1029189780939672 + } +} diff --git a/benchmarks/gb10-cute-p2-ring-summary.json b/benchmarks/gb10-cute-p2-ring-summary.json new file mode 100644 index 0000000..105b3e0 --- /dev/null +++ b/benchmarks/gb10-cute-p2-ring-summary.json @@ -0,0 +1,49 @@ +{ + "date": "2026-08-23", + "device": "NVIDIA GB10", + "cutlass_dsl": "4.6.2", + "row_capacity": 128, + "warmup": 10, + "iterations": 100, + "producer": "vortex_native_quantize_nvfp4_into", + "consumer": "CuTe SM121 prepacked block-scaled GEMM", + "results": { + "attn_qkv_proj": { + "ring_bytes": 387072, + "reference_ms": 0.47588897705078126, + "ring_ms": 0.4637001419067383, + "improvement_percent": 2.56127704818477, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "decision": "experimental; require full-M chunk-size validation" + }, + "attn_out_proj": { + "ring_bytes": 516096, + "reference_ms": 0.10898783683776855, + "ring_ms": 0.09652607917785644, + "improvement_percent": 11.43408110618966, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "decision": "advance to chunk-size and real-block validation" + }, + "mlp_fc1": { + "ring_bytes": 387072, + "reference_ms": 0.5960015869140625, + "ring_ms": 0.6573414611816406, + "improvement_percent": -10.29189780939672, + "fp4_difference_count": 0, + "block_scale_difference_count": 0, + "output_equal": true, + "decision": "reject current ring consumer; retain reference path" + } + }, + "policy": { + "attn_out_proj": "continue", + "attn_qkv_proj": "experimental", + "mlp_fc1": "fallback", + "mlp_fc2": "fallback" + }, + "next_gate": "Sweep ring row capacity using one global activation scale, then validate the accepted attention-output path in real blocks 0, 24, and 49." +} diff --git a/benchmarks/gb10-cute-qkv-block0-alternating.json b/benchmarks/gb10-cute-qkv-block0-alternating.json new file mode 100644 index 0000000..b1ad92d --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block0-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 0, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": 2040895232.0, + "candidate_checksum": 2040895232.0, + "baseline": { + "mean_s": 0.46145327799604274, + "p50_s": 0.46154195550479926, + "min_s": 0.4599941649939865, + "max_s": 0.4628595529939048 + }, + "ring": { + "mean_s": 0.4653922098320133, + "p50_s": 0.46529679899686016, + "min_s": 0.46371654397808015, + "max_s": 0.4670645549776964 + }, + "p50_improvement_percent": -0.8135432645455021 +} diff --git a/benchmarks/gb10-cute-qkv-block0-baseline.json b/benchmarks/gb10-cute-qkv-block0-baseline.json new file mode 100644 index 0000000..2a7e1d2 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block0-baseline.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 0, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.0039003503237230084, + "p50_s": 0.00361842499114573, + "p90_s": 0.004298052983358502, + "p95_s": 0.004383006482385099, + "p99_s": 0.004450969281606376, + "min_s": 0.003614665998611599, + "max_s": 0.0044679599814116955 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.011204699655839553, + "p50_s": 0.010762252961285412, + "p90_s": 0.012003415380604565, + "p95_s": 0.01215856068301946, + "p99_s": 0.012282676924951375, + "min_s": 0.010538140020798892, + "max_s": 0.012313705985434353 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 6.149309532095988e-06, + "p50_s": 4.4959597289562225e-06, + "p90_s": 8.566351607441903e-06, + "p95_s": 9.075150592252612e-06, + "p99_s": 9.48218978010118e-06, + "min_s": 4.368019290268421e-06, + "max_s": 9.583949577063322e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.7290663284560043e-05, + "p50_s": 8.816015906631947e-06, + "p90_s": 2.942399587482214e-05, + "p95_s": 3.199999337084591e-05, + "p99_s": 3.406079136766494e-05, + "min_s": 8.4799830801785e-06, + "max_s": 3.457599086686969e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.695992442468802e-06, + "p50_s": 3.6479905247688293e-06, + "p90_s": 3.993557766079903e-06, + "p95_s": 4.036753671243787e-06, + "p99_s": 4.071310395374894e-06, + "min_s": 3.360037226229906e-06, + "max_s": 4.079949576407671e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.001928791015719374, + "p50_s": 0.0017829880234785378, + "p90_s": 0.002139096811879426, + "p95_s": 0.002183610410429537, + "p99_s": 0.0022192212892696264, + "min_s": 0.001775261014699936, + "max_s": 0.0022281240089796484 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 1.439966338997086e-05, + "p50_s": 1.3343989849090576e-05, + "p90_s": 1.871921122074127e-05, + "p95_s": 1.9391113892197608e-05, + "p99_s": 1.9928636029362678e-05, + "min_s": 9.791983757168055e-06, + "max_s": 2.0063016563653946e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.0023788169880087176, + "p50_s": 0.002249772020149976, + "p90_s": 0.0026388911646790802, + "p95_s": 0.0026875310577452183, + "p99_s": 0.002726442972198129, + "min_s": 0.0021505079930648208, + "max_s": 0.0027361709508113563 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 7.8986631706357e-06, + "p50_s": 6.432004738599062e-06, + "p90_s": 1.0156782809644936e-05, + "p95_s": 1.062238006852567e-05, + "p99_s": 1.0994857875630258e-05, + "min_s": 6.176007445901632e-06, + "max_s": 1.1087977327406406e-05 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.03701395931420848, + "p50_s": 0.025930000992957503, + "p90_s": 0.05311313216807321, + "p95_s": 0.05651102356496267, + "p99_s": 0.05922933668247424, + "min_s": 0.025202961987815797, + "max_s": 0.05990891496185213 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 2.0917330402880907e-05, + "p50_s": 5.44002978131175e-06, + "p90_s": 4.272641381248832e-05, + "p95_s": 4.738721181638539e-05, + "p99_s": 5.1115850219503045e-05, + "min_s": 5.2639516070485115e-06, + "max_s": 5.204800982028246e-05 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.041458233628266804, + "p50_s": 0.029947368951980025, + "p90_s": 0.05808691458078101, + "p95_s": 0.061604357784381135, + "p99_s": 0.06441831234726124, + "min_s": 0.02930553094483912, + "max_s": 0.06512180098798126 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 2.4512002710253e-05, + "p50_s": 1.7551996279507875e-05, + "p90_s": 3.592000575736165e-05, + "p95_s": 3.8216006942093366e-05, + "p99_s": 4.005280788987874e-05, + "min_s": 1.547200372442603e-05, + "max_s": 4.0512008126825094e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012107417666508505, + "p50_s": 0.012135433964431286, + "p90_s": 0.012213666830211878, + "p95_s": 0.012223445938434451, + "p99_s": 0.012231269225012512, + "min_s": 0.011953593988437206, + "max_s": 0.012233225046657026 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.00539845967432484, + "p50_s": 0.005506518005859107, + "p90_s": 0.005652962799649686, + "p95_s": 0.005671268398873508, + "p99_s": 0.005685912878252567, + "min_s": 0.004999287019018084, + "max_s": 0.00568957399809733 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.005182119008774559, + "p50_s": 0.005078374990262091, + "p90_s": 0.00533373502548784, + "p95_s": 0.005365655029891059, + "p99_s": 0.005391191033413634, + "min_s": 0.0050704070017673075, + "max_s": 0.005397575034294277 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.005088572021729003, + "p50_s": 0.004968855006154627, + "p90_s": 0.005264303821604699, + "p95_s": 0.005301234923535958, + "p99_s": 0.005330779805080965, + "min_s": 0.004958695033565164, + "max_s": 0.005338166025467217 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2596873336394007, + "p50_s": 0.255025869992096, + "p90_s": 0.26813469716580585, + "p95_s": 0.2697733005625196, + "p99_s": 0.27108418327989053, + "min_s": 0.2526242269668728, + "max_s": 0.2714119039592333 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.004714257995753239, + "p50_s": 0.004697238968219608, + "p90_s": 0.004743498226162046, + "p95_s": 0.004749280633404851, + "p99_s": 0.004753906559199095, + "min_s": 0.004690471978392452, + "max_s": 0.004755063040647656 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.0720021439095335e-06, + "p50_s": 4.799978341907263e-06, + "p90_s": 5.862431135028601e-06, + "p95_s": 5.995237734168768e-06, + "p99_s": 6.101483013480901e-06, + "min_s": 4.2879837565124035e-06, + "max_s": 6.128044333308935e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.8890675467749436e-05, + "p50_s": 1.5839992556720972e-05, + "p90_s": 2.770560095086694e-05, + "p95_s": 2.9188802000135183e-05, + "p99_s": 3.037536283954978e-05, + "min_s": 1.0160030797123909e-05, + "max_s": 3.067200304940343e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.722670953720808e-06, + "p50_s": 3.760040272027254e-06, + "p90_s": 3.7984107621014117e-06, + "p95_s": 3.8032070733606814e-06, + "p99_s": 3.8070441223680975e-06, + "min_s": 3.5999692045152187e-06, + "max_s": 3.8080033846199512e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.002283606658844898, + "p50_s": 0.0022722999565303326, + "p90_s": 0.002299871202558279, + "p95_s": 0.002303317608311772, + "p99_s": 0.002306074732914567, + "min_s": 0.002271756005939096, + "max_s": 0.0023067640140652657 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 1.3381339764843384e-05, + "p50_s": 1.3264012522995472e-05, + "p90_s": 1.6271998174488544e-05, + "p95_s": 1.664799638092518e-05, + "p99_s": 1.6948794946074483e-05, + "min_s": 9.856012184172869e-06, + "max_s": 1.7023994587361813e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.0028784163296222687, + "p50_s": 0.0028789869975298643, + "p90_s": 0.0029005677904933693, + "p95_s": 0.0029032653896138073, + "p99_s": 0.0029054234689101575, + "min_s": 0.0028502990026026964, + "max_s": 0.0029059629887342453 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 8.378662945081791e-06, + "p50_s": 7.951981388032436e-06, + "p90_s": 9.014387615025043e-06, + "p95_s": 9.14718839339912e-06, + "p99_s": 9.253429016098379e-06, + "min_s": 7.90401827543974e-06, + "max_s": 9.279989171773195e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.008133968998057147, + "p50_s": 0.007810881012119353, + "p90_s": 0.008587405015714466, + "p95_s": 0.008684470516163856, + "p99_s": 0.008762122916523367, + "min_s": 0.007809489965438843, + "max_s": 0.008781536016613245 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.957340666403373e-06, + "p50_s": 5.856039933860302e-06, + "p90_s": 6.444775499403477e-06, + "p95_s": 6.518367445096373e-06, + "p99_s": 6.577241001650691e-06, + "min_s": 5.424022674560547e-06, + "max_s": 6.59195939078927e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.013398274313658476, + "p50_s": 0.013059511955361813, + "p90_s": 0.013909942365717142, + "p95_s": 0.014016246167011559, + "p99_s": 0.014101289208047093, + "min_s": 0.013012761017307639, + "max_s": 0.014122549968305975 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.008729871983329454, + "p50_s": 0.008632095996290445, + "p90_s": 0.00904886396601796, + "p95_s": 0.0091009599622339, + "p99_s": 0.009142636759206652, + "min_s": 0.00840446399524808, + "max_s": 0.00915305595844984 + }, + "norm2": { + "count": 3, + "mean_s": 0.0036775400318826237, + "p50_s": 0.0036539300344884396, + "p90_s": 0.0037461916450411085, + "p95_s": 0.003757724346360192, + "p99_s": 0.003766950507415459, + "min_s": 0.0036094330134801567, + "max_s": 0.0037692570476792753 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010796177666634321, + "p50_s": 0.010941468004602939, + "p90_s": 0.010954472806770353, + "p95_s": 0.01095609840704128, + "p99_s": 0.010957398887258022, + "min_s": 0.010489340987987816, + "max_s": 0.010957724007312208 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.4533129160602885e-06, + "p50_s": 4.399975296109915e-06, + "p90_s": 4.7199777327477935e-06, + "p95_s": 4.759978037327528e-06, + "p99_s": 4.7919782809913166e-06, + "min_s": 4.159985110163689e-06, + "max_s": 4.799978341907263e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 9.487984546770653e-06, + "p50_s": 9.295996278524399e-06, + "p90_s": 1.0576006025075912e-05, + "p95_s": 1.0736007243394852e-05, + "p99_s": 1.0864008218050004e-05, + "min_s": 8.271948900073767e-06, + "max_s": 1.0896008461713791e-05 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.477325662970543e-06, + "p50_s": 3.471970558166504e-06, + "p90_s": 3.548804670572281e-06, + "p95_s": 3.558408934623003e-06, + "p99_s": 3.5660923458635807e-06, + "min_s": 3.3919932320713997e-06, + "max_s": 3.568013198673725e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017434633142935734, + "p50_s": 0.0017429409781470895, + "p90_s": 0.0017490209778770803, + "p95_s": 0.0017497809778433291, + "p99_s": 0.0017503889778163284, + "min_s": 0.0017369079869240522, + "max_s": 0.0017505409778095782 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 1.0010650536666313e-05, + "p50_s": 9.999959729611874e-06, + "p90_s": 1.058878842741251e-05, + "p95_s": 1.066239201463759e-05, + "p99_s": 1.0721274884417652e-05, + "min_s": 9.295996278524399e-06, + "max_s": 1.0735995601862669e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.0022977933404035866, + "p50_s": 0.002274715981911868, + "p90_s": 0.0024394648033194246, + "p95_s": 0.002460058405995369, + "p99_s": 0.0024765332881361247, + "min_s": 0.0021380120306275785, + "max_s": 0.0024806520086713135 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.543996278196573e-06, + "p50_s": 6.464018952101469e-06, + "p90_s": 6.7327986471354965e-06, + "p95_s": 6.766396109014749e-06, + "p99_s": 6.793274078518152e-06, + "min_s": 6.367976311594248e-06, + "max_s": 6.799993570894003e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.03486676298780367, + "p50_s": 0.03488327999366447, + "p90_s": 0.0348947360063903, + "p95_s": 0.03489616800798103, + "p99_s": 0.03489731360925361, + "min_s": 0.03481940896017477, + "max_s": 0.03489760000957176 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.68533626695474e-06, + "p50_s": 5.6480057537555695e-06, + "p90_s": 5.852803587913513e-06, + "p95_s": 5.878403317183256e-06, + "p99_s": 5.89888310059905e-06, + "min_s": 5.504000000655651e-06, + "max_s": 5.904003046452999e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.038996499322820455, + "p50_s": 0.03899847296997905, + "p90_s": 0.03917223219759762, + "p95_s": 0.03919395210104994, + "p99_s": 0.039211328023811796, + "min_s": 0.03877535299398005, + "max_s": 0.03921567200450227 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.02435958234127611, + "p50_s": 0.02435267501277849, + "p90_s": 0.024421091785188764, + "p95_s": 0.024429643881740046, + "p99_s": 0.024436485558981078, + "min_s": 0.024287876032758504, + "max_s": 0.024438195978291333 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.271666208902995e-06, + "p50_s": 4.208006430417299e-06, + "p90_s": 4.579185042530298e-06, + "p95_s": 4.625582369044423e-06, + "p99_s": 4.662700230255723e-06, + "min_s": 3.935012500733137e-06, + "max_s": 4.671979695558548e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.533359505236149e-06, + "p50_s": 8.704024367034435e-06, + "p90_s": 9.216018952429295e-06, + "p95_s": 9.280018275603652e-06, + "p99_s": 9.331217734143137e-06, + "min_s": 7.552036549896002e-06, + "max_s": 9.34401759877801e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.471989960720142e-06, + "p50_s": 3.5199918784201145e-06, + "p90_s": 3.5199918784201145e-06, + "p95_s": 3.5199918784201145e-06, + "p99_s": 3.5199918784201145e-06, + "min_s": 3.375986125320196e-06, + "max_s": 3.5199918784201145e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.004591644974425435, + "p50_s": 0.004597686987835914, + "p90_s": 0.00461162697756663, + "p95_s": 0.00461336947628297, + "p99_s": 0.00461476347525604, + "min_s": 0.004562135960441083, + "max_s": 0.004615111974999309 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 9.818662268420061e-06, + "p50_s": 9.311945177614689e-06, + "p90_s": 1.0924809612333774e-05, + "p95_s": 1.112641766667366e-05, + "p99_s": 1.1287704110145569e-05, + "min_s": 8.816015906631947e-06, + "max_s": 1.1328025721013546e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.006046714649225275, + "p50_s": 0.006064180983230472, + "p90_s": 0.006119732977822423, + "p95_s": 0.006126676977146417, + "p99_s": 0.006132232176605612, + "min_s": 0.005942341987974942, + "max_s": 0.006133620976470411 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.224009363601605e-06, + "p50_s": 6.208021659404039e-06, + "p90_s": 6.40001380816102e-06, + "p95_s": 6.4240128267556436e-06, + "p99_s": 6.443212041631341e-06, + "min_s": 6.01599458605051e-06, + "max_s": 6.4480118453502655e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.05645514999438698, + "p50_s": 0.05642460100352764, + "p90_s": 0.05665761139243841, + "p95_s": 0.05668673769105226, + "p99_s": 0.05671003872994333, + "min_s": 0.0562249849899672, + "max_s": 0.056715863989666104 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.6533220534523325e-06, + "p50_s": 5.71197597309947e-06, + "p90_s": 5.724781658500433e-06, + "p95_s": 5.726382369175553e-06, + "p99_s": 5.72766293771565e-06, + "min_s": 5.520007107406855e-06, + "max_s": 5.727983079850674e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.06717234667545806, + "p50_s": 0.0670450460165739, + "p90_s": 0.06738878921605647, + "p95_s": 0.06743175711599178, + "p99_s": 0.06746613143594005, + "min_s": 0.06699726899387315, + "max_s": 0.0674747250159271 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.018475912996412564, + "p50_s": 0.008704703999683261, + "p90_s": 0.032231163978576664, + "p95_s": 0.035171971475938334, + "p99_s": 0.037524617473827675, + "min_s": 0.008610256016254425, + "max_s": 0.03811277897330001 + }, + "block_total": { + "count": 3, + "mean_s": 0.5345096856666108, + "p50_s": 0.535079502966255, + "p90_s": 0.5420267941895872, + "p95_s": 0.5428952055925038, + "p99_s": 0.5435899347148371, + "min_s": 0.5246859370381571, + "max_s": 0.5437636169954203 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.46528928333039704, + "p50_s": 0.4646527419681661, + "p90_s": 0.46618508918909357, + "p95_s": 0.4663766325917095, + "p99_s": 0.46652986731380225, + "min_s": 0.46464693202869967, + "max_s": 0.46656817599432543 + }, + "module_forward_checksum": 606380672.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 9093546496 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1360.0130000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1339.1809999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 603.1990000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 527.5030000000006, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 445.02400000000034, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 265.37599999999975, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 222.23999999999978, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 161.48800000000028, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "aten::to", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 160.35200000000077, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 154.1279999999988, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 148.03099999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::empty", + "count": 34, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 141.58399999999915, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009298432, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 138.3199999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 132.09499999999935, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 122.49499999999898, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 112.32000000000062, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 99.2470000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 93.92000000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 81.16800000000057, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 79.10399999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-block0-ring.json b/benchmarks/gb10-cute-qkv-block0-ring.json new file mode 100644 index 0000000..e95dd9c --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block0-ring.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 0, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.0038844093408746025, + "p50_s": 0.003442010027356446, + "p90_s": 0.004506341204978525, + "p95_s": 0.004639382602181286, + "p99_s": 0.004745815719943493, + "min_s": 0.003438793995883316, + "max_s": 0.004772423999384046 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.011301896010991186, + "p50_s": 0.010341919027268887, + "p90_s": 0.012674343818798662, + "p95_s": 0.012965896917739883, + "p99_s": 0.01319913939689286, + "min_s": 0.010306318989023566, + "max_s": 0.013257450016681105 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.6800199672579765e-06, + "p50_s": 4.656030796468258e-06, + "p90_s": 7.612793706357479e-06, + "p95_s": 7.982389070093631e-06, + "p99_s": 8.278065361082553e-06, + "min_s": 4.032044671475887e-06, + "max_s": 8.351984433829784e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.581335285057624e-05, + "p50_s": 8.608039934188128e-06, + "p90_s": 2.631038660183549e-05, + "p95_s": 2.8523179935291406e-05, + "p99_s": 3.0293414602056147e-05, + "min_s": 8.096045348793268e-06, + "max_s": 3.073597326874733e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.535998985171318e-06, + "p50_s": 3.535998985171318e-06, + "p90_s": 3.6255922168493273e-06, + "p95_s": 3.636791370809078e-06, + "p99_s": 3.645750693976879e-06, + "min_s": 3.4240074455738068e-06, + "max_s": 3.6479905247688293e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.002059137332253158, + "p50_s": 0.0018270040163770318, + "p90_s": 0.0024354639695957302, + "p95_s": 0.0025115214637480674, + "p99_s": 0.0025723674590699373, + "min_s": 0.0017628290224820375, + "max_s": 0.002587578957900405 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 2.2490644672264654e-05, + "p50_s": 2.297596074640751e-05, + "p90_s": 2.3987190797924995e-05, + "p95_s": 2.411359455436468e-05, + "p99_s": 2.421471755951643e-05, + "min_s": 2.025597495958209e-05, + "max_s": 2.4239998310804367e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002397505333647132, + "p50_s": 0.0022308279876597226, + "p90_s": 0.0026664879755117, + "p95_s": 0.002720945473993197, + "p99_s": 0.002764511472778395, + "min_s": 0.002186285040806979, + "max_s": 0.0027754029724746943 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 7.466665313889583e-06, + "p50_s": 7.599999662488699e-06, + "p90_s": 7.996789645403623e-06, + "p95_s": 8.046388393267989e-06, + "p99_s": 8.086067391559481e-06, + "min_s": 6.704009138047695e-06, + "max_s": 8.095987141132355e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.039801057001265384, + "p50_s": 0.025940386985894293, + "p90_s": 0.059222071792464706, + "p95_s": 0.063382282393286, + "p99_s": 0.06671045087394303, + "min_s": 0.02592029102379456, + "max_s": 0.0675424929941073 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 6.346691710253556e-06, + "p50_s": 5.4720439948141575e-06, + "p90_s": 7.63522693887353e-06, + "p95_s": 7.90562480688095e-06, + "p99_s": 8.121943101286887e-06, + "min_s": 5.39200846105814e-06, + "max_s": 8.176022674888372e-06 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.04436540632741526, + "p50_s": 0.03002823598217219, + "p90_s": 0.06444033759180456, + "p95_s": 0.06874185029300861, + "p99_s": 0.07218306045397184, + "min_s": 0.030024620005860925, + "max_s": 0.07304336299421266 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 1.6122687763224047e-05, + "p50_s": 1.315202098339796e-05, + "p90_s": 2.1305610425770285e-05, + "p95_s": 2.2324809106066822e-05, + "p99_s": 2.3140168050304055e-05, + "min_s": 1.1872034519910812e-05, + "max_s": 2.3344007786363363e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012314411326466749, + "p50_s": 0.012499674980062991, + "p90_s": 0.012545524595770985, + "p95_s": 0.012551255797734485, + "p99_s": 0.012555840759305284, + "min_s": 0.011886571999639273, + "max_s": 0.012556986999697983 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.005020225983268271, + "p50_s": 0.005004551960155368, + "p90_s": 0.005060359183698892, + "p95_s": 0.0050673350866418335, + "p99_s": 0.005072915808996186, + "min_s": 0.004981815000064671, + "max_s": 0.005074310989584774 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.005056855365789185, + "p50_s": 0.005042263015639037, + "p90_s": 0.0050791406421922146, + "p95_s": 0.0050837503455113614, + "p99_s": 0.005087438108166679, + "min_s": 0.005039943032898009, + "max_s": 0.005088360048830509 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.005133500671945512, + "p50_s": 0.005174166988581419, + "p90_s": 0.0052229606080800295, + "p95_s": 0.005229059810517356, + "p99_s": 0.005233939172467217, + "min_s": 0.004991176014300436, + "max_s": 0.005235159012954682 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2589895313140005, + "p50_s": 0.25650634698104113, + "p90_s": 0.26326487499754875, + "p95_s": 0.26410969099961223, + "p99_s": 0.264785543801263, + "min_s": 0.25550773995928466, + "max_s": 0.26495450700167567 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.0046977306677338975, + "p50_s": 0.004640695988200605, + "p90_s": 0.004778039990924299, + "p95_s": 0.0047952079912647605, + "p99_s": 0.00480894239153713, + "min_s": 0.004640120023395866, + "max_s": 0.004812375991605222 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 4.453351721167564e-06, + "p50_s": 4.191999323666096e-06, + "p90_s": 4.870421253144741e-06, + "p95_s": 4.955223994329571e-06, + "p99_s": 5.023066187277436e-06, + "min_s": 4.128029104322195e-06, + "max_s": 5.040026735514402e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.0901324761410555e-05, + "p50_s": 9.343959391117096e-06, + "p90_s": 1.3158377259969712e-05, + "p95_s": 1.3635179493576289e-05, + "p99_s": 1.401662128046155e-05, + "min_s": 9.248033165931702e-06, + "max_s": 1.4111981727182865e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.471989960720142e-06, + "p50_s": 3.3919932320713997e-06, + "p90_s": 3.6095967516303063e-06, + "p95_s": 3.6367971915751694e-06, + "p99_s": 3.6585575435310603e-06, + "min_s": 3.3599790185689926e-06, + "max_s": 3.663997631520033e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.002189169346820563, + "p50_s": 0.0021924120374023914, + "p90_s": 0.002197237592190504, + "p95_s": 0.0021978407865390183, + "p99_s": 0.0021983233420178293, + "min_s": 0.0021766520221717656, + "max_s": 0.0021984439808875322 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 1.1679988044003645e-05, + "p50_s": 1.0255957022309303e-05, + "p90_s": 1.3852771371603011e-05, + "p95_s": 1.4302373165264725e-05, + "p99_s": 1.4662054600194098e-05, + "min_s": 1.0032032150775194e-05, + "max_s": 1.475197495892644e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002833104692399502, + "p50_s": 0.0028447150252759457, + "p90_s": 0.0028486958239227534, + "p95_s": 0.002849193423753604, + "p99_s": 0.002849591503618285, + "min_s": 0.0028049080283381045, + "max_s": 0.0028496910235844553 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 7.119980485488971e-06, + "p50_s": 6.671994924545288e-06, + "p90_s": 7.926393300294876e-06, + "p95_s": 8.083193097263575e-06, + "p99_s": 8.208632934838535e-06, + "min_s": 6.447953637689352e-06, + "max_s": 8.239992894232273e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.008127884687079737, + "p50_s": 0.007850803027395159, + "p90_s": 0.008538391825277358, + "p95_s": 0.008624340425012632, + "p99_s": 0.008693099304800852, + "min_s": 0.007822562009096146, + "max_s": 0.008710289024747908 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.770668697853883e-06, + "p50_s": 5.69602707400918e-06, + "p90_s": 6.054400000721216e-06, + "p95_s": 6.099196616560221e-06, + "p99_s": 6.1350339092314244e-06, + "min_s": 5.471985787153244e-06, + "max_s": 6.143993232399225e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.013236238659980396, + "p50_s": 0.01298735401360318, + "p90_s": 0.01366794117493555, + "p95_s": 0.013753014570102095, + "p99_s": 0.013821073286235334, + "min_s": 0.012883274001069367, + "max_s": 0.013838087965268642 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.00891906803008169, + "p50_s": 0.008913073048461229, + "p90_s": 0.009082570637110621, + "p95_s": 0.009103757835691794, + "p99_s": 0.009120707594556734, + "min_s": 0.00871918600751087, + "max_s": 0.009124945034272969 + }, + "norm2": { + "count": 3, + "mean_s": 0.0035535029989356795, + "p50_s": 0.0035576900118030608, + "p90_s": 0.0036055227858014403, + "p95_s": 0.003611501882551238, + "p99_s": 0.003616285159951076, + "min_s": 0.0034853380057029426, + "max_s": 0.0036174809793010354 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010866791980030635, + "p50_s": 0.010817836970090866, + "p90_s": 0.010982598597183823, + "p95_s": 0.011003193800570443, + "p99_s": 0.011019669963279738, + "min_s": 0.010758749966043979, + "max_s": 0.011023789003957063 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.4586680208643275e-06, + "p50_s": 4.352012183517218e-06, + "p90_s": 4.8128305934369566e-06, + "p95_s": 4.870432894676924e-06, + "p99_s": 4.916514735668898e-06, + "min_s": 4.0959566831588745e-06, + "max_s": 4.928035195916891e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 9.039998985826969e-06, + "p50_s": 8.944014552980661e-06, + "p90_s": 9.775965008884668e-06, + "p95_s": 9.879958815872669e-06, + "p99_s": 9.963153861463071e-06, + "min_s": 8.192029781639576e-06, + "max_s": 9.98395262286067e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.536018387724956e-06, + "p50_s": 3.648048732429743e-06, + "p90_s": 3.660807851701975e-06, + "p95_s": 3.662402741611004e-06, + "p99_s": 3.663678653538227e-06, + "min_s": 3.296008799225092e-06, + "max_s": 3.663997631520033e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017847650063534577, + "p50_s": 0.0017553410143591464, + "p90_s": 0.0018440961954183876, + "p95_s": 0.0018551905930507928, + "p99_s": 0.001864066111156717, + "min_s": 0.001732669014018029, + "max_s": 0.001866284990683198 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 1.0303997745116552e-05, + "p50_s": 1.047999830916524e-05, + "p90_s": 1.0748824570327997e-05, + "p95_s": 1.0782427852973343e-05, + "p99_s": 1.0809310479089617e-05, + "min_s": 9.615963790565729e-06, + "max_s": 1.0816031135618687e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.0021633559954352677, + "p50_s": 0.0021477399859577417, + "p90_s": 0.002185807190835476, + "p95_s": 0.0021905655914451925, + "p99_s": 0.0021943723119329664, + "min_s": 0.002147004008293152, + "max_s": 0.0021953239920549095 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.517337169498205e-06, + "p50_s": 6.352027412503958e-06, + "p90_s": 6.799993570894003e-06, + "p95_s": 6.8559893406927586e-06, + "p99_s": 6.900785956531763e-06, + "min_s": 6.287998985499144e-06, + "max_s": 6.911985110491514e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.03497030300786719, + "p50_s": 0.035002851975150406, + "p90_s": 0.035098020010627805, + "p95_s": 0.035109916015062484, + "p99_s": 0.03511943281861022, + "min_s": 0.034786245028954, + "max_s": 0.035121812019497156 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.631998647004366e-06, + "p50_s": 5.536014214158058e-06, + "p90_s": 5.855970084667206e-06, + "p95_s": 5.895964568480849e-06, + "p99_s": 5.927960155531764e-06, + "min_s": 5.424022674560547e-06, + "max_s": 5.935959052294493e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.03900541601857791, + "p50_s": 0.038999997021164745, + "p90_s": 0.03917357780737803, + "p95_s": 0.039195275405654685, + "p99_s": 0.03921263348427601, + "min_s": 0.03879927803063765, + "max_s": 0.039216973003931344 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.0237522956643564, + "p50_s": 0.023783078999258578, + "p90_s": 0.023799194977618756, + "p95_s": 0.023801209474913775, + "p99_s": 0.023802821072749794, + "min_s": 0.023670584021601826, + "max_s": 0.023803223972208798 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.831973152856032e-06, + "p50_s": 5.007954314351082e-06, + "p90_s": 5.251169204711914e-06, + "p95_s": 5.281571066007018e-06, + "p99_s": 5.305892555043101e-06, + "min_s": 4.175992216914892e-06, + "max_s": 5.311972927302122e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.869334124028683e-06, + "p50_s": 8.800008799880743e-06, + "p90_s": 9.478384163230659e-06, + "p95_s": 9.563181083649398e-06, + "p99_s": 9.63101861998439e-06, + "min_s": 8.160015568137169e-06, + "max_s": 9.647978004068136e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.3919932320713997e-06, + "p50_s": 3.3760443329811096e-06, + "p90_s": 3.452785313129425e-06, + "p95_s": 3.4623779356479647e-06, + "p99_s": 3.470052033662796e-06, + "min_s": 3.3279648050665855e-06, + "max_s": 3.471970558166504e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.004335538988622527, + "p50_s": 0.004312007979024202, + "p90_s": 0.004394120781216771, + "p95_s": 0.004404384881490842, + "p99_s": 0.004412596161710098, + "min_s": 0.004279960005078465, + "max_s": 0.004414648981764913 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 1.5887975071867306e-05, + "p50_s": 1.0015966836363077e-05, + "p90_s": 2.4607975501567125e-05, + "p95_s": 2.643197658471763e-05, + "p99_s": 2.7891177451238038e-05, + "min_s": 9.391980711370707e-06, + "max_s": 2.8255977667868137e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.005852811639973273, + "p50_s": 0.005791350966319442, + "p90_s": 0.005944655789062381, + "p95_s": 0.005963818891905248, + "p99_s": 0.005979149374179542, + "min_s": 0.005784101958852261, + "max_s": 0.0059829819947481155 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.6400001135965186e-06, + "p50_s": 6.6400389187037945e-06, + "p90_s": 6.742391269654036e-06, + "p95_s": 6.7551853135228155e-06, + "p99_s": 6.7654205486178394e-06, + "min_s": 6.511982064694166e-06, + "max_s": 6.767979357391596e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.056130341331784926, + "p50_s": 0.0560961120063439, + "p90_s": 0.0565045592142269, + "p95_s": 0.056555615115212274, + "p99_s": 0.056596459836000576, + "min_s": 0.05568824097281322, + "max_s": 0.05660667101619765 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.583996729304393e-06, + "p50_s": 5.599984433501959e-06, + "p90_s": 5.676771979779005e-06, + "p95_s": 5.686370423063636e-06, + "p99_s": 5.694049177691341e-06, + "min_s": 5.456036888062954e-06, + "max_s": 5.695968866348267e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.06640448931527014, + "p50_s": 0.06639227096457034, + "p90_s": 0.06670863500330597, + "p95_s": 0.06674818050814792, + "p99_s": 0.06677981691202148, + "min_s": 0.06603347096825019, + "max_s": 0.06678772601298988 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.01474788433794553, + "p50_s": 0.008609394019003958, + "p90_s": 0.023345611605327575, + "p95_s": 0.025187638803618028, + "p99_s": 0.02666126056225039, + "min_s": 0.008604592992924154, + "max_s": 0.02702966600190848 + }, + "block_total": { + "count": 3, + "mean_s": 0.5313544006785378, + "p50_s": 0.52417680202052, + "p90_s": 0.5474372819997371, + "p95_s": 0.5503448419971392, + "p99_s": 0.5526708899950609, + "min_s": 0.5166339980205521, + "max_s": 0.5532524019945413 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.46708817364803207, + "p50_s": 0.4669776560040191, + "p90_s": 0.467878839164041, + "p95_s": 0.4679914870590437, + "p99_s": 0.4680816053750459, + "min_s": 0.4661827299860306, + "max_s": 0.46810413495404646 + }, + "module_forward_checksum": 606380672.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 7351755264 + }, + "profiler_top_events": [ + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 512.799, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 465.9349999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 369.58299999999963, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 335.5350000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 325.1509999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 236.92700000000127, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 13, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 206.33499999999913, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 194.91200000000026, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 159.58399999999983, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 136.27199999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::empty", + "count": 29, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 133.95199999999943, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 6267507200, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 133.83899999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 125.10400000000027, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 112.76800000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 111.69599999999991, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 102.35199999999986, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 87.96799999999985, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 80.92799999999943, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "aten::slice", + "count": 19, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 80.81600000000071, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 65.34299999999985, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-block24-alternating.json b/benchmarks/gb10-cute-qkv-block24-alternating.json new file mode 100644 index 0000000..5232e0b --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 10, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -1751403776.0, + "candidate_checksum": -1751403776.0, + "baseline": { + "mean_s": 0.46800651020603257, + "p50_s": 0.4678640030033421, + "min_s": 0.4662869300227612, + "max_s": 0.46959813201101497 + }, + "ring": { + "mean_s": 0.4705851764010731, + "p50_s": 0.4703063364722766, + "min_s": 0.46895833703456447, + "max_s": 0.472499904979486 + }, + "p50_improvement_percent": -0.5220178199768499 +} diff --git a/benchmarks/gb10-cute-qkv-block24-baseline.json b/benchmarks/gb10-cute-qkv-block24-baseline.json new file mode 100644 index 0000000..20e977b --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-baseline.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.0036721593351103365, + "p50_s": 0.0036359680234454572, + "p90_s": 0.00376709041884169, + "p95_s": 0.003783480718266219, + "p99_s": 0.003796592957805842, + "min_s": 0.0035806389641948044, + "max_s": 0.003799871017690748 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.010726557637099177, + "p50_s": 0.010602172988001257, + "p90_s": 0.010938596178311855, + "p95_s": 0.010980649077100679, + "p99_s": 0.01101429139613174, + "min_s": 0.010554797947406769, + "max_s": 0.011022701975889504 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.6746648624539375e-06, + "p50_s": 4.847999662160873e-06, + "p90_s": 6.84480182826519e-06, + "p95_s": 7.094402099028229e-06, + "p99_s": 7.294082315638661e-06, + "min_s": 4.83199255540967e-06, + "max_s": 7.344002369791269e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.602132882302006e-05, + "p50_s": 1.003197394311428e-05, + "p90_s": 2.5776028633117675e-05, + "p95_s": 2.7744035469368098e-05, + "p99_s": 2.9318440938368442e-05, + "min_s": 8.319970220327377e-06, + "max_s": 2.9712042305618525e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.562658093869686e-06, + "p50_s": 3.5199918784201145e-06, + "p90_s": 3.68637265637517e-06, + "p95_s": 3.7071702536195515e-06, + "p99_s": 3.7238083314150572e-06, + "min_s": 3.4400145523250103e-06, + "max_s": 3.7279678508639336e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.001872149354312569, + "p50_s": 0.0017504799761809409, + "p90_s": 0.002067510422784835, + "p95_s": 0.0021071392286103217, + "p99_s": 0.0021388422732707114, + "min_s": 0.0017192000523209572, + "max_s": 0.0021467680344358087 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 1.5653010147313278e-05, + "p50_s": 1.8463993910700083e-05, + "p90_s": 1.87960104085505e-05, + "p95_s": 1.8837512470781803e-05, + "p99_s": 1.8870714120566843e-05, + "min_s": 9.616021998226643e-06, + "max_s": 1.8879014533013105e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002274436662749698, + "p50_s": 0.0022630870225839317, + "p90_s": 0.002373551798518747, + "p95_s": 0.002387359895510599, + "p99_s": 0.0023984063731040807, + "min_s": 0.0021590549731627107, + "max_s": 0.002401167992502451 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.943999323993921e-06, + "p50_s": 6.480026058852673e-06, + "p90_s": 7.785600610077381e-06, + "p95_s": 7.948797428980469e-06, + "p99_s": 8.07935488410294e-06, + "min_s": 6.239977665245533e-06, + "max_s": 8.111994247883558e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.02569908833053584, + "p50_s": 0.025752091023605317, + "p90_s": 0.02613150858087465, + "p95_s": 0.02617893577553332, + "p99_s": 0.026216877531260254, + "min_s": 0.025118810997810215, + "max_s": 0.026226362970191985 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 6.245332770049572e-06, + "p50_s": 5.568028427660465e-06, + "p90_s": 7.244781590998172e-06, + "p95_s": 7.454375736415386e-06, + "p99_s": 7.622051052749157e-06, + "min_s": 5.504000000655651e-06, + "max_s": 7.6639698818326e-06 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.029945935336096834, + "p50_s": 0.02986503398278728, + "p90_s": 0.030461424414534123, + "p95_s": 0.030535973218502478, + "p99_s": 0.03059561226167716, + "min_s": 0.029362250003032386, + "max_s": 0.030610522022470832 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 1.9002686409900587e-05, + "p50_s": 2.0368024706840515e-05, + "p90_s": 2.22496222704649e-05, + "p95_s": 2.2484821965917948e-05, + "p99_s": 2.2672981722280386e-05, + "min_s": 1.392001286149025e-05, + "max_s": 2.2720021661370993e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012115544329086939, + "p50_s": 0.012150077964179218, + "p90_s": 0.012263204413466157, + "p95_s": 0.012277345219627023, + "p99_s": 0.012288657864555717, + "min_s": 0.01190506899729371, + "max_s": 0.01229148602578789 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.005235358665231615, + "p50_s": 0.005258975026663393, + "p90_s": 0.005414558190386742, + "p95_s": 0.005434006085852161, + "p99_s": 0.005449564402224496, + "min_s": 0.004993646987713873, + "max_s": 0.00545345398131758 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.005438665684778243, + "p50_s": 0.00541571102803573, + "p90_s": 0.005487903009634465, + "p95_s": 0.005496927007334306, + "p99_s": 0.00550414620549418, + "min_s": 0.005394335021264851, + "max_s": 0.005505951005034149 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.0051273749947237475, + "p50_s": 0.005076878995168954, + "p90_s": 0.0052603541989810765, + "p95_s": 0.005283288599457592, + "p99_s": 0.005301636119838804, + "min_s": 0.00499902298906818, + "max_s": 0.005306222999934107 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2554921513268103, + "p50_s": 0.255616124952212, + "p90_s": 0.25568432337604463, + "p95_s": 0.2556928481790237, + "p99_s": 0.255699668021407, + "min_s": 0.2551589560462162, + "max_s": 0.2557013729820028 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.004698617655473451, + "p50_s": 0.0046985429944470525, + "p90_s": 0.004699272592552007, + "p95_s": 0.004699363792315126, + "p99_s": 0.004699436752125621, + "min_s": 0.004697854979895055, + "max_s": 0.004699454992078245 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 4.762647828708093e-06, + "p50_s": 4.624016582965851e-06, + "p90_s": 5.2511692047119146e-06, + "p95_s": 5.329563282430173e-06, + "p99_s": 5.392278544604778e-06, + "min_s": 4.255969543009996e-06, + "max_s": 5.40795736014843e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.0826683137565851e-05, + "p50_s": 1.0176037903875113e-05, + "p90_s": 1.1814420577138663e-05, + "p95_s": 1.2019218411296606e-05, + "p99_s": 1.218305667862296e-05, + "min_s": 1.0079995263367891e-05, + "max_s": 1.222401624545455e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.573348900924126e-06, + "p50_s": 3.5840203054249287e-06, + "p90_s": 3.6223907954990866e-06, + "p95_s": 3.627187106758356e-06, + "p99_s": 3.6310241557657717e-06, + "min_s": 3.5040429793298244e-06, + "max_s": 3.631983418017626e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.002271252994736036, + "p50_s": 0.002267103991471231, + "p90_s": 0.002277253591455519, + "p95_s": 0.002278522291453555, + "p99_s": 0.002279537251451984, + "min_s": 0.0022668640012852848, + "max_s": 0.0022797909914515913 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 1.067200597996513e-05, + "p50_s": 1.0624004062265158e-05, + "p90_s": 1.105921110138297e-05, + "p95_s": 1.1113611981272697e-05, + "p99_s": 1.1157132685184478e-05, + "min_s": 1.022400101646781e-05, + "max_s": 1.1168012861162424e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.0028463726630434394, + "p50_s": 0.0028423359617590904, + "p90_s": 0.00285377842374146, + "p95_s": 0.002855208731489256, + "p99_s": 0.002856352977687493, + "min_s": 0.0028401429881341755, + "max_s": 0.002856639039237052 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.75199165319403e-06, + "p50_s": 6.671994924545288e-06, + "p90_s": 7.1072019636631005e-06, + "p95_s": 7.1616028435528275e-06, + "p99_s": 7.205123547464608e-06, + "min_s": 6.367976311594248e-06, + "max_s": 7.2160037234425545e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.00781480900089567, + "p50_s": 0.007808863010723144, + "p90_s": 0.007827895798254759, + "p95_s": 0.00783027489669621, + "p99_s": 0.00783217817544937, + "min_s": 0.007802909996826202, + "max_s": 0.007832653995137662 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.680000564704339e-06, + "p50_s": 5.664012860506773e-06, + "p90_s": 5.791976582258939e-06, + "p95_s": 5.80797204747796e-06, + "p99_s": 5.820768419653177e-06, + "min_s": 5.552021320909262e-06, + "max_s": 5.8239675126969814e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.013016370648983866, + "p50_s": 0.013008972979150712, + "p90_s": 0.013036953774280846, + "p95_s": 0.013040451373672112, + "p99_s": 0.013043249453185126, + "min_s": 0.012996189994737506, + "max_s": 0.01304394897306338 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.008527667979554584, + "p50_s": 0.008532046980690211, + "p90_s": 0.00853673176607117, + "p95_s": 0.00853731736424379, + "p99_s": 0.008537785842781887, + "min_s": 0.00851305399555713, + "max_s": 0.00853790296241641 + }, + "norm2": { + "count": 3, + "mean_s": 0.0035164790072788796, + "p50_s": 0.0035144950379617512, + "p90_s": 0.003523019806016237, + "p95_s": 0.0035240854020230474, + "p99_s": 0.0035249378788284958, + "min_s": 0.0035097909858450294, + "max_s": 0.003525150998029858 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010573251328120628, + "p50_s": 0.010518733994103968, + "p90_s": 0.010858023609034716, + "p95_s": 0.01090043481090106, + "p99_s": 0.010934363772394136, + "min_s": 0.010258173977490515, + "max_s": 0.010942846012767404 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.634668584913015e-06, + "p50_s": 4.608009476214647e-06, + "p90_s": 5.004799459129572e-06, + "p95_s": 5.054398206993937e-06, + "p99_s": 5.09407720528543e-06, + "min_s": 4.191999323666096e-06, + "max_s": 5.103996954858303e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.896032037834326e-06, + "p50_s": 8.848030120134354e-06, + "p90_s": 9.232014417648316e-06, + "p95_s": 9.28001245483756e-06, + "p99_s": 9.318410884588956e-06, + "min_s": 8.51205550134182e-06, + "max_s": 9.328010492026806e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.4399951497713723e-06, + "p50_s": 3.439956344664097e-06, + "p90_s": 3.516790457069874e-06, + "p95_s": 3.526394721120596e-06, + "p99_s": 3.5340781323611736e-06, + "min_s": 3.3440301194787025e-06, + "max_s": 3.535998985171318e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017615193501114845, + "p50_s": 0.0017409750143997371, + "p90_s": 0.0018063198192976415, + "p95_s": 0.0018144879199098797, + "p99_s": 0.00182102240039967, + "min_s": 0.0017209270154125988, + "max_s": 0.0018226560205221176 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 9.466661140322685e-06, + "p50_s": 9.503972250968218e-06, + "p90_s": 9.542389307171106e-06, + "p95_s": 9.547191439196467e-06, + "p99_s": 9.551033144816756e-06, + "min_s": 9.34401759877801e-06, + "max_s": 9.551993571221828e-06 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.002172234351746738, + "p50_s": 0.0021623510401695967, + "p90_s": 0.0021976286079734564, + "p95_s": 0.002202038303948939, + "p99_s": 0.002205566060729325, + "min_s": 0.002147904015146196, + "max_s": 0.0022064479999244213 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.416017034401496e-06, + "p50_s": 6.399990525096655e-06, + "p90_s": 6.489630322903395e-06, + "p95_s": 6.500835297629237e-06, + "p99_s": 6.509799277409911e-06, + "min_s": 6.336020305752754e-06, + "max_s": 6.51204027235508e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.07115556466548394, + "p50_s": 0.07123408204643056, + "p90_s": 0.07148396357661113, + "p95_s": 0.0715151987678837, + "p99_s": 0.07154018692090176, + "min_s": 0.07068617799086496, + "max_s": 0.07154643395915627 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.6160109428068e-06, + "p50_s": 5.664012860506773e-06, + "p90_s": 5.715189035981893e-06, + "p95_s": 5.721586057916284e-06, + "p99_s": 5.726703675463796e-06, + "min_s": 5.456036888062954e-06, + "max_s": 5.727983079850674e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.07517527565748121, + "p50_s": 0.07530217699240893, + "p90_s": 0.07546080739703029, + "p95_s": 0.07548063619760798, + "p99_s": 0.0754964992380701, + "min_s": 0.07472318498184904, + "max_s": 0.07550046499818563 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.025796906343506027, + "p50_s": 0.025601482018828392, + "p90_s": 0.026077552419155835, + "p95_s": 0.026137061219196768, + "p99_s": 0.02618466825922951, + "min_s": 0.025592666992451996, + "max_s": 0.026196570019237697 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.560026961068313e-06, + "p50_s": 4.56004636362195e-06, + "p90_s": 4.726427141577005e-06, + "p95_s": 4.747224738821387e-06, + "p99_s": 4.763862816616893e-06, + "min_s": 4.352012183517218e-06, + "max_s": 4.768022336065769e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.975989961375793e-06, + "p50_s": 9.119976311922073e-06, + "p90_s": 9.299209341406823e-06, + "p95_s": 9.321613470092414e-06, + "p99_s": 9.33953677304089e-06, + "min_s": 8.463975973427296e-06, + "max_s": 9.34401759877801e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.5733294983704886e-06, + "p50_s": 3.5520060919225216e-06, + "p90_s": 3.6544050090014937e-06, + "p95_s": 3.667204873636365e-06, + "p99_s": 3.677444765344262e-06, + "min_s": 3.4879776649177074e-06, + "max_s": 3.6800047382712364e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.00451310898642987, + "p50_s": 0.004505342978518456, + "p90_s": 0.004527206195052713, + "p95_s": 0.004529939097119496, + "p99_s": 0.004532125418772921, + "min_s": 0.004501311981584877, + "max_s": 0.004532671999186277 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 9.279989171773195e-06, + "p50_s": 9.535986464470625e-06, + "p90_s": 9.753589984029531e-06, + "p95_s": 9.780790423974394e-06, + "p99_s": 9.802550775930286e-06, + "min_s": 8.495990186929703e-06, + "max_s": 9.807990863919258e-06 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.005867129017133266, + "p50_s": 0.005761470005381852, + "p90_s": 0.0060237164259888225, + "p95_s": 0.006056497228564694, + "p99_s": 0.006082721870625391, + "min_s": 0.005750639014877379, + "max_s": 0.006089278031140566 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.517337169498205e-06, + "p50_s": 6.52798917144537e-06, + "p90_s": 6.681610830128194e-06, + "p95_s": 6.700813537463546e-06, + "p99_s": 6.716175703331828e-06, + "min_s": 6.304006092250347e-06, + "max_s": 6.720016244798899e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.0607306759726877, + "p50_s": 0.058016403985675424, + "p90_s": 0.06506514557404444, + "p95_s": 0.06594623827259057, + "p99_s": 0.06665111243142748, + "min_s": 0.05734829296125099, + "max_s": 0.06682733097113669 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.6586771582563715e-06, + "p50_s": 5.567970219999552e-06, + "p90_s": 5.849602166563272e-06, + "p95_s": 5.884806159883737e-06, + "p99_s": 5.912969354540109e-06, + "min_s": 5.488051101565361e-06, + "max_s": 5.920010153204203e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.07119058868071686, + "p50_s": 0.06835102604236454, + "p90_s": 0.07540559160988779, + "p95_s": 0.07628741230582818, + "p99_s": 0.07699286886258051, + "min_s": 0.06805150699801743, + "max_s": 0.07716923300176859 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.013778018668138733, + "p50_s": 0.010043838003184646, + "p90_s": 0.02003887001192197, + "p95_s": 0.021288249013014137, + "p99_s": 0.02228775221388787, + "min_s": 0.008752589987125248, + "max_s": 0.022537628014106303 + }, + "block_total": { + "count": 3, + "mean_s": 0.5541369766773035, + "p50_s": 0.5570170399732888, + "p90_s": 0.5594101664144546, + "p95_s": 0.5597093072196002, + "p99_s": 0.5599486198637169, + "min_s": 0.5453854420338757, + "max_s": 0.560008448024746 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.5096157593264555, + "p50_s": 0.5090722020249814, + "p90_s": 0.5104809443932027, + "p95_s": 0.5106570371892303, + "p99_s": 0.5107979114260524, + "min_s": 0.5089419459691271, + "max_s": 0.510833129985258 + }, + "module_forward_checksum": 296883328.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 9093546496 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1367.4550000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1346.1589999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1125.328, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 666.7999999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 591.2480000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 588.7200000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 445.0400000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 305.08799999999974, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 261.40800000000036, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 210.1280000000006, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "aten::to", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 192.3839999999982, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 184.64000000000033, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 173.08799999999974, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 154.47999999999956, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 136.70399999999972, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "aten::empty", + "count": 34, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 132.73599999999925, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009298432, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 129.66399999999976, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 109.07200000000012, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 99.28000000000065, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::silu", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 99.15200000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 1084227584, + "input_shapes": "[[37810, 14336]]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json b/benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json new file mode 100644 index 0000000..ac52ed7 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -895169280.0, + "candidate_checksum": -895169280.0, + "baseline": { + "mean_s": 0.46958410650404403, + "p50_s": 0.4692447630222887, + "min_s": 0.46773986698826775, + "max_s": 0.47218111396068707 + }, + "ring": { + "mean_s": 0.4762414453798556, + "p50_s": 0.4739541109884158, + "min_s": 0.47236386401345953, + "max_s": 0.4925510979956016 + }, + "p50_improvement_percent": -1.0036016035203765 +} diff --git a/benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json b/benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json new file mode 100644 index 0000000..bb2af7c --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -168955712.0, + "candidate_checksum": -168955712.0, + "baseline": { + "mean_s": 0.4668582236530104, + "p50_s": 0.46715522548765875, + "min_s": 0.4650274849846028, + "max_s": 0.4681780419778079 + }, + "ring": { + "mean_s": 0.46820701367687434, + "p50_s": 0.46829428849741817, + "min_s": 0.4667344400077127, + "max_s": 0.46996626403415576 + }, + "p50_improvement_percent": -0.2438296625217884 +} diff --git a/benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json b/benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json new file mode 100644 index 0000000..62a8c4b --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -168955712.0, + "candidate_checksum": -168955712.0, + "baseline": { + "mean_s": 0.46774995284310233, + "p50_s": 0.4674104950099718, + "min_s": 0.4658880060305819, + "max_s": 0.47020024503581226 + }, + "ring": { + "mean_s": 0.49012390718174476, + "p50_s": 0.4898388920119032, + "min_s": 0.48933237104211, + "max_s": 0.4917365289875306 + }, + "p50_improvement_percent": -4.798436757705438 +} diff --git a/benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json b/benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json new file mode 100644 index 0000000..b5dc6d3 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -168955712.0, + "candidate_checksum": -168955712.0, + "baseline": { + "mean_s": 0.46479812582644325, + "p50_s": 0.4648084765067324, + "min_s": 0.46373577596386895, + "max_s": 0.4655176450032741 + }, + "ring": { + "mean_s": 0.5324759804934729, + "p50_s": 0.5321614120039158, + "min_s": 0.5311248479993083, + "max_s": 0.5355711780139245 + }, + "p50_improvement_percent": -14.49047057045394 +} diff --git a/benchmarks/gb10-cute-qkv-block24-ring.json b/benchmarks/gb10-cute-qkv-block24-ring.json new file mode 100644 index 0000000..87e5dba --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block24-ring.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.003741738328244537, + "p50_s": 0.003866238985210657, + "p90_s": 0.003901209402829409, + "p95_s": 0.0039055807050317527, + "p99_s": 0.003909077746793627, + "min_s": 0.0034490239922888577, + "max_s": 0.0039099520072340965 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.011052168323658407, + "p50_s": 0.010806893988046795, + "p90_s": 0.012067155598197134, + "p95_s": 0.012224688299465925, + "p99_s": 0.012350714460480958, + "min_s": 0.009967389982193708, + "max_s": 0.012382221000734717 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.536014214158058e-06, + "p50_s": 4.143978003412485e-06, + "p90_s": 7.62563431635499e-06, + "p95_s": 8.060841355472804e-06, + "p99_s": 8.409006986767054e-06, + "min_s": 3.968016244471073e-06, + "max_s": 8.496048394590616e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.4229328371584415e-05, + "p50_s": 8.367991540580988e-06, + "p90_s": 2.274239668622613e-05, + "p95_s": 2.4539197329431773e-05, + "p99_s": 2.597663784399629e-05, + "min_s": 7.983995601534843e-06, + "max_s": 2.6335997972637415e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.509320473919312e-06, + "p50_s": 3.4240074455738068e-06, + "p90_s": 3.6927871406078338e-06, + "p95_s": 3.7263846024870874e-06, + "p99_s": 3.7532625719904902e-06, + "min_s": 3.343971911817789e-06, + "max_s": 3.7599820643663406e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.0017316586648424466, + "p50_s": 0.0017092640046030283, + "p90_s": 0.0017795104067772627, + "p95_s": 0.001788291207049042, + "p99_s": 0.0017953158472664653, + "min_s": 0.0016886399826034904, + "max_s": 0.0017970720073208213 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 1.3424005980292955e-05, + "p50_s": 1.1440017260611057e-05, + "p90_s": 1.796798314899206e-05, + "p95_s": 1.8783978885039683e-05, + "p99_s": 1.9436775473877788e-05, + "min_s": 9.232026059180498e-06, + "max_s": 1.9599974621087313e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002303071009616057, + "p50_s": 0.0022418550215661526, + "p90_s": 0.002481432631611824, + "p95_s": 0.002511379832867533, + "p99_s": 0.0025353375938721003, + "min_s": 0.0021260309731587768, + "max_s": 0.002541327034123242 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.789341568946838e-06, + "p50_s": 6.608024705201387e-06, + "p90_s": 7.401604671031237e-06, + "p95_s": 7.500802166759968e-06, + "p99_s": 7.580160163342953e-06, + "min_s": 6.160000339150429e-06, + "max_s": 7.599999662488699e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.029993231679933768, + "p50_s": 0.026411275030113757, + "p90_s": 0.03558764460030943, + "p95_s": 0.03673469079658389, + "p99_s": 0.03765232775360346, + "min_s": 0.025686683016829193, + "max_s": 0.03788173699285835 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 6.229345065852006e-06, + "p50_s": 5.4559786804020405e-06, + "p90_s": 7.32481712475419e-06, + "p95_s": 7.558421930298209e-06, + "p99_s": 7.745305774733424e-06, + "min_s": 5.44002978131175e-06, + "max_s": 7.792026735842228e-06 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.03412259600978965, + "p50_s": 0.030302010010927916, + "p90_s": 0.039898885227739815, + "p95_s": 0.041098494629841296, + "p99_s": 0.04205818215152249, + "min_s": 0.029767673986498266, + "max_s": 0.042298104031942785 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 1.5023998760928711e-05, + "p50_s": 1.2496020644903183e-05, + "p90_s": 1.977919600903988e-05, + "p95_s": 2.0689592929556962e-05, + "p99_s": 2.1417910465970633e-05, + "min_s": 1.0975985787808895e-05, + "max_s": 2.1599989850074053e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012241016685341796, + "p50_s": 0.01193398202303797, + "p90_s": 0.012705834838561714, + "p95_s": 0.012802316440502183, + "p99_s": 0.012879501722054557, + "min_s": 0.011890269990544766, + "max_s": 0.01289879804244265 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.014036514330655336, + "p50_s": 0.0053002709755674005, + "p90_s": 0.026539997407235207, + "p95_s": 0.029194963211193678, + "p99_s": 0.031318935854360457, + "min_s": 0.004959343001246452, + "max_s": 0.031849929015152156 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.007838361353302995, + "p50_s": 0.005219135025981814, + "p90_s": 0.011728727829176933, + "p95_s": 0.012542426929576323, + "p99_s": 0.013193386209895835, + "min_s": 0.00493982300395146, + "max_s": 0.013356126029975712 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.01099991767356793, + "p50_s": 0.005040751013439149, + "p90_s": 0.019409106194507333, + "p95_s": 0.021205150592140853, + "p99_s": 0.02264198611024767, + "min_s": 0.004957807017490268, + "max_s": 0.023001194989774376 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2562586222969306, + "p50_s": 0.25644913397263736, + "p90_s": 0.2567798731615767, + "p95_s": 0.2568212155601941, + "p99_s": 0.25685428947908806, + "min_s": 0.2554641749593429, + "max_s": 0.2568625579588115 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.00469084734019513, + "p50_s": 0.004691840033046901, + "p90_s": 0.004693656810559332, + "p95_s": 0.004693883907748386, + "p99_s": 0.0046940655854996296, + "min_s": 0.0046865909826010466, + "max_s": 0.00469411100493744 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 4.229329836865266e-06, + "p50_s": 4.1120219975709915e-06, + "p90_s": 4.470394924283028e-06, + "p95_s": 4.515191540122033e-06, + "p99_s": 4.551028832793236e-06, + "min_s": 4.01597935706377e-06, + "max_s": 4.559988155961037e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 9.45067343612512e-06, + "p50_s": 9.440002031624317e-06, + "p90_s": 1.010557170957327e-05, + "p95_s": 1.0188767919316889e-05, + "p99_s": 1.0255324887111782e-05, + "min_s": 8.640054147690535e-06, + "max_s": 1.0271964129060507e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.5146561761697135e-06, + "p50_s": 3.4240074455738068e-06, + "p90_s": 3.667175769805908e-06, + "p95_s": 3.697571810334921e-06, + "p99_s": 3.7218886427581313e-06, + "min_s": 3.3919932320713997e-06, + "max_s": 3.7279678508639336e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.002265583665575832, + "p50_s": 0.0022627509897574782, + "p90_s": 0.002275923010893166, + "p95_s": 0.002277569513535127, + "p99_s": 0.0022788867156486956, + "min_s": 0.00225478399079293, + "max_s": 0.002279216016177088 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 1.0005314834415913e-05, + "p50_s": 9.807990863919258e-06, + "p90_s": 1.0319985449314117e-05, + "p95_s": 1.0383984772488475e-05, + "p99_s": 1.0435184231027961e-05, + "min_s": 9.759969543665648e-06, + "max_s": 1.0447984095662832e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002860164638453474, + "p50_s": 0.0028614879702217877, + "p90_s": 0.002871010371018201, + "p95_s": 0.002872200671117753, + "p99_s": 0.0028731529111973944, + "min_s": 0.0028456149739213288, + "max_s": 0.0028733909712173045 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.378686521202326e-06, + "p50_s": 6.432004738599062e-06, + "p90_s": 6.508838851004839e-06, + "p95_s": 6.518443115055561e-06, + "p99_s": 6.526126526296139e-06, + "min_s": 6.176007445901632e-06, + "max_s": 6.528047379106283e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.007843875675462186, + "p50_s": 0.007834365998860449, + "p90_s": 0.007871806004550308, + "p95_s": 0.00787648600526154, + "p99_s": 0.007880230005830527, + "min_s": 0.007816095021553338, + "max_s": 0.007881166005972773 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.5093357029060526e-06, + "p50_s": 5.552021320909262e-06, + "p90_s": 5.6416145525872706e-06, + "p95_s": 5.652813706547021e-06, + "p99_s": 5.661773029714823e-06, + "min_s": 5.311972927302122e-06, + "max_s": 5.664012860506773e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.013048968006235858, + "p50_s": 0.013041116995736957, + "p90_s": 0.013074909010902048, + "p95_s": 0.013079133012797683, + "p99_s": 0.013082512214314192, + "min_s": 0.013022430008277297, + "max_s": 0.01308335701469332 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.008436798331482956, + "p50_s": 0.008493119035847485, + "p90_s": 0.008522289409302174, + "p95_s": 0.008525935705984011, + "p99_s": 0.00852885274332948, + "min_s": 0.008287693955935538, + "max_s": 0.008529582002665848 + }, + "norm2": { + "count": 3, + "mean_s": 0.0034889166515010097, + "p50_s": 0.0034742390271276236, + "p90_s": 0.0035121661610901357, + "p95_s": 0.0035169070528354492, + "p99_s": 0.0035206997662317004, + "min_s": 0.0034708629827946424, + "max_s": 0.0035216479445807636 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010772093994698176, + "p50_s": 0.010788750019855797, + "p90_s": 0.010858983569778501, + "p95_s": 0.01086776276351884, + "p99_s": 0.01087478611851111, + "min_s": 0.010650990006979555, + "max_s": 0.010876541957259178 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.058626169959704e-06, + "p50_s": 3.951950930058956e-06, + "p90_s": 4.271953366696835e-06, + "p95_s": 4.3119536712765695e-06, + "p99_s": 4.343953914940357e-06, + "min_s": 3.871973603963852e-06, + "max_s": 4.351953975856304e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.293330514182648e-06, + "p50_s": 8.01600981503725e-06, + "p90_s": 8.860812522470951e-06, + "p95_s": 8.966412860900165e-06, + "p99_s": 9.050893131643534e-06, + "min_s": 7.791968528181314e-06, + "max_s": 9.072013199329376e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.4186717433234057e-06, + "p50_s": 3.456021659076214e-06, + "p90_s": 3.4816330298781395e-06, + "p95_s": 3.48483445122838e-06, + "p99_s": 3.4873955883085728e-06, + "min_s": 3.311957698315382e-06, + "max_s": 3.488035872578621e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017874666567270954, + "p50_s": 0.0017595840035937726, + "p90_s": 0.0018278975854627787, + "p95_s": 0.0018364367831964047, + "p99_s": 0.0018432681413833051, + "min_s": 0.0017578399856574833, + "max_s": 0.0018449759809300303 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 9.296015681078037e-06, + "p50_s": 9.328010492026806e-06, + "p90_s": 9.340816177427769e-06, + "p95_s": 9.342416888102889e-06, + "p99_s": 9.343697456642985e-06, + "min_s": 9.216018952429295e-06, + "max_s": 9.34401759877801e-06 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.002177828651232024, + "p50_s": 0.002151614986360073, + "p90_s": 0.0022210165858268737, + "p95_s": 0.002229691785760224, + "p99_s": 0.002236631945706904, + "min_s": 0.002143503981642425, + "max_s": 0.002238366985693574 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.623993006845315e-06, + "p50_s": 6.607966497540474e-06, + "p90_s": 6.659189239144325e-06, + "p95_s": 6.665592081844807e-06, + "p99_s": 6.670714356005192e-06, + "min_s": 6.592017598450184e-06, + "max_s": 6.671994924545288e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.03496598266065121, + "p50_s": 0.03502911300165579, + "p90_s": 0.03507212101249024, + "p95_s": 0.03507749701384455, + "p99_s": 0.035081797814928, + "min_s": 0.03478596196509898, + "max_s": 0.03508287301519886 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.49866429840525e-06, + "p50_s": 5.4559786804020405e-06, + "p90_s": 5.622406024485826e-06, + "p95_s": 5.643209442496299e-06, + "p99_s": 5.659852176904679e-06, + "min_s": 5.376001354306936e-06, + "max_s": 5.664012860506773e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.03901222701339672, + "p50_s": 0.03901776799466461, + "p90_s": 0.039129883213900034, + "p95_s": 0.03914389761630446, + "p99_s": 0.039155109138228, + "min_s": 0.038861001026816666, + "max_s": 0.039157912018708885 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.02408325932143877, + "p50_s": 0.0241031949990429, + "p90_s": 0.024252827803138644, + "p95_s": 0.02427153190365061, + "p99_s": 0.024286495184060185, + "min_s": 0.02385634696111083, + "max_s": 0.02429023600416258 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.234665539115667e-06, + "p50_s": 4.01597935706377e-06, + "p90_s": 4.617613740265369e-06, + "p95_s": 4.692818038165569e-06, + "p99_s": 4.7529814764857296e-06, + "min_s": 3.9199949242174625e-06, + "max_s": 4.768022336065769e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 7.589328257987897e-06, + "p50_s": 7.71199120208621e-06, + "p90_s": 7.929594721645116e-06, + "p95_s": 7.95679516158998e-06, + "p99_s": 7.978555513545871e-06, + "min_s": 7.071997970342636e-06, + "max_s": 7.983995601534843e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.722670953720808e-06, + "p50_s": 3.503984771668911e-06, + "p90_s": 4.195212386548519e-06, + "p95_s": 4.28161583840847e-06, + "p99_s": 4.3507385998964315e-06, + "min_s": 3.296008799225092e-06, + "max_s": 4.368019290268421e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.004541535027480374, + "p50_s": 0.004574159043841064, + "p90_s": 0.004587253439240158, + "p95_s": 0.004588890238665044, + "p99_s": 0.004590199678204954, + "min_s": 0.004459919000510126, + "max_s": 0.004590527038089931 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 8.880005528529486e-06, + "p50_s": 8.608039934188128e-06, + "p90_s": 9.3247857876122e-06, + "p95_s": 9.414379019290208e-06, + "p99_s": 9.486053604632616e-06, + "min_s": 8.52800440043211e-06, + "max_s": 9.503972250968218e-06 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.005774937667107831, + "p50_s": 0.005734590988140553, + "p90_s": 0.0058928117970936, + "p95_s": 0.005912589398212731, + "p99_s": 0.005928411479108036, + "min_s": 0.005657855013851076, + "max_s": 0.005932366999331862 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.197350254903237e-06, + "p50_s": 6.128044333308935e-06, + "p90_s": 6.396824028342963e-06, + "p95_s": 6.430421490222216e-06, + "p99_s": 6.457299459725618e-06, + "min_s": 5.999987479299307e-06, + "max_s": 6.464018952101469e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.05555110399533684, + "p50_s": 0.05553032597526908, + "p90_s": 0.05586563399992883, + "p95_s": 0.0559075475030113, + "p99_s": 0.055941078305477274, + "min_s": 0.05517352500464767, + "max_s": 0.05594946100609377 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.4613337852060795e-06, + "p50_s": 5.4879928939044476e-06, + "p90_s": 5.6543736718595025e-06, + "p95_s": 5.6751712691038845e-06, + "p99_s": 5.691809346899391e-06, + "min_s": 5.200039595365524e-06, + "max_s": 5.695968866348267e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.06594285130267963, + "p50_s": 0.06600016396259889, + "p90_s": 0.06628018877236172, + "p95_s": 0.06631519187358208, + "p99_s": 0.06634319435455836, + "min_s": 0.06547819497063756, + "max_s": 0.06635019497480243 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.013789538682127992, + "p50_s": 0.00966468604747206, + "p90_s": 0.019895839621312916, + "p95_s": 0.02117473381804302, + "p99_s": 0.022197849175427106, + "min_s": 0.009250301984138787, + "max_s": 0.02245362801477313 + }, + "block_total": { + "count": 3, + "mean_s": 0.5336582703360667, + "p50_s": 0.5072318860329688, + "p90_s": 0.5710783508140594, + "p95_s": 0.5790591589116957, + "p99_s": 0.5854438053898048, + "min_s": 0.5067029579658993, + "max_s": 0.587039967009332 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.47701150766806677, + "p50_s": 0.4728726129978895, + "p90_s": 0.48509467700496317, + "p95_s": 0.48662243500584734, + "p99_s": 0.48784464140655476, + "min_s": 0.4700117169995792, + "max_s": 0.48815019300673157 + }, + "module_forward_checksum": 296883328.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 7351755264 + }, + "profiler_top_events": [ + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 581.328, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 512.4480000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 422.6239999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 247.44000000000028, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 13, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 218.31999999999857, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 209.37599999999975, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 187.68000000000043, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 175.6159999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 164.1119999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 139.95200000000023, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 123.92000000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "aten::empty", + "count": 29, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 123.24799999999709, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 6267507200, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 121.84000000000015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::zeros", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 106.24000000000069, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 100.2559999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 95.00799999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 88.44800000000032, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 76.6240000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "aten::silu", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 64.27199999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 1084227584, + "input_shapes": "[[37810, 14336]]" + }, + { + "key": "aten::copy_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 63.455999999999904, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1], [], []]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-block49-alternating.json b/benchmarks/gb10-cute-qkv-block49-alternating.json new file mode 100644 index 0000000..4d93e4e --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block49-alternating.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "block_index": 49, + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -1799476346880.0, + "candidate_checksum": -1799476346880.0, + "baseline": { + "mean_s": 0.4619793013262097, + "p50_s": 0.4623519679880701, + "min_s": 0.46040739299496636, + "max_s": 0.46288129599997774 + }, + "ring": { + "mean_s": 0.46425590649596415, + "p50_s": 0.46476577199064195, + "min_s": 0.4613973450032063, + "max_s": 0.46552810998400673 + }, + "p50_improvement_percent": -0.5220706668721542 +} diff --git a/benchmarks/gb10-cute-qkv-block49-baseline.json b/benchmarks/gb10-cute-qkv-block49-baseline.json new file mode 100644 index 0000000..d26063d --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block49-baseline.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 49, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.0036217596692343554, + "p50_s": 0.00359812198439613, + "p90_s": 0.0037101987865753474, + "p95_s": 0.0037242083868477493, + "p99_s": 0.0037354160670656712, + "min_s": 0.0035289390361867845, + "max_s": 0.0037382179871201515 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.01067996733278657, + "p50_s": 0.010696462995838374, + "p90_s": 0.011010011786129327, + "p95_s": 0.011049205384915696, + "p99_s": 0.011080560263944791, + "min_s": 0.010255040018819273, + "max_s": 0.011088398983702064 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.525342809657256e-06, + "p50_s": 4.208006430417299e-06, + "p90_s": 7.382419425994158e-06, + "p95_s": 7.779221050441265e-06, + "p99_s": 8.096662349998951e-06, + "min_s": 4.191999323666096e-06, + "max_s": 8.176022674888372e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.5418666104475658e-05, + "p50_s": 8.544011507183313e-06, + "p90_s": 2.5260786060243845e-05, + "p95_s": 2.7350382879376408e-05, + "p99_s": 2.9022060334682465e-05, + "min_s": 8.27200710773468e-06, + "max_s": 2.9439979698508978e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.674688438574473e-06, + "p50_s": 3.6479905247688293e-06, + "p90_s": 3.827223554253578e-06, + "p95_s": 3.849627682939172e-06, + "p99_s": 3.867550985887647e-06, + "min_s": 3.5040429793298244e-06, + "max_s": 3.872031811624765e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.0017329036879042785, + "p50_s": 0.0017396770417690277, + "p90_s": 0.0017431330401450396, + "p95_s": 0.001743565039942041, + "p99_s": 0.0017439106397796422, + "min_s": 0.001715036982204765, + "max_s": 0.0017439970397390425 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 1.3610658546288809e-05, + "p50_s": 1.2303993571549654e-05, + "p90_s": 1.84479751624167e-05, + "p95_s": 1.9215972861275078e-05, + "p99_s": 1.983037102036178e-05, + "min_s": 8.544011507183313e-06, + "max_s": 1.9983970560133457e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.0022982683343191943, + "p50_s": 0.0023595320526510477, + "p90_s": 0.0024083896074444055, + "p95_s": 0.0024144968017935755, + "p99_s": 0.002419382557272911, + "min_s": 0.0021146689541637897, + "max_s": 0.002420603996142745 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 7.045338861644268e-06, + "p50_s": 6.511982064694166e-06, + "p90_s": 7.868779357522726e-06, + "p95_s": 8.038379019126295e-06, + "p99_s": 8.174058748409153e-06, + "min_s": 6.416055839508772e-06, + "max_s": 8.207978680729866e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.02569509532380228, + "p50_s": 0.02566098398528993, + "p90_s": 0.026279260776937007, + "p95_s": 0.026356545375892895, + "p99_s": 0.0264183730550576, + "min_s": 0.02499047201126814, + "max_s": 0.026433829974848777 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 6.298670389999946e-06, + "p50_s": 5.504000000655651e-06, + "p90_s": 7.46243167668581e-06, + "p95_s": 7.70723563618958e-06, + "p99_s": 7.903078803792597e-06, + "min_s": 5.439971573650837e-06, + "max_s": 7.95203959569335e-06 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.029824565979652107, + "p50_s": 0.02990254497854039, + "p90_s": 0.030253315379377456, + "p95_s": 0.030297161679482085, + "p99_s": 0.030332238719565793, + "min_s": 0.02923014498082921, + "max_s": 0.03034100797958672 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 1.4346345172574123e-05, + "p50_s": 1.1824013199657202e-05, + "p90_s": 1.82872056029737e-05, + "p95_s": 1.909510465338826e-05, + "p99_s": 1.9741423893719913e-05, + "min_s": 1.1312018614262342e-05, + "max_s": 1.9903003703802824e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012300620651027808, + "p50_s": 0.012445547967217863, + "p90_s": 0.012557216011919082, + "p95_s": 0.012571174517506735, + "p99_s": 0.012582341321976855, + "min_s": 0.011871180962771177, + "max_s": 0.012585133023094386 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.005062365322373807, + "p50_s": 0.005007288011256605, + "p90_s": 0.005225092789623886, + "p95_s": 0.005252318386919796, + "p99_s": 0.005274098864756525, + "min_s": 0.00490026397164911, + "max_s": 0.005279543984215707 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.00505693598339955, + "p50_s": 0.00496911199297756, + "p90_s": 0.005203018360771239, + "p95_s": 0.005232256656745448, + "p99_s": 0.005255647293524816, + "min_s": 0.004940201004501432, + "max_s": 0.005261494952719659 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.00494270701892674, + "p50_s": 0.00493679300416261, + "p90_s": 0.004955710633657872, + "p95_s": 0.004958075337344781, + "p99_s": 0.004959967100294307, + "min_s": 0.004930888011585921, + "max_s": 0.0049604400410316885 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2531811236597908, + "p50_s": 0.2535298540024087, + "p90_s": 0.25376300599891694, + "p95_s": 0.25379214999848043, + "p99_s": 0.2538154651981313, + "min_s": 0.25219222297891974, + "max_s": 0.253821293998044 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.0048227393223593635, + "p50_s": 0.004690104979090393, + "p90_s": 0.005012331414036453, + "p95_s": 0.00505260971840471, + "p99_s": 0.005084832361899316, + "min_s": 0.004685224965214729, + "max_s": 0.005092888022772968 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 4.410685505717993e-06, + "p50_s": 4.272034857422113e-06, + "p90_s": 4.822446499019861e-06, + "p95_s": 4.89124795421958e-06, + "p99_s": 4.946289118379355e-06, + "min_s": 3.999972250312567e-06, + "max_s": 4.960049409419298e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 9.290330732862154e-06, + "p50_s": 9.215029422193766e-06, + "p90_s": 9.74057475104928e-06, + "p95_s": 9.806267917156219e-06, + "p99_s": 9.858822450041771e-06, + "min_s": 8.78400169312954e-06, + "max_s": 9.871961083263159e-06 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.445330852021774e-06, + "p50_s": 3.4240074455738068e-06, + "p90_s": 3.5136006772518156e-06, + "p95_s": 3.5247998312115673e-06, + "p99_s": 3.533759154379368e-06, + "min_s": 3.375986125320196e-06, + "max_s": 3.535998985171318e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.0022702256489234665, + "p50_s": 0.0022497729514725506, + "p90_s": 0.0023021497880108656, + "p95_s": 0.0023086968925781545, + "p99_s": 0.0023139345762319863, + "min_s": 0.002245659998152405, + "max_s": 0.002315243997145444 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 9.96800372377038e-06, + "p50_s": 9.695999324321747e-06, + "p90_s": 1.0348809882998465e-05, + "p95_s": 1.0430411202833055e-05, + "p99_s": 1.0495692258700728e-05, + "min_s": 9.695999324321747e-06, + "max_s": 1.0512012522667646e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002868635676956425, + "p50_s": 0.0028517399914562702, + "p90_s": 0.002900878433138132, + "p95_s": 0.002907020738348365, + "p99_s": 0.0029119345825165508, + "min_s": 0.0028410039958544075, + "max_s": 0.0029131630435585976 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.453308742493391e-06, + "p50_s": 6.511982064694166e-06, + "p90_s": 6.588769610971213e-06, + "p95_s": 6.5983680542558435e-06, + "p99_s": 6.606046808883548e-06, + "min_s": 6.239977665245533e-06, + "max_s": 6.607966497540474e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.008090866671409458, + "p50_s": 0.007834755000658333, + "p90_s": 0.008461467805318534, + "p95_s": 0.00853980690590106, + "p99_s": 0.008602478186367078, + "min_s": 0.007819699007086456, + "max_s": 0.008618146006483585 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.562673322856426e-06, + "p50_s": 5.4879928939044476e-06, + "p90_s": 5.795189645141363e-06, + "p95_s": 5.833589239045977e-06, + "p99_s": 5.864308914169669e-06, + "min_s": 5.328038241714239e-06, + "max_s": 5.871988832950592e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.013308762339875102, + "p50_s": 0.013069211039692163, + "p90_s": 0.013700198195874692, + "p95_s": 0.013779071590397507, + "p99_s": 0.013842170306015759, + "min_s": 0.01299913099501282, + "max_s": 0.013857944984920323 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.008683037323256334, + "p50_s": 0.008554771018680185, + "p90_s": 0.008906488574575632, + "p95_s": 0.008950453269062564, + "p99_s": 0.008985625024652109, + "min_s": 0.008499922987539321, + "max_s": 0.008994417963549495 + }, + "norm2": { + "count": 3, + "mean_s": 0.003726057999301702, + "p50_s": 0.003699098015204072, + "p90_s": 0.0038113028276711702, + "p95_s": 0.0038253284292295575, + "p99_s": 0.0038365489104762673, + "min_s": 0.0036397219519130886, + "max_s": 0.003839354030787945 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010754756338428706, + "p50_s": 0.010686943016480654, + "p90_s": 0.010987435781862587, + "p95_s": 0.011024997377535328, + "p99_s": 0.011055046654073522, + "min_s": 0.010514767025597394, + "max_s": 0.01106255897320807 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.288003159066041e-06, + "p50_s": 4.031986463814974e-06, + "p90_s": 4.684797022491694e-06, + "p95_s": 4.766398342326284e-06, + "p99_s": 4.831679398193956e-06, + "min_s": 3.984023351222277e-06, + "max_s": 4.847999662160873e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.207998083283504e-06, + "p50_s": 7.983995601534843e-06, + "p90_s": 8.828798308968544e-06, + "p95_s": 8.934398647397758e-06, + "p99_s": 9.018878918141127e-06, + "min_s": 7.599999662488699e-06, + "max_s": 9.039998985826969e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.503984771668911e-06, + "p50_s": 3.4879776649177074e-06, + "p90_s": 3.5775708965957167e-06, + "p95_s": 3.5887700505554675e-06, + "p99_s": 3.5977293737232683e-06, + "min_s": 3.4240074455738068e-06, + "max_s": 3.5999692045152187e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017449889952937763, + "p50_s": 0.0017389570130035281, + "p90_s": 0.0017614722019061446, + "p95_s": 0.0017642866005189717, + "p99_s": 0.0017665381194092334, + "min_s": 0.0017289089737460017, + "max_s": 0.0017671009991317987 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 1.4506649070729813e-05, + "p50_s": 9.312003385275602e-06, + "p90_s": 2.1907163318246604e-05, + "p95_s": 2.3481558309867977e-05, + "p99_s": 2.4741074303165077e-05, + "min_s": 9.15199052542448e-06, + "max_s": 2.5055953301489353e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.002230454992968589, + "p50_s": 0.002145772974472493, + "p90_s": 0.002364140178542584, + "p95_s": 0.0023914360790513456, + "p99_s": 0.0024132727994583547, + "min_s": 0.0021268600248731673, + "max_s": 0.002418731979560107 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.261320474247138e-06, + "p50_s": 6.27199187874794e-06, + "p90_s": 6.425566971302032e-06, + "p95_s": 6.444763857871294e-06, + "p99_s": 6.460121367126704e-06, + "min_s": 6.0480087995529175e-06, + "max_s": 6.4639607444405556e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.03514027533431848, + "p50_s": 0.03508026502095163, + "p90_s": 0.03532921141013503, + "p95_s": 0.03536032970878296, + "p99_s": 0.03538522434770129, + "min_s": 0.03494911297457293, + "max_s": 0.03539144800743088 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.6320180495580035e-06, + "p50_s": 5.631998647004366e-06, + "p90_s": 5.657610017806292e-06, + "p95_s": 5.660811439156532e-06, + "p99_s": 5.663372576236725e-06, + "min_s": 5.600042641162872e-06, + "max_s": 5.664012860506773e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.039202652658180646, + "p50_s": 0.03935540997190401, + "p90_s": 0.039365841995459054, + "p95_s": 0.039367145998403436, + "p99_s": 0.039368189200758935, + "min_s": 0.03888409800129011, + "max_s": 0.03936845000134781 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.024196713328516733, + "p50_s": 0.024238202022388577, + "p90_s": 0.024260946782305837, + "p95_s": 0.024263789877295494, + "p99_s": 0.02426606435328722, + "min_s": 0.024085304990876466, + "max_s": 0.02426663297228515 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.272015454868476e-06, + "p50_s": 4.384026397019625e-06, + "p90_s": 4.435202572494745e-06, + "p95_s": 4.441599594429136e-06, + "p99_s": 4.446717211976647e-06, + "min_s": 3.984023351222277e-06, + "max_s": 4.447996616363525e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.181338974585136e-06, + "p50_s": 8.447968866676092e-06, + "p90_s": 8.460821118205785e-06, + "p95_s": 8.462427649646997e-06, + "p99_s": 8.463712874799967e-06, + "min_s": 7.632013875991106e-06, + "max_s": 8.464034181088209e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.4186717433234057e-06, + "p50_s": 3.4240074455738068e-06, + "p90_s": 3.4240074455738068e-06, + "p95_s": 3.4240074455738068e-06, + "p99_s": 3.4240074455738068e-06, + "min_s": 3.4080003388226032e-06, + "max_s": 3.4240074455738068e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.004601721011567861, + "p50_s": 0.0045881689875386655, + "p90_s": 0.004628655395936221, + "p95_s": 0.004633716196985916, + "p99_s": 0.004637764837825671, + "min_s": 0.004578217049129307, + "max_s": 0.00463877699803561 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 1.3728005190690359e-05, + "p50_s": 1.2575997970998287e-05, + "p90_s": 1.8028798513114454e-05, + "p95_s": 1.8710398580878974e-05, + "p99_s": 1.925567863509059e-05, + "min_s": 9.216018952429295e-06, + "max_s": 1.9391998648643494e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.005966662662103772, + "p50_s": 0.005981990019790828, + "p90_s": 0.0060327299637719985, + "p95_s": 0.006039072456769645, + "p99_s": 0.006044146451167762, + "min_s": 0.005872583016753197, + "max_s": 0.0060454149497672915 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.144012634952863e-06, + "p50_s": 6.176007445901632e-06, + "p90_s": 6.30401773378253e-06, + "p95_s": 6.320019019767642e-06, + "p99_s": 6.332820048555731e-06, + "min_s": 5.920010153204203e-06, + "max_s": 6.336020305752754e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.05552383201817671, + "p50_s": 0.05537804099731147, + "p90_s": 0.055793770635500554, + "p95_s": 0.055845736840274184, + "p99_s": 0.055887309804093094, + "min_s": 0.05529575201217085, + "max_s": 0.05589770304504782 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.546646813551585e-06, + "p50_s": 5.504000000655651e-06, + "p90_s": 5.619158037006855e-06, + "p95_s": 5.633552791550755e-06, + "p99_s": 5.645068595185876e-06, + "min_s": 5.4879928939044476e-06, + "max_s": 5.647947546094656e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.06617354867436613, + "p50_s": 0.06602008803747594, + "p90_s": 0.06640077200718224, + "p95_s": 0.06644835750339553, + "p99_s": 0.06648642590036616, + "min_s": 0.06600461498601362, + "max_s": 0.06649594299960881 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.014910061300421754, + "p50_s": 0.008941200969275087, + "p90_s": 0.023517421761061995, + "p95_s": 0.02533944936003536, + "p99_s": 0.02679707143921405, + "min_s": 0.008627505972981453, + "max_s": 0.027161476959008723 + }, + "block_total": { + "count": 3, + "mean_s": 0.5105496386337715, + "p50_s": 0.5060367499827407, + "p90_s": 0.5189121219678782, + "p95_s": 0.5205215434660204, + "p99_s": 0.5218090806645341, + "min_s": 0.5034812009544112, + "max_s": 0.5221309649641626 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.4635661483431856, + "p50_s": 0.4635968530201353, + "p90_s": 0.4642748417914845, + "p95_s": 0.46435959038790314, + "p99_s": 0.46442738926503807, + "min_s": 0.46265725302509964, + "max_s": 0.4644443389843218 + }, + "module_forward_checksum": -778077863936.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 9093546496 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1357.5660000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1349.8219999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 380.12699999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 333.52000000000044, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 282.47900000000027, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 191.07200000000012, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 165.34399999999982, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 150.27199999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 131.98399999999992, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 127.2159999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::to", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 112.27199999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 111.93600000000015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 110.38299999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 109.64799999999968, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::empty", + "count": 34, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 104.27200000000116, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009298432, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 95.03900000000067, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 88.20800000000008, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 71.7430000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 66.44799999999987, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 62.83199999999988, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-block49-ring.json b/benchmarks/gb10-cute-qkv-block49-ring.json new file mode 100644 index 0000000..64e1597 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-block49-ring.json @@ -0,0 +1,852 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 49, + "attention": "sage2", + "warmup": 2, + "iterations": 3, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 3, + "mean_s": 0.003877114000109335, + "p50_s": 0.003574107016902417, + "p90_s": 0.004309529426973313, + "p95_s": 0.0044014572282321755, + "p99_s": 0.004474999469239265, + "min_s": 0.0035638499539345503, + "max_s": 0.004493385029491037 + }, + "modulate_msa": { + "count": 3, + "mean_s": 0.011260746008095643, + "p50_s": 0.010931248019915074, + "p90_s": 0.012151918408926576, + "p95_s": 0.012304502207553015, + "p99_s": 0.012426569246454165, + "min_s": 0.0103939039981924, + "max_s": 0.012457086006179452 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 3, + "mean_s": 5.386633953700463e-06, + "p50_s": 4.335946869105101e-06, + "p90_s": 7.0239766500890255e-06, + "p95_s": 7.359980372712016e-06, + "p99_s": 7.628783350810409e-06, + "min_s": 4.127970896661282e-06, + "max_s": 7.695984095335007e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.5333333673576515e-05, + "p50_s": 8.84797191247344e-06, + "p90_s": 2.498881658539176e-05, + "p95_s": 2.700642216950655e-05, + "p99_s": 2.862050663679838e-05, + "min_s": 8.128001354634762e-06, + "max_s": 2.902402775362134e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 3, + "mean_s": 3.562658093869686e-06, + "p50_s": 3.4239492379128933e-06, + "p90_s": 3.782415296882391e-06, + "p95_s": 3.827223554253578e-06, + "p99_s": 3.8630701601505284e-06, + "min_s": 3.3919932320713997e-06, + "max_s": 3.872031811624765e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 3, + "mean_s": 0.0019726533403930566, + "p50_s": 0.001826349995099008, + "p90_s": 0.002202194812707603, + "p95_s": 0.0022491754149086777, + "p99_s": 0.0022867598966695366, + "min_s": 0.0017954540089704096, + "max_s": 0.0022961560171097517 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 3, + "mean_s": 1.755197687695424e-05, + "p50_s": 1.7903978005051613e-05, + "p90_s": 2.3907190188765527e-05, + "p95_s": 2.4657591711729766e-05, + "p99_s": 2.5257912930101157e-05, + "min_s": 9.343959391117096e-06, + "max_s": 2.5407993234694004e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002370401343796402, + "p50_s": 0.0021540760062634945, + "p90_s": 0.0026746008079499006, + "p95_s": 0.0027396664081607014, + "p99_s": 0.0027917188883293423, + "min_s": 0.00215239601675421, + "max_s": 0.002804732008371502 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 7.146678399294615e-06, + "p50_s": 6.608024705201387e-06, + "p90_s": 7.977581117302179e-06, + "p95_s": 8.148775668814778e-06, + "p99_s": 8.285731310024857e-06, + "min_s": 6.51204027235508e-06, + "max_s": 8.319970220327377e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 3, + "mean_s": 0.03147544667202359, + "p50_s": 0.026172921003308147, + "p90_s": 0.039067967410665004, + "p95_s": 0.040679848211584616, + "p99_s": 0.0419693528523203, + "min_s": 0.025961690000258386, + "max_s": 0.04229172901250422 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 3, + "mean_s": 6.479987253745397e-06, + "p50_s": 5.583977326750755e-06, + "p90_s": 7.811188697814942e-06, + "p95_s": 8.089590119197965e-06, + "p99_s": 8.312311256304384e-06, + "min_s": 5.4879928939044476e-06, + "max_s": 8.367991540580988e-06 + }, + "attn_qkv_proj": { + "count": 3, + "mean_s": 0.03591976532091697, + "p50_s": 0.030206882976926863, + "p90_s": 0.04406598538625986, + "p95_s": 0.04579837318742648, + "p99_s": 0.04718428342835978, + "min_s": 0.030021651997230947, + "max_s": 0.0475307609885931 + }, + "attn_qkv_split_view": { + "count": 3, + "mean_s": 1.6997335478663445e-05, + "p50_s": 1.4527991879731417e-05, + "p90_s": 2.1683203522115948e-05, + "p95_s": 2.257760497741401e-05, + "p99_s": 2.3293126141652467e-05, + "min_s": 1.2992008123546839e-05, + "max_s": 2.3472006432712078e-05 + }, + "attn_qk_rms_rope": { + "count": 3, + "mean_s": 0.012158526010656109, + "p50_s": 0.012062589987181127, + "p90_s": 0.012462397222407162, + "p95_s": 0.012512373126810417, + "p99_s": 0.012552353850333021, + "min_s": 0.011850639013573527, + "max_s": 0.012562349031213671 + }, + "attn_q_transpose_contiguous": { + "count": 3, + "mean_s": 0.007270698668435216, + "p50_s": 0.005053576023783535, + "p90_s": 0.010375015193130822, + "p95_s": 0.01104019508929923, + "p99_s": 0.01157233900623396, + "min_s": 0.0050531449960544705, + "max_s": 0.011705374985467643 + }, + "attn_k_transpose_contiguous": { + "count": 3, + "mean_s": 0.00565119698876515, + "p50_s": 0.005379768030252308, + "p90_s": 0.006368463987018912, + "p95_s": 0.006492050981614738, + "p99_s": 0.006590920577291399, + "min_s": 0.004958184959832579, + "max_s": 0.006615637976210564 + }, + "attn_v_transpose_contiguous": { + "count": 3, + "mean_s": 0.005205602676142007, + "p50_s": 0.005224872031249106, + "p90_s": 0.005333428806625307, + "p95_s": 0.005346998403547332, + "p99_s": 0.005357854081084952, + "min_s": 0.005031367996707559, + "max_s": 0.005360568000469357 + }, + "attention_kernel": { + "count": 3, + "mean_s": 0.2570537893722455, + "p50_s": 0.25454208703013137, + "p90_s": 0.26159199584508314, + "p95_s": 0.26247323444695214, + "p99_s": 0.2631782253284473, + "min_s": 0.25326480803778395, + "max_s": 0.2633544730488211 + }, + "attn_output_reshape": { + "count": 3, + "mean_s": 0.004732787667308003, + "p50_s": 0.00468229700345546, + "p90_s": 0.004807135392911732, + "p95_s": 0.004822740191593766, + "p99_s": 0.004835224030539393, + "min_s": 0.004677721008192748, + "max_s": 0.0048383449902758 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 3, + "mean_s": 4.373335589965184e-06, + "p50_s": 4.224013537168503e-06, + "p90_s": 4.607997834682465e-06, + "p95_s": 4.6559958718717095e-06, + "p99_s": 4.694394301623106e-06, + "min_s": 4.191999323666096e-06, + "max_s": 4.703993909060955e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 3, + "mean_s": 1.0650682573517164e-05, + "p50_s": 1.0048039257526398e-05, + "p90_s": 1.1801626533269882e-05, + "p95_s": 1.2020824942737818e-05, + "p99_s": 1.2196183670312166e-05, + "min_s": 9.66398511081934e-06, + "max_s": 1.2240023352205753e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 3, + "mean_s": 3.53597958261768e-06, + "p50_s": 3.503984771668911e-06, + "p90_s": 3.6448007449507716e-06, + "p95_s": 3.662402741611004e-06, + "p99_s": 3.6764843389391897e-06, + "min_s": 3.4239492379128933e-06, + "max_s": 3.6800047382712364e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 3, + "mean_s": 0.0022405670024454594, + "p50_s": 0.0022424450144171715, + "p90_s": 0.002249599387869239, + "p95_s": 0.0022504936845507474, + "p99_s": 0.002251209121895954, + "min_s": 0.002227868011686951, + "max_s": 0.0022513879812322557 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 3, + "mean_s": 1.0266647829363743e-05, + "p50_s": 1.0079995263367891e-05, + "p90_s": 1.0655971709638835e-05, + "p95_s": 1.0727968765422702e-05, + "p99_s": 1.0785566410049796e-05, + "min_s": 9.91998240351677e-06, + "max_s": 1.079996582120657e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 3, + "mean_s": 0.002841467969119549, + "p50_s": 0.002840427972842008, + "p90_s": 0.002845663169864565, + "p95_s": 0.002846317569492385, + "p99_s": 0.0028468410891946405, + "min_s": 0.002837003965396434, + "max_s": 0.0028469719691202044 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 3, + "mean_s": 6.853311788290739e-06, + "p50_s": 6.783986464142799e-06, + "p90_s": 7.1551650762557985e-06, + "p95_s": 7.201562402769923e-06, + "p99_s": 7.238680263981223e-06, + "min_s": 6.52798917144537e-06, + "max_s": 7.247959729284048e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 3, + "mean_s": 0.007955518667586148, + "p50_s": 0.007876515970565379, + "p90_s": 0.008085207198746502, + "p95_s": 0.008111293602269143, + "p99_s": 0.008132162725087256, + "min_s": 0.007852660026401281, + "max_s": 0.008137380005791783 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 3, + "mean_s": 5.562673322856426e-06, + "p50_s": 5.4879928939044476e-06, + "p90_s": 5.718402098864317e-06, + "p95_s": 5.7472032494843004e-06, + "p99_s": 5.770244169980288e-06, + "min_s": 5.424022674560547e-06, + "max_s": 5.776004400104284e-06 + }, + "attn_out_proj": { + "count": 3, + "mean_s": 0.01312111063937967, + "p50_s": 0.013056731957476586, + "p90_s": 0.013249423156958075, + "p95_s": 0.01327350955689326, + "p99_s": 0.013292778676841407, + "min_s": 0.01300900400383398, + "max_s": 0.013297595956828445 + }, + "gate_msa": { + "count": 3, + "mean_s": 0.008771182018487403, + "p50_s": 0.008706916007213295, + "p90_s": 0.008864918421022595, + "p95_s": 0.008884668722748757, + "p99_s": 0.008900468964129686, + "min_s": 0.008702211023773998, + "max_s": 0.008904419024474919 + }, + "norm2": { + "count": 3, + "mean_s": 0.003586383982716749, + "p50_s": 0.0035896420013159513, + "p90_s": 0.0036038891877979043, + "p95_s": 0.003605670086108148, + "p99_s": 0.0036070948047563435, + "min_s": 0.003562058962415904, + "max_s": 0.003607450984418392 + }, + "modulate_mlp": { + "count": 3, + "mean_s": 0.010889343665136645, + "p50_s": 0.01104473602026701, + "p90_s": 0.011215244000777603, + "p95_s": 0.011236557498341427, + "p99_s": 0.011253608296392485, + "min_s": 0.010365423979237676, + "max_s": 0.01125787099590525 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 3, + "mean_s": 4.560007558514674e-06, + "p50_s": 4.415982402861118e-06, + "p90_s": 4.851189441978932e-06, + "p95_s": 4.905590321868658e-06, + "p99_s": 4.9491110257804395e-06, + "min_s": 4.3040490709245205e-06, + "max_s": 4.959991201758385e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.78933739537994e-06, + "p50_s": 8.608039934188128e-06, + "p90_s": 9.23519255593419e-06, + "p95_s": 9.313586633652448e-06, + "p99_s": 9.376301895827056e-06, + "min_s": 8.367991540580988e-06, + "max_s": 9.391980711370707e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 3, + "mean_s": 3.509359279026588e-06, + "p50_s": 3.503984771668911e-06, + "p90_s": 3.6064302548766136e-06, + "p95_s": 3.6192359402775764e-06, + "p99_s": 3.6294804885983467e-06, + "min_s": 3.392051439732313e-06, + "max_s": 3.6320416256785393e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 3, + "mean_s": 0.0017761682975105941, + "p50_s": 0.0017401569639332592, + "p90_s": 0.0018271209788508712, + "p95_s": 0.0018379914807155728, + "p99_s": 0.001846687882207334, + "min_s": 0.0017394859460182488, + "max_s": 0.0018488619825802743 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 3, + "mean_s": 1.0149359392623106e-05, + "p50_s": 9.776034858077765e-06, + "p90_s": 1.0953645687550305e-05, + "p95_s": 1.1100847041234373e-05, + "p99_s": 1.1218608124181627e-05, + "min_s": 9.423994924873114e-06, + "max_s": 1.1248048394918442e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 3, + "mean_s": 0.0022371909581124783, + "p50_s": 0.002281755965668708, + "p90_s": 0.0022871831548400222, + "p95_s": 0.0022878615534864368, + "p99_s": 0.002288404272403568, + "min_s": 0.002141276956535876, + "max_s": 0.002288539952132851 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 3, + "mean_s": 6.458683249851068e-06, + "p50_s": 6.336020305752754e-06, + "p90_s": 6.720004603266716e-06, + "p95_s": 6.768002640455961e-06, + "p99_s": 6.8064010702073575e-06, + "min_s": 6.224028766155243e-06, + "max_s": 6.8160006776452065e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 3, + "mean_s": 0.03520335033924008, + "p50_s": 0.03511902003083378, + "p90_s": 0.03545432800892741, + "p95_s": 0.0354962415061891, + "p99_s": 0.035529772303998466, + "min_s": 0.03495287598343566, + "max_s": 0.03553815500345081 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 3, + "mean_s": 5.594648731251557e-06, + "p50_s": 5.6800199672579765e-06, + "p90_s": 5.79517800360918e-06, + "p95_s": 5.809572758153081e-06, + "p99_s": 5.821088561788201e-06, + "min_s": 5.279958713799715e-06, + "max_s": 5.8239675126969814e-06 + }, + "mlp_fc1": { + "count": 3, + "mean_s": 0.039302336323695876, + "p50_s": 0.03923152602510527, + "p90_s": 0.039652977988589556, + "p95_s": 0.03970565948402509, + "p99_s": 0.03974780468037352, + "min_s": 0.03891714196652174, + "max_s": 0.03975834097946063 + }, + "mlp_swiglu": { + "count": 3, + "mean_s": 0.02406409801915288, + "p50_s": 0.023905293026473373, + "p90_s": 0.024343602627050133, + "p95_s": 0.024398391327122227, + "p99_s": 0.0244422222871799, + "min_s": 0.023833821003790945, + "max_s": 0.02445318002719432 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 3, + "mean_s": 4.298674563566844e-06, + "p50_s": 4.336005076766014e-06, + "p90_s": 4.591979086399079e-06, + "p95_s": 4.6239758376032116e-06, + "p99_s": 4.649573238566518e-06, + "min_s": 3.9040460251271725e-06, + "max_s": 4.6559725888073444e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 3, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 3, + "mean_s": 8.506661591430506e-06, + "p50_s": 8.463975973427296e-06, + "p90_s": 8.899183012545108e-06, + "p95_s": 8.953583892434835e-06, + "p99_s": 8.997104596346618e-06, + "min_s": 8.048024028539658e-06, + "max_s": 9.007984772324562e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 3, + "mean_s": 3.5466509871184826e-06, + "p50_s": 3.4879776649177074e-06, + "p90_s": 3.641599323600531e-06, + "p95_s": 3.6608020309358834e-06, + "p99_s": 3.6761641968041657e-06, + "min_s": 3.471970558166504e-06, + "max_s": 3.6800047382712364e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 3, + "mean_s": 0.004390478308778256, + "p50_s": 0.004361385013908148, + "p90_s": 0.004447554564103484, + "p95_s": 0.004458325757877901, + "p99_s": 0.004466942712897435, + "min_s": 0.0043409529607743025, + "max_s": 0.004469096951652318 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 3, + "mean_s": 1.036798736701409e-05, + "p50_s": 9.519979357719421e-06, + "p90_s": 1.2169592082500458e-05, + "p95_s": 1.2500793673098087e-05, + "p99_s": 1.2765754945576191e-05, + "min_s": 8.751987479627132e-06, + "max_s": 1.2831995263695717e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 3, + "mean_s": 0.0059309463443545, + "p50_s": 0.00591752800391987, + "p90_s": 0.0060289128334261475, + "p95_s": 0.006042835937114432, + "p99_s": 0.00605397442006506, + "min_s": 0.005818551988340914, + "max_s": 0.006056759040802717 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 3, + "mean_s": 6.405326227347056e-06, + "p50_s": 6.52798917144537e-06, + "p90_s": 6.566406227648258e-06, + "p95_s": 6.571208359673619e-06, + "p99_s": 6.575050065293908e-06, + "min_s": 6.111979018896818e-06, + "max_s": 6.57601049169898e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 3, + "mean_s": 0.05613339397435387, + "p50_s": 0.056144060974474996, + "p90_s": 0.05626750335795805, + "p95_s": 0.05628293365589343, + "p99_s": 0.056295277894241734, + "min_s": 0.0559577569947578, + "max_s": 0.05629836395382881 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 3, + "mean_s": 5.423983869453271e-06, + "p50_s": 5.423964466899633e-06, + "p90_s": 5.462381523102522e-06, + "p95_s": 5.467183655127883e-06, + "p99_s": 5.471025360748172e-06, + "min_s": 5.376001354306936e-06, + "max_s": 5.471985787153244e-06 + }, + "mlp_fc2": { + "count": 3, + "mean_s": 0.06653381067250545, + "p50_s": 0.06663658900652081, + "p90_s": 0.06664222099352628, + "p95_s": 0.06664292499190197, + "p99_s": 0.06664348819060252, + "min_s": 0.06632121402071789, + "max_s": 0.06664362899027765 + }, + "gate_mlp": { + "count": 3, + "mean_s": 0.014599877002183348, + "p50_s": 0.008974290976766497, + "p90_s": 0.022819262195844205, + "p95_s": 0.024549883598228916, + "p99_s": 0.025934380720136686, + "min_s": 0.008544835029169917, + "max_s": 0.02628050500061363 + }, + "block_total": { + "count": 3, + "mean_s": 0.5241046730079688, + "p50_s": 0.5232612389954738, + "p90_s": 0.5255022222059779, + "p95_s": 0.5257823451072908, + "p99_s": 0.5260064434283414, + "min_s": 0.5229903120198287, + "max_s": 0.5260624680086039 + } + }, + "module_forward": { + "count": 3, + "mean_s": 0.4765262039921557, + "p50_s": 0.4681359010282904, + "p90_s": 0.4901721953880042, + "p95_s": 0.4929267321829684, + "p99_s": 0.4951303616189398, + "min_s": 0.46576144197024405, + "max_s": 0.49568126897793263 + }, + "module_forward_checksum": -778077863936.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 7351755264 + }, + "profiler_top_events": [ + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 500.23900000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 416.1590000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 323.05600000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 217.82299999999896, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 201.44000000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 13, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 188.896000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 186.01600000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 175.42400000000006, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 147.2470000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 133.7439999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 129.29600000000028, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::empty", + "count": 29, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 120.14400000000344, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 6267507200, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 112.3680000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 109.55199999999968, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 107.92000000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 86.096, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 77.87199999999984, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 70.12799999999925, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 65.21600000000035, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 19, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 63.07199999999966, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json b/benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json new file mode 100644 index 0000000..31fd672 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json @@ -0,0 +1,38 @@ +{ + "device": "NVIDIA GB10", + "workload": { + "width": 1344, + "height": 768, + "frames": 124, + "tokens": 37810, + "seed": 440420 + }, + "method": "Alternating baseline and ring execution inside one loaded block", + "capacity_2048": { + "block_0": { + "equal": true, + "baseline_p50_ms": 461.54195550479926, + "ring_p50_ms": 465.29679899686016, + "improvement_percent": -0.8135432645455021 + }, + "block_24": { + "equal": true, + "baseline_p50_ms": 467.8640030033421, + "ring_p50_ms": 470.3063364722766, + "improvement_percent": -0.5220178199768499 + }, + "block_49": { + "equal": true, + "baseline_p50_ms": 462.3519679880701, + "ring_p50_ms": 464.76577199064195, + "improvement_percent": -0.5220706668721542 + } + }, + "block_24_capacity_sweep": { + "3072": -1.0036016035203765, + "4096": -0.2438296625217884, + "8192": -4.798436757705438, + "37888": -14.49047057045394 + }, + "decision": "Reject runtime dispatch. Keep opt-in and disabled until launch fusion or a different persistent scheduler passes the alternating block gate. Skip trajectory validation." +} diff --git a/benchmarks/gb10-cute-qkv-runtime-block24-2048.json b/benchmarks/gb10-cute-qkv-runtime-block24-2048.json new file mode 100644 index 0000000..2bc5f89 --- /dev/null +++ b/benchmarks/gb10-cute-qkv-runtime-block24-2048.json @@ -0,0 +1,42 @@ +{ + "device": "NVIDIA GB10", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "rows": 2048, + "role": "h3_attn_qkv", + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "ring_ms": 1.6192415237426758, + "reference_ms": 1.859343910217285, + "improvement_percent": 12.913285441989618 +} diff --git a/benchmarks/gb10-cute-qkv-runtime-block24-full.json b/benchmarks/gb10-cute-qkv-runtime-block24-full.json new file mode 100644 index 0000000..4f1b72d --- /dev/null +++ b/benchmarks/gb10-cute-qkv-runtime-block24-full.json @@ -0,0 +1,42 @@ +{ + "device": "NVIDIA GB10", + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "block_index": 24, + "rows": 37810, + "role": "h3_attn_qkv", + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "ring_ms": 31.741008758544922, + "reference_ms": 46.7579345703125, + "improvement_percent": 32.11631555107678 +} diff --git a/benchmarks/gb10-deployed-modulate-fusion-block24-profile.json b/benchmarks/gb10-deployed-modulate-fusion-block24-profile.json new file mode 100644 index 0000000..b62a449 --- /dev/null +++ b/benchmarks/gb10-deployed-modulate-fusion-block24-profile.json @@ -0,0 +1,1392 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 10, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 10, + "mean_s": 0.003516814694739878, + "p50_s": 0.0035167409805580974, + "p90_s": 0.0035479393962305037, + "p95_s": 0.0035552402172470466, + "p99_s": 0.0035610808740602806, + "min_s": 0.0034472449915483594, + "max_s": 0.003562541038263589 + }, + "modulate_msa": { + "count": 10, + "mean_s": 0.010366741399047896, + "p50_s": 0.01035376702202484, + "p90_s": 0.010706545307766647, + "p95_s": 0.01078284414834343, + "p99_s": 0.010843883220804856, + "min_s": 0.009973047010134906, + "max_s": 0.010859142988920212 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.526402335613966e-06, + "p50_s": 4.10401844419539e-06, + "p90_s": 4.755170084536074e-06, + "p95_s": 6.42557861283421e-06, + "p99_s": 7.761905435472728e-06, + "min_s": 3.903987817466259e-06, + "max_s": 8.095987141132355e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 1.0039989138022066e-05, + "p50_s": 8.119968697428703e-06, + "p90_s": 1.0654376819729798e-05, + "p95_s": 1.935919281095264e-05, + "p99_s": 2.6323045603930952e-05, + "min_s": 7.39202369004488e-06, + "max_s": 2.8064008802175522e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 10, + "mean_s": 3.5263889003545047e-06, + "p50_s": 3.42397834174335e-06, + "p90_s": 3.747199662029743e-06, + "p95_s": 4.049576818943023e-06, + "p99_s": 4.2914785444736485e-06, + "min_s": 3.2800016924738884e-06, + "max_s": 4.351953975856304e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 10, + "mean_s": 0.0016631810984108597, + "p50_s": 0.0016464549989905208, + "p90_s": 0.0016899974027182907, + "p95_s": 0.0017555822065332904, + "p99_s": 0.0018080500495852904, + "min_s": 0.0016271350323222578, + "max_s": 0.0018211670103482902 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 10, + "mean_s": 1.1579185957089066e-05, + "p50_s": 9.391980711370707e-06, + "p90_s": 1.7219223082065582e-05, + "p95_s": 1.8961611203849312e-05, + "p99_s": 2.0355521701276303e-05, + "min_s": 9.023991879075766e-06, + "max_s": 2.070399932563305e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.0022007547901012003, + "p50_s": 0.002188237995142117, + "p90_s": 0.002310799522092566, + "p95_s": 0.0023156302486313507, + "p99_s": 0.0023194948298623785, + "min_s": 0.002091966976877302, + "max_s": 0.0023204609751701355 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.332807242870331e-06, + "p50_s": 6.1680038925260305e-06, + "p90_s": 6.833561928942799e-06, + "p95_s": 7.344779442064463e-06, + "p99_s": 7.753753452561796e-06, + "min_s": 5.824025720357895e-06, + "max_s": 7.855996955186129e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 10, + "mean_s": 0.02569694590056315, + "p50_s": 0.025759841984836385, + "p90_s": 0.0262955051031895, + "p95_s": 0.026411317544989286, + "p99_s": 0.02650396749842912, + "min_s": 0.024846857995726168, + "max_s": 0.026527129986789078 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 10, + "mean_s": 5.633599357679486e-06, + "p50_s": 5.3679978009313345e-06, + "p90_s": 5.988759221509098e-06, + "p95_s": 6.8743858719244576e-06, + "p99_s": 7.582887192256748e-06, + "min_s": 5.231995601207018e-06, + "max_s": 7.760012522339821e-06 + }, + "attn_qkv_proj": { + "count": 10, + "mean_s": 0.029643844993552194, + "p50_s": 0.029725486994720995, + "p90_s": 0.030352766008581965, + "p95_s": 0.030365185998380183, + "p99_s": 0.03037512199021876, + "min_s": 0.0287816229974851, + "max_s": 0.030377605988178402 + }, + "attn_qkv_split_view": { + "count": 10, + "mean_s": 1.3008003588765859e-05, + "p50_s": 1.2336007785052061e-05, + "p90_s": 1.554721384309232e-05, + "p95_s": 1.7541609122417863e-05, + "p99_s": 1.9137125345878303e-05, + "min_s": 9.984010830521584e-06, + "max_s": 1.9536004401743412e-05 + }, + "attn_qk_rms_rope": { + "count": 10, + "mean_s": 0.012352611508686096, + "p50_s": 0.01237390103051439, + "p90_s": 0.012589196878252551, + "p95_s": 0.012807608451112173, + "p99_s": 0.01298233770939987, + "min_s": 0.011864198022522032, + "max_s": 0.013026020023971796 + }, + "attention_kernel": { + "count": 10, + "mean_s": 0.27108906718785875, + "p50_s": 0.26181144698057324, + "p90_s": 0.29291597349219956, + "p95_s": 0.312738003252889, + "p99_s": 0.32859562706144063, + "min_s": 0.25998799898661673, + "max_s": 0.3325600330135785 + }, + "attn_output_reshape": { + "count": 10, + "mean_s": 4.281604196876287e-06, + "p50_s": 4.128029104322195e-06, + "p90_s": 4.7808163799345495e-06, + "p95_s": 4.9104215577244755e-06, + "p99_s": 5.0141056999564175e-06, + "min_s": 3.936002030968666e-06, + "max_s": 5.040026735514402e-06 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.358391743153334e-06, + "p50_s": 4.128000000491738e-06, + "p90_s": 5.2239862270653244e-06, + "p95_s": 5.2599760238081215e-06, + "p99_s": 5.288767861202359e-06, + "min_s": 3.936002030968666e-06, + "max_s": 5.295965820550919e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 9.428808698430658e-06, + "p50_s": 8.216011337935925e-06, + "p90_s": 1.2791971676051616e-05, + "p95_s": 1.2899993453174828e-05, + "p99_s": 1.29864108748734e-05, + "min_s": 7.936032488942146e-06, + "max_s": 1.3008015230298042e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 10, + "mean_s": 3.443198511376977e-06, + "p50_s": 3.4320109989494085e-06, + "p90_s": 3.6207784432917833e-06, + "p95_s": 3.642388037405908e-06, + "p99_s": 3.659675712697208e-06, + "min_s": 3.296008799225092e-06, + "max_s": 3.663997631520033e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 10, + "mean_s": 0.002261684410041198, + "p50_s": 0.0022500380291603506, + "p90_s": 0.0022729580232407898, + "p95_s": 0.0023227820202009752, + "p99_s": 0.0023626412177691234, + "min_s": 0.002238414017483592, + "max_s": 0.0023726060171611607 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 10, + "mean_s": 1.038879854604602e-05, + "p50_s": 9.832001524046063e-06, + "p90_s": 1.1913618072867393e-05, + "p95_s": 1.224480802193284e-05, + "p99_s": 1.2509759981185198e-05, + "min_s": 9.440002031624317e-06, + "max_s": 1.2575997970998287e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.0029140072059817613, + "p50_s": 0.0028348854975774884, + "p90_s": 0.002980089106131345, + "p95_s": 0.0032665550534147765, + "p99_s": 0.0034957278112415224, + "min_s": 0.002825118019245565, + "max_s": 0.003553021000698209 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.366416346281767e-06, + "p50_s": 6.287998985499144e-06, + "p90_s": 6.824050797149538e-06, + "p95_s": 6.93204638082534e-06, + "p99_s": 7.018442847765983e-06, + "min_s": 5.904003046452999e-06, + "max_s": 7.0400419645011425e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 10, + "mean_s": 0.008031879493501037, + "p50_s": 0.007886153471190482, + "p90_s": 0.00864636250771582, + "p95_s": 0.008688777242787181, + "p99_s": 0.008722709030844272, + "min_s": 0.007817146019078791, + "max_s": 0.008731191977858543 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 10, + "mean_s": 5.478394450619817e-06, + "p50_s": 5.496025551110506e-06, + "p90_s": 5.662406329065561e-06, + "p95_s": 5.727208917960524e-06, + "p99_s": 5.779050989076496e-06, + "min_s": 5.183974280953407e-06, + "max_s": 5.792011506855488e-06 + }, + "attn_out_proj": { + "count": 10, + "mean_s": 0.013287241506623105, + "p50_s": 0.01306043702061288, + "p90_s": 0.013964672799920664, + "p95_s": 0.014308530397829599, + "p99_s": 0.014583616476156748, + "min_s": 0.012968517025001347, + "max_s": 0.014652387995738536 + }, + "gate_msa": { + "count": 10, + "mean_s": 0.008519085415173322, + "p50_s": 0.0083449850208126, + "p90_s": 0.008864006400108338, + "p95_s": 0.009121759195113554, + "p99_s": 0.009327961431117728, + "min_s": 0.008315177052281797, + "max_s": 0.009379511990118772 + }, + "norm2": { + "count": 10, + "mean_s": 0.003516377799678594, + "p50_s": 0.003482725005596876, + "p90_s": 0.003730286500649527, + "p95_s": 0.0037477892503375186, + "p99_s": 0.0037617914500879127, + "min_s": 0.003390222031157464, + "max_s": 0.003765292000025511 + }, + "modulate_mlp": { + "count": 10, + "mean_s": 0.010381480591604486, + "p50_s": 0.010417310986667871, + "p90_s": 0.010713717294856906, + "p95_s": 0.011112517665605991, + "p99_s": 0.011431557962205262, + "min_s": 0.009898118965793401, + "max_s": 0.011511318036355078 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 10, + "mean_s": 4.295998951420188e-06, + "p50_s": 4.0879822336137295e-06, + "p90_s": 4.825583891943097e-06, + "p95_s": 4.9408088671043505e-06, + "p99_s": 5.032988847233355e-06, + "min_s": 4.00003045797348e-06, + "max_s": 5.056033842265606e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 10, + "mean_s": 8.068798342719674e-06, + "p50_s": 7.96798849478364e-06, + "p90_s": 8.828815771266818e-06, + "p95_s": 8.88641516212374e-06, + "p99_s": 8.932494674809277e-06, + "min_s": 7.375958375632763e-06, + "max_s": 8.944014552980661e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 10, + "mean_s": 3.4864002373069526e-06, + "p50_s": 3.4399854484945536e-06, + "p90_s": 3.6512385122478006e-06, + "p95_s": 3.7376245018094775e-06, + "p99_s": 3.8067332934588194e-06, + "min_s": 3.328023012727499e-06, + "max_s": 3.824010491371155e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 10, + "mean_s": 0.001673643506364897, + "p50_s": 0.001655454485444352, + "p90_s": 0.0017760470509529114, + "p95_s": 0.0017949110479094088, + "p99_s": 0.0018100022454746066, + "min_s": 0.001621982955839485, + "max_s": 0.0018137750448659062 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 10, + "mean_s": 1.0702403960749507e-05, + "p50_s": 9.176001185551286e-06, + "p90_s": 1.4611223014071582e-05, + "p95_s": 1.7001610831357533e-05, + "p99_s": 1.8913921085186306e-05, + "min_s": 8.89599323272705e-06, + "max_s": 1.9391998648643494e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 10, + "mean_s": 0.002208998193964362, + "p50_s": 0.002138246491085738, + "p90_s": 0.0023599260370247067, + "p95_s": 0.0025579620152711862, + "p99_s": 0.002716390797868371, + "min_s": 0.0020945259602740407, + "max_s": 0.002755997993517667 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 10, + "mean_s": 6.103987107053399e-06, + "p50_s": 6.063986802473664e-06, + "p90_s": 6.3119980040937655e-06, + "p95_s": 6.419993587769568e-06, + "p99_s": 6.506390054710209e-06, + "min_s": 5.776004400104284e-06, + "max_s": 6.52798917144537e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 10, + "mean_s": 0.0395361577917356, + "p50_s": 0.03482407398405485, + "p90_s": 0.04009287208900785, + "p95_s": 0.061921360049745945, + "p99_s": 0.0793841504183365, + "min_s": 0.033949602977372706, + "max_s": 0.08374984801048413 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 10, + "mean_s": 5.504011642187834e-06, + "p50_s": 5.54401776753366e-06, + "p90_s": 5.731184501200914e-06, + "p95_s": 5.7455908972769976e-06, + "p99_s": 5.757116014137864e-06, + "min_s": 5.13601116836071e-06, + "max_s": 5.759997293353081e-06 + }, + "mlp_fc1": { + "count": 10, + "mean_s": 0.04350133700063452, + "p50_s": 0.03864074300508946, + "p90_s": 0.044060093007283266, + "p95_s": 0.06623608849185979, + "p99_s": 0.08397688487952111, + "min_s": 0.0377592800068669, + "max_s": 0.0884120839764364 + }, + "mlp_swiglu": { + "count": 10, + "mean_s": 0.02421614230261184, + "p50_s": 0.024195923004299402, + "p90_s": 0.024524036498041822, + "p95_s": 0.024536607231129895, + "p99_s": 0.02454666381760035, + "min_s": 0.02381745196180418, + "max_s": 0.02454917796421796 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 10, + "mean_s": 4.011200508102774e-06, + "p50_s": 3.951980033889413e-06, + "p90_s": 4.32159285992384e-06, + "p95_s": 4.400801844894886e-06, + "p99_s": 4.464169032871723e-06, + "min_s": 3.7919962778687477e-06, + "max_s": 4.4800108298659325e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 10, + "mean_s": 7.80478585511446e-06, + "p50_s": 7.807975634932518e-06, + "p90_s": 8.39359126985073e-06, + "p95_s": 8.508790051564574e-06, + "p99_s": 8.60094907693565e-06, + "min_s": 7.039983756840229e-06, + "max_s": 8.623988833278418e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 10, + "mean_s": 3.423896851018071e-06, + "p50_s": 3.4479890018701553e-06, + "p90_s": 3.5391014534980056e-06, + "p95_s": 3.553062560968101e-06, + "p99_s": 3.564231446944177e-06, + "min_s": 3.2160314731299877e-06, + "max_s": 3.567023668438196e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 10, + "mean_s": 0.004466716205934062, + "p50_s": 0.00445973250316456, + "p90_s": 0.004518093593651428, + "p95_s": 0.0045484847767511384, + "p99_s": 0.004572797723230906, + "min_s": 0.004411837027873844, + "max_s": 0.004578875959850848 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 10, + "mean_s": 9.115197462961077e-06, + "p50_s": 9.271985618397593e-06, + "p90_s": 9.45759820751846e-06, + "p95_s": 9.60881297942251e-06, + "p99_s": 9.729784796945752e-06, + "min_s": 8.351984433829784e-06, + "max_s": 9.760027751326561e-06 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 10, + "mean_s": 0.00589796609710902, + "p50_s": 0.005918154987739399, + "p90_s": 0.0060729933262337, + "p95_s": 0.006076693648356012, + "p99_s": 0.006079653906053864, + "min_s": 0.005693643004633486, + "max_s": 0.006080393970478326 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 10, + "mean_s": 6.0623977333307265e-06, + "p50_s": 6.007991032674909e-06, + "p90_s": 6.356823723763227e-06, + "p95_s": 6.378407124429941e-06, + "p99_s": 6.395673844963312e-06, + "min_s": 5.792011506855488e-06, + "max_s": 6.399990525096655e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 10, + "mean_s": 0.05543897160096094, + "p50_s": 0.055413160007447004, + "p90_s": 0.05576852477970533, + "p95_s": 0.05588342241535429, + "p99_s": 0.05597534052387346, + "min_s": 0.05484625697135925, + "max_s": 0.05599832005100325 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 10, + "mean_s": 5.4095988161861895e-06, + "p50_s": 5.376001354306936e-06, + "p90_s": 5.587184568867088e-06, + "p95_s": 5.601617158390582e-06, + "p99_s": 5.613163230009377e-06, + "min_s": 5.279958713799715e-06, + "max_s": 5.616049747914076e-06 + }, + "mlp_fc2": { + "count": 10, + "mean_s": 0.06587910819798708, + "p50_s": 0.06583923898870125, + "p90_s": 0.06626416540239007, + "p95_s": 0.06628907018457539, + "p99_s": 0.06630899401032367, + "min_s": 0.0654114960343577, + "max_s": 0.06631397496676072 + }, + "gate_mlp": { + "count": 10, + "mean_s": 0.009955293708480894, + "p50_s": 0.008543304516933858, + "p90_s": 0.010343933285912495, + "p95_s": 0.016277333159814576, + "p99_s": 0.021024053058936263, + "min_s": 0.008402952051255852, + "max_s": 0.02221073303371668 + }, + "block_total": { + "count": 10, + "mean_s": 0.5063105094013736, + "p50_s": 0.49055024900007993, + "p90_s": 0.5626000013318845, + "p95_s": 0.5668660971510691, + "p99_s": 0.5702789738064167, + "min_s": 0.4873858370119706, + "max_s": 0.5711321929702535 + } + }, + "module_forward": { + "count": 10, + "mean_s": 0.468727556290105, + "p50_s": 0.46669237900641747, + "p90_s": 0.47172567221568895, + "p95_s": 0.4803373321017716, + "p99_s": 0.4872266600106377, + "min_s": 0.4628840069635771, + "max_s": 0.48894899198785424 + }, + "module_forward_checksum": -23765568.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 9093805568 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1399.279, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1339.6789999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1062.143, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 790.6550000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "aten::contiguous", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 676.1750000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "aten::clone", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 665.4230000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 565.4390000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 515.6469999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 240.76800000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::empty", + "count": 42, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 161.05600000000118, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009559552, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 144.70400000000018, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [3, 5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 126.6869999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 106.64000000000033, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 99.10400000000027, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 95.64800000000014, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 84.79999999999927, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 84.40000000000055, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 82.54399999999987, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 76.83200000000033, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 53.56800000000112, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.42399999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 45.039999999999964, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 44.86399999999958, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::to", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 38.496000000001914, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::empty_like", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 32.57599999999957, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [], [], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.98400000000038, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.167999999999665, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::split", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 29.360000000000582, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], []]" + }, + { + "key": "aten::silu", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24.271999999999935, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 1084227584, + "input_shapes": "[[37810, 14336]]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 22.640000000000327, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::narrow", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.144000000000233, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::mul_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.751999999999498, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [37810, 14336]]" + }, + { + "key": "aten::reshape", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.904000000000451, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.407999999999447, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.264000000001033, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::chunk", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.527999999999338, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.88799999999992, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.368000000001302, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.423999999999978, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::split", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.216000000000349, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.200000000000273, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810], []]" + }, + { + "key": "aten::slice", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.024000000000342, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.22400000000016, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.631999999999721, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376], [], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.895999999999731, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::narrow", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.895999999998821, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::squeeze", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.9840000000003783, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.6640000000006694, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4719999999997526, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.3599999999996726, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::view", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.8159999999998035, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.639999999999418, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.6239999999997963, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.4160000000001673, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.1920000000000073, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.8239999999996144, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7760000000007494, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.5039999999999054, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.4720000000006621, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2960000000002765, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.9600000000000364, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8800000000001091, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128]]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8159999999998035, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.815999999998894, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7520000000004075, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7200000000002547, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7039999999997235, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6239999999997963, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.47999999999956344, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.4639999999999418, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.44799999999941065, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.41600000000016735, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.3999999999996362, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.35199999999986176, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.3040000000000873, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.25600000000031287, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "[memory]", + "count": 37, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -9093500416, + "input_shapes": "[]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-four-gemm-nvfp4-roofline.json b/benchmarks/gb10-four-gemm-nvfp4-roofline.json new file mode 100644 index 0000000..2d7dd57 --- /dev/null +++ b/benchmarks/gb10-four-gemm-nvfp4-roofline.json @@ -0,0 +1,95 @@ +{ + "status": "complete", + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "tokens": 37810, + "block": 24, + "seed": 440420 + }, + "method": { + "flops": "2*M*N*K dense-equivalent operations", + "l2_traffic": "Nsight Compute lts__t_bytes.sum with cache control disabled", + "off_chip_proxy": "32 bytes times warm-cache L2 read plus write lookup-miss sectors", + "practical_compute_ceiling_tflops_per_s": 321.455895, + "theoretical_memory_bandwidth_gb_per_s": 273.0, + "practical_crossover_flops_per_byte": 1177.49412, + "note": "GB10 exposes no direct physical DRAM-byte counter; classification uses the documented bandwidth and the warm-cache off-chip request proxy." + }, + "gemms": [ + { + "role": "attention_qkv", + "shape_mnk": [37810, 21504, 5376], + "kernel": "standard", + "duration_ms": 27.195296, + "dense_equivalent_tflops": 8.742088, + "achieved_tflops_per_s": 321.455895, + "l2_requested_gb": 40.129071, + "off_chip_proxy_gb": 4.368326, + "off_chip_arithmetic_intensity": 2001.244307, + "ideal_unique_bytes_gb": 1.805498, + "off_chip_amplification": 2.419458, + "classification": "compute-bound" + }, + { + "role": "attention_output", + "shape_mnk": [37810, 5376, 7168], + "kernel": "standard", + "duration_ms": 9.25072, + "dense_equivalent_tflops": 2.914029, + "achieved_tflops_per_s": 315.005686, + "l2_requested_gb": 13.24078, + "off_chip_proxy_gb": 1.337147, + "off_chip_arithmetic_intensity": 2179.28899, + "ideal_unique_bytes_gb": 0.580659, + "off_chip_amplification": 2.302809, + "classification": "compute-bound" + }, + { + "role": "mlp_fc1", + "shape_mnk": [37810, 28672, 5376], + "kernel": "standard", + "duration_ms": 37.024512, + "dense_equivalent_tflops": 11.656118, + "achieved_tflops_per_s": 314.821641, + "l2_requested_gb": 53.507764, + "off_chip_proxy_gb": 5.823759, + "off_chip_arithmetic_intensity": 2001.476751, + "ideal_unique_bytes_gb": 2.369218, + "off_chip_amplification": 2.458093, + "classification": "compute-bound" + }, + { + "role": "mlp_fc2", + "shape_mnk": [37810, 5376, 14336], + "kernel": "stream-k", + "duration_ms": 54.973312, + "dense_equivalent_tflops": 5.828059, + "achieved_tflops_per_s": 106.016148, + "l2_requested_gb": 26.07999, + "off_chip_proxy_gb": 12.161131, + "off_chip_arithmetic_intensity": 479.236573, + "ideal_unique_bytes_gb": 0.754785, + "off_chip_amplification": 16.112046, + "classification": "bandwidth-bound", + "roofline_at_273_gb_per_s_ms": 44.546268, + "measured_bandwidth_efficiency_percent": 81.032534, + "traffic_required_to_reach_practical_compute_ridge_gb": 4.949544, + "required_traffic_reduction_percent": 59.299, + "practical_compute_floor_ms": 18.130197 + } + ], + "decision": { + "primary_target": "mlp_fc2", + "do_not_target_first": ["attention_qkv", "attention_output", "mlp_fc1"], + "reason": "QKV, attention output, and FC1 cluster at 315-321 dense-equivalent TFLOP/s and lie above the practical ridge point. FC2 uses stream-K, reaches only 106 TFLOP/s, and generates 12.16 GB of off-chip request traffic, 16.1x its ideal unique bytes.", + "next_experiment": "Benchmark an exact FC2 schedule that reduces K-split/stream-K rereads and reuses the fused SwiGLU-produced activation tiles. Require bit-exact BF16 output and at least 3% complete-block improvement before integration." + }, + "source_artifacts": [ + "benchmarks/gb10-fully-fused-block24-fresh-l2.csv", + "benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv", + "benchmarks/gb10-fully-fused-fresh-nsight-summary.json" + ] +} diff --git a/benchmarks/gb10-fully-fused-block24-fresh-capture.json b/benchmarks/gb10-fully-fused-block24-fresh-capture.json new file mode 100644 index 0000000..fb4db12 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-capture.json @@ -0,0 +1,31 @@ +{ + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "fused_elementwise": true, + "fused_nvfp4_modulation": true, + "fused_nvfp4_swiglu": true, + "nvfp4_scale_backend": "vortex", + "sage_qkv_layout": "strided_nhd", + "module_forward_checksum": 273326784.0, + "capture": "one warmed block between cudaProfilerStart/Stop" +} \ No newline at end of file diff --git a/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv b/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv new file mode 100644 index 0000000..c1dfc65 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv @@ -0,0 +1,47 @@ +==WARNING== Note: Running with uncontrolled GPU caches. Profiling results may be inconsistent. +==PROF== Connected to process 61 (/usr/bin/python3.12) +==PROF== Disconnected from process 61 +==PROF== Report: /output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.ncu-rep +"ID","Process ID","Process Name","Host Name","Kernel Name","Context","Stream","Block Size","Grid Size","Device","CC","c2clink__enabled_mask","c2clink__present","device__attribute_architecture","device__attribute_async_engine_count","device__attribute_can_flush_remote_writes","device__attribute_can_map_host_memory","device__attribute_can_tex2d_gather","device__attribute_can_use_64_bit_stream_mem_ops","device__attribute_can_use_64_bit_stream_mem_ops_v1","device__attribute_can_use_host_pointer_for_registered_mem","device__attribute_can_use_stream_mem_ops_v1","device__attribute_can_use_stream_wait_value_nor","device__attribute_can_use_stream_wait_value_nor_v1","device__attribute_chip","device__attribute_clock_rate","device__attribute_cluster_launch","device__attribute_compute_capability_major","device__attribute_compute_capability_minor","device__attribute_compute_mode","device__attribute_compute_preemption_supported","device__attribute_concurrent_kernels","device__attribute_concurrent_managed_access","device__attribute_confidential_computing_mode","device__attribute_cooperative_launch","device__attribute_cooperative_multi_device_launch","device__attribute_deferred_mapping_cuda_array_supported","device__attribute_device_index","device__attribute_direct_managed_mem_access_from_host","device__attribute_display_name","device__attribute_dma_buf_supported","device__attribute_ecc_enabled","device__attribute_fb_bus_width","device__attribute_fbp_count","device__attribute_generic_compression_supported","device__attribute_global_l1_cache_supported","device__attribute_global_memory_bus_width","device__attribute_gpu_direct_rdma_flush_writes_options","device__attribute_gpu_direct_rdma_supported","device__attribute_gpu_direct_rdma_with_cuda_vmm_supported","device__attribute_gpu_direct_rdma_writes_ordering","device__attribute_gpu_overlap","device__attribute_gpu_pci_device_id","device__attribute_gpu_pci_ext_device_id","device__attribute_gpu_pci_ext_downstream_link_rate","device__attribute_gpu_pci_ext_downstream_link_width","device__attribute_gpu_pci_ext_gen","device__attribute_gpu_pci_ext_gpu_gen","device__attribute_gpu_pci_ext_gpu_link_rate","device__attribute_gpu_pci_ext_gpu_link_width","device__attribute_gpu_pci_revision_id","device__attribute_gpu_pci_sub_system_id","device__attribute_handle_type_fabric_supported","device__attribute_handle_type_posix_file_descriptor_supported","device__attribute_handle_type_win32_handle_supported","device__attribute_handle_type_win32_kmt_handle_supported","device__attribute_host_native_atomic_supported","device__attribute_host_numa_id","device__attribute_host_register_supported","device__attribute_implementation","device__attribute_integrated","device__attribute_ipc_event_supported","device__attribute_kernel_exec_timeout","device__attribute_l2_cache_size","device__attribute_l2s_count","device__attribute_limits_max_cta_per_sm","device__attribute_limits_num_tpcs","device__attribute_local_l1_cache_supported","device__attribute_managed_memory","device__attribute_max_access_policy_window_size","device__attribute_max_block_dim_x","device__attribute_max_block_dim_y","device__attribute_max_block_dim_z","device__attribute_max_blocks_per_multiprocessor","device__attribute_max_gpu_frequency_khz","device__attribute_max_grid_dim_x","device__attribute_max_grid_dim_y","device__attribute_max_grid_dim_z","device__attribute_max_ipc_per_multiprocessor","device__attribute_max_ipc_per_scheduler","device__attribute_max_mem_frequency_khz","device__attribute_max_persisting_l2_cache_size","device__attribute_max_pitch","device__attribute_max_registers_per_block","device__attribute_max_registers_per_multiprocessor","device__attribute_max_registers_per_thread","device__attribute_max_shared_memory_per_block","device__attribute_max_shared_memory_per_block_optin","device__attribute_max_shared_memory_per_multiprocessor","device__attribute_max_threads_per_block","device__attribute_max_threads_per_multiprocessor","device__attribute_max_warps_per_multiprocessor","device__attribute_max_warps_per_scheduler","device__attribute_maximum_surface1d_layered_layers","device__attribute_maximum_surface1d_layered_width","device__attribute_maximum_surface1d_width","device__attribute_maximum_surface2d_height","device__attribute_maximum_surface2d_layered_height","device__attribute_maximum_surface2d_layered_layers","device__attribute_maximum_surface2d_layered_width","device__attribute_maximum_surface2d_width","device__attribute_maximum_surface3d_depth","device__attribute_maximum_surface3d_height","device__attribute_maximum_surface3d_width","device__attribute_maximum_surfacecubemap_layered_layers","device__attribute_maximum_surfacecubemap_layered_width","device__attribute_maximum_surfacecubemap_width","device__attribute_maximum_texture1d_layered_layers","device__attribute_maximum_texture1d_layered_width","device__attribute_maximum_texture1d_linear_width","device__attribute_maximum_texture1d_mipmapped_width","device__attribute_maximum_texture1d_width","device__attribute_maximum_texture2d_gather_height","device__attribute_maximum_texture2d_gather_width","device__attribute_maximum_texture2d_height","device__attribute_maximum_texture2d_layered_height","device__attribute_maximum_texture2d_layered_layers","device__attribute_maximum_texture2d_layered_width","device__attribute_maximum_texture2d_linear_height","device__attribute_maximum_texture2d_linear_pitch","device__attribute_maximum_texture2d_linear_width","device__attribute_maximum_texture2d_mipmapped_height","device__attribute_maximum_texture2d_mipmapped_width","device__attribute_maximum_texture2d_width","device__attribute_maximum_texture3d_depth","device__attribute_maximum_texture3d_depth_alternate","device__attribute_maximum_texture3d_height","device__attribute_maximum_texture3d_height_alternate","device__attribute_maximum_texture3d_width","device__attribute_maximum_texture3d_width_alternate","device__attribute_maximum_texturecubemap_layered_layers","device__attribute_maximum_texturecubemap_layered_width","device__attribute_maximum_texturecubemap_width","device__attribute_mem_sync_domain_count","device__attribute_memory_clock_rate","device__attribute_memory_pools_supported","device__attribute_mempool_supported_handle_types","device__attribute_mps_enabled","device__attribute_multi_gpu_board","device__attribute_multi_gpu_board_group_id","device__attribute_multicast_supported","device__attribute_multiprocessor_count","device__attribute_num_l2s_per_fbp","device__attribute_num_schedulers_per_multiprocessor","device__attribute_num_tex_per_multiprocessor","device__attribute_numa_config","device__attribute_pageable_memory_access","device__attribute_pageable_memory_access_uses_host_page_tables","device__attribute_pci_bus_id","device__attribute_pci_device_id","device__attribute_pci_domain_id","device__attribute_ram_location","device__attribute_ram_type","device__attribute_reserved_shared_memory_per_block","device__attribute_sass_level","device__attribute_single_to_double_precision_perf_ratio","device__attribute_sparse_cuda_array_supported","device__attribute_stream_priorities_supported","device__attribute_surface_alignment","device__attribute_tcc_driver","device__attribute_tensor_map_access_supported","device__attribute_texture_alignment","device__attribute_texture_pitch_alignment","device__attribute_total_constant_memory","device__attribute_total_memory","device__attribute_unified_addressing","device__attribute_unified_function_pointers","device__attribute_virtual_address_management_supported","device__attribute_warp_size","gpu__time_duration.sum","launch__barrier_count","launch__block_dim_x","launch__block_dim_y","launch__block_dim_z","launch__block_size","launch__cluster_dim_x","launch__cluster_dim_y","launch__cluster_dim_z","launch__cluster_max_active","launch__cluster_max_potential_size","launch__cluster_scheduling_policy","launch__cluster_size","launch__context_id","launch__device_id","launch__func_cache_config","launch__function_pcs","launch__grid_dim_x","launch__grid_dim_y","launch__grid_dim_z","launch__grid_size","launch__kernel_name","launch__occupancy_cluster_gpu_pct","launch__occupancy_cluster_pct","launch__occupancy_limit_barriers","launch__occupancy_limit_blocks","launch__occupancy_limit_registers","launch__occupancy_limit_shared_mem","launch__occupancy_limit_warps","launch__occupancy_per_barrier_count","launch__occupancy_per_block_size","launch__occupancy_per_cluster_size","launch__occupancy_per_register_count","launch__occupancy_per_shared_mem_size","launch__persisting_l2_cache_size","launch__preferred_cluster_dim_x","launch__preferred_cluster_dim_y","launch__preferred_cluster_dim_z","launch__preferred_cluster_size","launch__registers_per_thread","launch__registers_per_thread_allocated","launch__shared_mem_config_size","launch__shared_mem_per_block","launch__shared_mem_per_block_allocated","launch__shared_mem_per_block_driver","launch__shared_mem_per_block_dynamic","launch__shared_mem_per_block_static","launch__sm_count","launch__stack_size","launch__stream_id","launch__thread_count","launch__tpc_count","launch__tpc_enabled","launch__uses_cdp","launch__uses_green_context","launch__uses_mps","launch__uses_nvlink_centric_scheduling","launch__uses_vgpu","launch__waves_per_multiprocessor","lts__t_sectors_op_read_lookup_miss.avg","lts__t_sectors_op_read_lookup_miss.max","lts__t_sectors_op_read_lookup_miss.min","lts__t_sectors_op_read_lookup_miss.sum","lts__t_sectors_op_write_lookup_miss.avg","lts__t_sectors_op_write_lookup_miss.max","lts__t_sectors_op_write_lookup_miss.min","lts__t_sectors_op_write_lookup_miss.sum","numa__cpu_affinity","numa__dev_display_name_all","numa__id_cpu","numa__id_memory","nvlink__bandwidth","nvlink__count_logical","nvlink__count_physical","nvlink__destination_ports","nvlink__dev0Id","nvlink__dev0type","nvlink__dev1Id","nvlink__dev1type","nvlink__dev_display_name_all","nvlink__enabled_mask","nvlink__is_direct_link","nvlink__is_nvswitch_connected","nvlink__max_count","nvlink__peer_access","nvlink__peer_atomic","nvlink__source_ports","nvlink__system_access","nvlink__system_atomic","profiler__perfworks_session_reuse","profiler__replayer_passes","profiler__replayer_passes_type_warmup","sm__maximum_warps_avg_per_active_cycle","sm__maximum_warps_per_active_cycle_pct","smsp__maximum_warps_avg_per_active_cycle" +"","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","ns","","block","block","block","","","","","cluster","block","","","","","","","","","","","","%","%","block","block","block","block","block","","","","","","byte","","","","","register/thread","register/thread","byte","byte/block","byte/block","byte/block","byte/block","byte/block","SM","","","thread","","","","","","","","","sector","sector","sector","sector","sector","sector","sector","sector","","","","","","","","","","","","","","","","","","","","","","","","pass","pass","warp","%","warp" +"0","61","python3.12","127.0.0.1","void at::::vectorized_layer_norm_kernel(int, T2, const T1 *, const T1 *, const T1 *, T2 *, T2 *, T1 *)","1","7","(32, 4, 1)","(37810, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3917408","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","1","1","37810","","0","0","24","24","12","28","12","336","2540","0","5264","956","4718592","0","0","0","0","38","40","32768","1048","1152","1024","24","0","48","1024","7","4839680","24","all","0","0","0","0","0","65.64","820245.50","824918","817137","13123928","794156.19","794335","794060","12706499","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","0","1","0","48","100","12" +"1","61","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","12160","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","161.81","270","64","2589","129.19","193","73","2067","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"2","61","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","9952","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","133.94","208","48","2143","128.75","202","68","2060","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"3","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(6216, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","80160","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","6216","1","1","6216","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","795648","24","all","0","0","0","0","0","10.79","54.62","132","0","874","24871.62","24919","24832","397946","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"4","61","python3.12","127.0.0.1","void ::partial_absmax_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3948416","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","8","28","12","288","1722","0","5264","1256","4718592","0","0","0","0","60","64","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.67","794602.12","794790","794484","12713634","46.56","121","0","745","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"5","61","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","21696","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","44.88","98","0","718","8.69","44","0","139","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"6","61","python3.12","127.0.0.1","void ::quantize_nvfp4_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, const float *, unsigned char *, unsigned char *, long, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2799872","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","6","16","6","512","2484","0","4976","1176","4718592","0","0","0","0","39","40","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","131.56","794672.69","794826","794534","12714763","204837.31","204913","204776","3277397","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"7","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","14528","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","18.38","67","0","294","4","10","0","64","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"8","61","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 168, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","27228896","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","168","1","49728","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","19095552","24","all","0","0","0","0","0","1036","5353448.62","5376398","5342153","85655178","3178438.75","3179022","3178147","50855020","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"9","61","python3.12","127.0.0.1","void comfy::::rope_kernel<__nv_bfloat16, __nv_bfloat16, __nv_bfloat16, 1, 1, 1, 1, 0>(const T1 *, const T1 *, const T2 *, const T3 *, const T3 *, T1 *, T1 *, long, long, long, int, int, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, float)","1","7","(128, 1, 1)","(529340, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","12278624","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","529340","1","1","529340","","0","0","24","24","4","32","12","216","456","0","5264","676","4718592","0","0","0","0","117","120","32768","1024","1024","1024","0","0","48","1024","7","67755520","24","all","0","0","0","0","0","2756.98","2152169.81","2157322","2144862","34434717","1110.19","1494","860","17763","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","16","33.33","4" +"10","61","python3.12","127.0.0.1","void at::reduce_kernel<128, 4, at::ReduceOp, unsigned int, c10::BFloat16, 4, 4>>(T3)","1","7","(32, 4, 1)","(56, 37, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2500512","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","56","37","1","2072","","0","0","24","24","8","20","12","288","1710","0","5264","1696","4718592","0","0","0","0","60","64","65536","3088","3200","1024","2048","16","48","1024","7","265216","24","all","0","0","0","0","0","5.40","1061648.38","1066978","1055750","16986374","2145.06","2226","2072","34321","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"11","61","python3.12","127.0.0.1","void QuantInt8Kernel<128, 32, 1, 0, 0, __nv_bfloat16>(T6 *, T6 *, signed char *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(512, 1, 1)","(1184, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4123296","1","512","1","1","512","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1184","56","1","66304","","0","0","24","24","4","6","3","752","2252","0","3760","3424","4718592","0","0","0","0","27","32","8192","1156","1280","1024","0","132","48","1024","7","33947648","24","all","0","0","0","0","0","460.44","1061247.75","1070985","1056154","16979964","529881.19","529998","529732","8478099","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"12","61","python3.12","127.0.0.1","void QuantInt8Kernel<128, 64, 1, 0, 1, __nv_bfloat16>(T6 *, T6 *, signed char *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(1024, 1, 1)","(591, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4019968","1","1024","1","1","1024","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","591","56","1","33096","","0","0","24","24","2","6","1","800","2252","0","2080","6368","4718592","0","0","0","0","25","32","8192","1156","1280","1024","0","132","48","1024","7","33890304","24","all","0","0","0","0","0","689.50","1060403.38","1065378","1055285","16966454","529638.25","529732","529513","8474212","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"13","61","python3.12","127.0.0.1","void TransposePadPermuteKernel<128, 64, 1, __nv_bfloat16>(T4 *, T4 *, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(1024, 1, 1)","(591, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5422144","1","1024","1","1","1024","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","591","56","1","33096","","0","0","24","24","2","1","1","800","1057","0","2080","6368","4718592","0","0","0","0","24","24","65536","33792","33792","1024","0","32768","48","1024","7","33890304","24","all","0","0","0","0","0","689.50","1059967.56","1066970","1054726","16959481","1059222.19","1059320","1059104","16947555","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"14","61","python3.12","127.0.0.1","void MeanScaleKernel<64, 0, __nv_bfloat16>(T3 *, signed char *, float *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(256, 1, 1)","(56, 1, 128)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5691392","1","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","56","1","128","7168","","0","0","24","24","6","23","6","512","2533","0","4976","2360","4718592","0","0","0","0","40","40","32768","1284","1408","1024","0","260","48","1024","7","1835008","24","all","0","0","0","0","0","24.89","1925826.75","1928326","1922586","30813228","470158.56","471030","469086","7522537","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"15","61","python3.12","127.0.0.1","void qk_int_sv_f8_attn_kernel<128, 64, 32, 64, 128, 1, 2, 2, float, 1, __nv_bfloat16, 1, 0, 0, 1, 0, 1>(signed char *, signed char *, signed char *, T11 *, float *, float *, float *, float *, float *, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, float)","1","7","(32, 4, 1)","(296, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","257743840","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","56","1","16576","","0","0","24","24","2","3","12","152","101","0","2732","1200","4718592","0","0","0","0","255","256","102400","33792","33792","1024","32768","0","48","1024","7","2121728","24","all","0","0","0","0","0","172.67","2576930.94","2592118","2563870","41230895","1079616.19","1086171","1076688","17273859","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","8","16.67","2" +"16","61","python3.12","127.0.0.1","::partial_absmax_bf16_vec_kernel(const uint4 *, const unsigned short *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2340576","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","12","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","40","40","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.44","1058459.06","1058628","1058293","16935345","36.12","87","0","578","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"17","61","python3.12","127.0.0.1","::final_scale_warp_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3200","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","15.75","64","0","252","2.69","8","0","43","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"18","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::bfloat16_copy_kernel_cuda(at::TensorIteratorBase &)::[lambda(float) (instance 1)], std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","13568","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","18","78","0","288","3.81","8","1","61","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"19","61","python3.12","127.0.0.1","void at::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, memory::LoadWithCast<1>, memory::StoreWithCast<1>>(int, T1, T2, T4, T5, T6, T7)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4736","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","72.25","168","0","1156","2.38","8","0","38","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"20","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(8288, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","77824","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","8288","1","1","8288","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","1060864","24","all","0","0","0","0","0","14.39","134.25","412","16","2148","33173.06","33198","33152","530769","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"21","61","python3.12","127.0.0.1","void comfy::::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, 1, 1>(const T1 *, const float *, T2 *, T3 *, unsigned long, unsigned long, unsigned long, unsigned long, float)","1","7","(128, 1, 1)","(529536, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3305536","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","529536","1","1","529536","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","67780608","24","all","0","0","0","0","0","919.33","1059736.62","1060354","1059310","16955786","151033.81","151292","150696","2416541","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"22","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","7136","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","17.12","65","0","274","2.75","14","0","44","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"23","61","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 42, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","8537312","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","42","1","12432","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","4773888","24","all","0","0","0","0","0","259","1817245.56","1819027","1813874","29075929","794369.38","794435","794304","12709910","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"24","61","python3.12","127.0.0.1","_gate_add_kernel","1","7","(128, 1, 1)","(37810, 6, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5829152","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","6","1","226860","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","29038080","24","all","0","0","0","0","0","393.85","1591085.75","1591478","1590664","25457372","168.25","413","6","2692","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"25","61","python3.12","127.0.0.1","void at::::vectorized_layer_norm_kernel(int, T2, const T1 *, const T1 *, const T1 *, T2 *, T2 *, T1 *)","1","7","(32, 4, 1)","(37810, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3777824","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","1","1","37810","","0","0","24","24","12","28","12","336","2540","0","5264","956","4718592","0","0","0","0","38","40","32768","1048","1152","1024","24","0","48","1024","7","4839680","24","all","0","0","0","0","0","65.64","816189.69","827623","809429","13059035","794128.75","794256","794052","12706060","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"26","61","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","6208","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","158.06","216","80","2529","128.50","200","65","2056","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"27","61","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4384","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","127.44","179","80","2039","128.50","194","68","2056","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"28","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(6216, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","69344","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","6216","1","1","6216","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","795648","24","all","0","0","0","0","0","10.79","43","112","2","688","24869.69","24917","24832","397915","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"29","61","python3.12","127.0.0.1","void ::partial_absmax_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3989568","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","8","28","12","288","1722","0","5264","1256","4718592","0","0","0","0","60","64","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.67","795642.19","796200","795134","12730275","203","263","128","3248","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"30","61","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","7296","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","19.88","68","0","318","2.44","10","0","39","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"31","61","python3.12","127.0.0.1","void ::quantize_nvfp4_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, const float *, unsigned char *, unsigned char *, long, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2821344","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","6","16","6","512","2484","0","4976","1176","4718592","0","0","0","0","39","40","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","131.56","794676.25","794817","794536","12714820","204845.12","204937","204776","3277522","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"32","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","26496","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","19.44","74","0","311","5.25","13","0","84","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"33","61","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 224, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","36742080","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","224","1","66304","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","25460736","24","all","0","0","0","0","0","1381.33","7136783.69","7167717","7118657","114188539","4237745","4238420","4237430","67803920","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"34","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(16576, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","163424","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","16576","1","1","16576","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","2121728","24","all","0","0","0","0","0","28.78","24.38","58","0","390","48825.25","49504","48419","781204","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"35","61","python3.12","127.0.0.1","::partial_absmax_swiglu_bf16_kernel(const unsigned short *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","13170368","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","16","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","28","32","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.44","4238501.25","4239692","4237682","67816020","922.31","1325","519","14757","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"36","61","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3616","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","7.56","44","0","121","2.31","10","0","37","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"37","61","python3.12","127.0.0.1","::quantize_nvfp4_swiglu_bf16_kernel(const unsigned short *, const float *, unsigned char *, unsigned char *, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","10651552","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","5","16","6","472","2052","0","4976","1128","4718592","0","0","0","0","45","48","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","157.87","4235033","4235090","4234914","67760528","546174.44","546284","546105","8738791","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","40","83.33","10" +"38","61","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","24128","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","16.94","51","0","271","5.25","13","0","84","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"39","61","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k","1","7","(384, 1, 1)","(42, 296, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","54593632","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","42","296","1","12432","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","4773888","24","all","0","0","0","0","0","259","22956311.19","22958936","22954426","367300979","795898.06","796466","795537","12734369","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"40","61","python3.12","127.0.0.1","_gate_add_kernel","1","7","(128, 1, 1)","(37810, 6, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5628032","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","6","1","226860","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","29038080","24","all","0","0","0","0","0","393.85","1591199.12","1591656","1590898","25459186","118.25","175","78","1892","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" diff --git a/benchmarks/gb10-fully-fused-block24-fresh-l2.csv b/benchmarks/gb10-fully-fused-block24-fresh-l2.csv new file mode 100644 index 0000000..e828d81 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-l2.csv @@ -0,0 +1,46 @@ +==PROF== Connected to process 59 (/usr/bin/python3.12) +==PROF== Disconnected from process 59 +==PROF== Report: /output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2.ncu-rep +"ID","Process ID","Process Name","Host Name","Kernel Name","Context","Stream","Block Size","Grid Size","Device","CC","c2clink__enabled_mask","c2clink__present","device__attribute_architecture","device__attribute_async_engine_count","device__attribute_can_flush_remote_writes","device__attribute_can_map_host_memory","device__attribute_can_tex2d_gather","device__attribute_can_use_64_bit_stream_mem_ops","device__attribute_can_use_64_bit_stream_mem_ops_v1","device__attribute_can_use_host_pointer_for_registered_mem","device__attribute_can_use_stream_mem_ops_v1","device__attribute_can_use_stream_wait_value_nor","device__attribute_can_use_stream_wait_value_nor_v1","device__attribute_chip","device__attribute_clock_rate","device__attribute_cluster_launch","device__attribute_compute_capability_major","device__attribute_compute_capability_minor","device__attribute_compute_mode","device__attribute_compute_preemption_supported","device__attribute_concurrent_kernels","device__attribute_concurrent_managed_access","device__attribute_confidential_computing_mode","device__attribute_cooperative_launch","device__attribute_cooperative_multi_device_launch","device__attribute_deferred_mapping_cuda_array_supported","device__attribute_device_index","device__attribute_direct_managed_mem_access_from_host","device__attribute_display_name","device__attribute_dma_buf_supported","device__attribute_ecc_enabled","device__attribute_fb_bus_width","device__attribute_fbp_count","device__attribute_generic_compression_supported","device__attribute_global_l1_cache_supported","device__attribute_global_memory_bus_width","device__attribute_gpu_direct_rdma_flush_writes_options","device__attribute_gpu_direct_rdma_supported","device__attribute_gpu_direct_rdma_with_cuda_vmm_supported","device__attribute_gpu_direct_rdma_writes_ordering","device__attribute_gpu_overlap","device__attribute_gpu_pci_device_id","device__attribute_gpu_pci_ext_device_id","device__attribute_gpu_pci_ext_downstream_link_rate","device__attribute_gpu_pci_ext_downstream_link_width","device__attribute_gpu_pci_ext_gen","device__attribute_gpu_pci_ext_gpu_gen","device__attribute_gpu_pci_ext_gpu_link_rate","device__attribute_gpu_pci_ext_gpu_link_width","device__attribute_gpu_pci_revision_id","device__attribute_gpu_pci_sub_system_id","device__attribute_handle_type_fabric_supported","device__attribute_handle_type_posix_file_descriptor_supported","device__attribute_handle_type_win32_handle_supported","device__attribute_handle_type_win32_kmt_handle_supported","device__attribute_host_native_atomic_supported","device__attribute_host_numa_id","device__attribute_host_register_supported","device__attribute_implementation","device__attribute_integrated","device__attribute_ipc_event_supported","device__attribute_kernel_exec_timeout","device__attribute_l2_cache_size","device__attribute_l2s_count","device__attribute_limits_max_cta_per_sm","device__attribute_limits_num_tpcs","device__attribute_local_l1_cache_supported","device__attribute_managed_memory","device__attribute_max_access_policy_window_size","device__attribute_max_block_dim_x","device__attribute_max_block_dim_y","device__attribute_max_block_dim_z","device__attribute_max_blocks_per_multiprocessor","device__attribute_max_gpu_frequency_khz","device__attribute_max_grid_dim_x","device__attribute_max_grid_dim_y","device__attribute_max_grid_dim_z","device__attribute_max_ipc_per_multiprocessor","device__attribute_max_ipc_per_scheduler","device__attribute_max_mem_frequency_khz","device__attribute_max_persisting_l2_cache_size","device__attribute_max_pitch","device__attribute_max_registers_per_block","device__attribute_max_registers_per_multiprocessor","device__attribute_max_registers_per_thread","device__attribute_max_shared_memory_per_block","device__attribute_max_shared_memory_per_block_optin","device__attribute_max_shared_memory_per_multiprocessor","device__attribute_max_threads_per_block","device__attribute_max_threads_per_multiprocessor","device__attribute_max_warps_per_multiprocessor","device__attribute_max_warps_per_scheduler","device__attribute_maximum_surface1d_layered_layers","device__attribute_maximum_surface1d_layered_width","device__attribute_maximum_surface1d_width","device__attribute_maximum_surface2d_height","device__attribute_maximum_surface2d_layered_height","device__attribute_maximum_surface2d_layered_layers","device__attribute_maximum_surface2d_layered_width","device__attribute_maximum_surface2d_width","device__attribute_maximum_surface3d_depth","device__attribute_maximum_surface3d_height","device__attribute_maximum_surface3d_width","device__attribute_maximum_surfacecubemap_layered_layers","device__attribute_maximum_surfacecubemap_layered_width","device__attribute_maximum_surfacecubemap_width","device__attribute_maximum_texture1d_layered_layers","device__attribute_maximum_texture1d_layered_width","device__attribute_maximum_texture1d_linear_width","device__attribute_maximum_texture1d_mipmapped_width","device__attribute_maximum_texture1d_width","device__attribute_maximum_texture2d_gather_height","device__attribute_maximum_texture2d_gather_width","device__attribute_maximum_texture2d_height","device__attribute_maximum_texture2d_layered_height","device__attribute_maximum_texture2d_layered_layers","device__attribute_maximum_texture2d_layered_width","device__attribute_maximum_texture2d_linear_height","device__attribute_maximum_texture2d_linear_pitch","device__attribute_maximum_texture2d_linear_width","device__attribute_maximum_texture2d_mipmapped_height","device__attribute_maximum_texture2d_mipmapped_width","device__attribute_maximum_texture2d_width","device__attribute_maximum_texture3d_depth","device__attribute_maximum_texture3d_depth_alternate","device__attribute_maximum_texture3d_height","device__attribute_maximum_texture3d_height_alternate","device__attribute_maximum_texture3d_width","device__attribute_maximum_texture3d_width_alternate","device__attribute_maximum_texturecubemap_layered_layers","device__attribute_maximum_texturecubemap_layered_width","device__attribute_maximum_texturecubemap_width","device__attribute_mem_sync_domain_count","device__attribute_memory_clock_rate","device__attribute_memory_pools_supported","device__attribute_mempool_supported_handle_types","device__attribute_mps_enabled","device__attribute_multi_gpu_board","device__attribute_multi_gpu_board_group_id","device__attribute_multicast_supported","device__attribute_multiprocessor_count","device__attribute_num_l2s_per_fbp","device__attribute_num_schedulers_per_multiprocessor","device__attribute_num_tex_per_multiprocessor","device__attribute_numa_config","device__attribute_pageable_memory_access","device__attribute_pageable_memory_access_uses_host_page_tables","device__attribute_pci_bus_id","device__attribute_pci_device_id","device__attribute_pci_domain_id","device__attribute_ram_location","device__attribute_ram_type","device__attribute_reserved_shared_memory_per_block","device__attribute_sass_level","device__attribute_single_to_double_precision_perf_ratio","device__attribute_sparse_cuda_array_supported","device__attribute_stream_priorities_supported","device__attribute_surface_alignment","device__attribute_tcc_driver","device__attribute_tensor_map_access_supported","device__attribute_texture_alignment","device__attribute_texture_pitch_alignment","device__attribute_total_constant_memory","device__attribute_total_memory","device__attribute_unified_addressing","device__attribute_unified_function_pointers","device__attribute_virtual_address_management_supported","device__attribute_warp_size","gpu__time_duration.sum","launch__barrier_count","launch__block_dim_x","launch__block_dim_y","launch__block_dim_z","launch__block_size","launch__cluster_dim_x","launch__cluster_dim_y","launch__cluster_dim_z","launch__cluster_max_active","launch__cluster_max_potential_size","launch__cluster_scheduling_policy","launch__cluster_size","launch__context_id","launch__device_id","launch__func_cache_config","launch__function_pcs","launch__grid_dim_x","launch__grid_dim_y","launch__grid_dim_z","launch__grid_size","launch__kernel_name","launch__occupancy_cluster_gpu_pct","launch__occupancy_cluster_pct","launch__occupancy_limit_barriers","launch__occupancy_limit_blocks","launch__occupancy_limit_registers","launch__occupancy_limit_shared_mem","launch__occupancy_limit_warps","launch__occupancy_per_barrier_count","launch__occupancy_per_block_size","launch__occupancy_per_cluster_size","launch__occupancy_per_register_count","launch__occupancy_per_shared_mem_size","launch__persisting_l2_cache_size","launch__preferred_cluster_dim_x","launch__preferred_cluster_dim_y","launch__preferred_cluster_dim_z","launch__preferred_cluster_size","launch__registers_per_thread","launch__registers_per_thread_allocated","launch__shared_mem_config_size","launch__shared_mem_per_block","launch__shared_mem_per_block_allocated","launch__shared_mem_per_block_driver","launch__shared_mem_per_block_dynamic","launch__shared_mem_per_block_static","launch__sm_count","launch__stack_size","launch__stream_id","launch__thread_count","launch__tpc_count","launch__tpc_enabled","launch__uses_cdp","launch__uses_green_context","launch__uses_mps","launch__uses_nvlink_centric_scheduling","launch__uses_vgpu","launch__waves_per_multiprocessor","lts__t_bytes.avg","lts__t_bytes.max","lts__t_bytes.min","lts__t_bytes.sum","numa__cpu_affinity","numa__dev_display_name_all","numa__id_cpu","numa__id_memory","nvlink__bandwidth","nvlink__count_logical","nvlink__count_physical","nvlink__destination_ports","nvlink__dev0Id","nvlink__dev0type","nvlink__dev1Id","nvlink__dev1type","nvlink__dev_display_name_all","nvlink__enabled_mask","nvlink__is_direct_link","nvlink__is_nvswitch_connected","nvlink__max_count","nvlink__peer_access","nvlink__peer_atomic","nvlink__source_ports","nvlink__system_access","nvlink__system_atomic","profiler__perfworks_session_reuse","profiler__replayer_passes","profiler__replayer_passes_type_warmup","sm__maximum_warps_avg_per_active_cycle","sm__maximum_warps_per_active_cycle_pct","smsp__maximum_warps_avg_per_active_cycle" +"","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","","ns","","block","block","block","","","","","cluster","block","","","","","","","","","","","","%","%","block","block","block","block","block","","","","","","byte","","","","","register/thread","register/thread","byte","byte/block","byte/block","byte/block","byte/block","byte/block","SM","","","thread","","","","","","","","","byte","byte","byte","byte","","","","","","","","","","","","","","","","","","","","","","","","pass","pass","warp","%","warp" +"0","59","python3.12","127.0.0.1","void at::::vectorized_layer_norm_kernel(int, T2, const T1 *, const T1 *, const T1 *, T2 *, T2 *, T1 *)","1","7","(32, 4, 1)","(37810, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3835936","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","1","1","37810","","0","0","24","24","12","28","12","336","2540","0","5264","956","4718592","0","0","0","0","38","40","32768","1048","1152","1024","24","0","48","1024","7","4839680","24","all","0","0","0","0","0","65.64","76492618","77883424","75811744","1223881888","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","0","1","0","48","100","12" +"1","59","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","11872","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","12318","19712","8000","197088","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"2","59","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5888","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","12198","17152","9088","195168","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"3","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(6216, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","62304","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","6216","1","1","6216","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","795648","24","all","0","0","0","0","0","10.79","800118","805408","795584","12801888","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"4","59","python3.12","127.0.0.1","void ::partial_absmax_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3924992","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","8","28","12","288","1722","0","5264","1256","4718592","0","0","0","0","60","64","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.67","27016280","27704320","26280288","432260480","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"5","59","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","11488","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","2848","8960","128","45568","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"6","59","python3.12","127.0.0.1","void ::quantize_nvfp4_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, const float *, unsigned char *, unsigned char *, long, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2828000","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","6","16","6","512","2484","0","4976","1176","4718592","0","0","0","0","39","40","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","131.56","121705592","130178336","114206400","1947289472","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"7","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","8672","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","1924","7616","64","30784","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"8","59","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 168, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","27195296","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","168","1","49728","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","19095552","24","all","0","0","0","0","0","1036","2508066908","2509524896","2506560864","40129070528","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"9","59","python3.12","127.0.0.1","void comfy::::rope_kernel<__nv_bfloat16, __nv_bfloat16, __nv_bfloat16, 1, 1, 1, 1, 0>(const T1 *, const T1 *, const T2 *, const T3 *, const T3 *, T1 *, T1 *, long, long, long, int, int, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, float)","1","7","(128, 1, 1)","(529340, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","12414464","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","529340","1","1","529340","","0","0","24","24","4","32","12","216","456","0","5264","676","4718592","0","0","0","0","117","120","32768","1024","1024","1024","0","0","48","1024","7","67755520","24","all","0","0","0","0","0","2756.98","147135740","147526592","146694336","2354171840","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","16","33.33","4" +"10","59","python3.12","127.0.0.1","void at::reduce_kernel<128, 4, at::ReduceOp, unsigned int, c10::BFloat16, 4, 4>>(T3)","1","7","(32, 4, 1)","(56, 37, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2480096","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","56","37","1","2072","","0","0","24","24","8","20","12","288","1710","0","5264","1696","4718592","0","0","0","0","60","64","65536","3088","3200","1024","2048","16","48","1024","7","265216","24","all","0","0","0","0","0","5.40","34178678","34353408","34027392","546858848","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"11","59","python3.12","127.0.0.1","void QuantInt8Kernel<128, 32, 1, 0, 0, __nv_bfloat16>(T6 *, T6 *, signed char *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(512, 1, 1)","(1184, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4156928","1","512","1","1","512","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1184","56","1","66304","","0","0","24","24","4","6","3","752","2252","0","3760","3424","4718592","0","0","0","0","27","32","8192","1156","1280","1024","0","132","48","1024","7","33947648","24","all","0","0","0","0","0","460.44","51025298","51401984","50802592","816404768","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"12","59","python3.12","127.0.0.1","void QuantInt8Kernel<128, 64, 1, 0, 1, __nv_bfloat16>(T6 *, T6 *, signed char *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(1024, 1, 1)","(591, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4006400","1","1024","1","1","1024","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","591","56","1","33096","","0","0","24","24","2","6","1","800","2252","0","2080","6368","4718592","0","0","0","0","25","32","8192","1156","1280","1024","0","132","48","1024","7","33890304","24","all","0","0","0","0","0","689.50","50947822","51175552","50750656","815165152","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"13","59","python3.12","127.0.0.1","void TransposePadPermuteKernel<128, 64, 1, __nv_bfloat16>(T4 *, T4 *, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(1024, 1, 1)","(591, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5426240","1","1024","1","1","1024","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","591","56","1","33096","","0","0","24","24","2","1","1","800","1057","0","2080","6368","4718592","0","0","0","0","24","24","65536","33792","33792","1024","0","32768","48","1024","7","33890304","24","all","0","0","0","0","0","689.50","67786576","67981088","67585408","1084585216","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"14","59","python3.12","127.0.0.1","void MeanScaleKernel<64, 0, __nv_bfloat16>(T3 *, signed char *, float *, float *, float, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int)","1","7","(256, 1, 1)","(56, 1, 128)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5602336","1","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","56","1","128","7168","","0","0","24","24","6","23","6","512","2533","0","4976","2360","4718592","0","0","0","0","40","40","32768","1284","1408","1024","0","260","48","1024","7","1835008","24","all","0","0","0","0","0","24.89","84760040","84777376","84739840","1356160640","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"15","59","python3.12","127.0.0.1","void qk_int_sv_f8_attn_kernel<128, 64, 32, 64, 128, 1, 2, 2, float, 1, __nv_bfloat16, 1, 0, 0, 1, 0, 1>(signed char *, signed char *, signed char *, T11 *, float *, float *, float *, float *, float *, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, float)","1","7","(32, 4, 1)","(296, 56, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","258715776","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","56","1","16576","","0","0","24","24","2","3","12","152","101","0","2732","1200","4718592","0","0","0","0","255","256","102400","33792","33792","1024","32768","0","48","1024","7","2121728","24","all","0","0","0","0","0","172.67","10094185780","10097011776","10090711136","161506972480","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","8","16.67","2" +"16","59","python3.12","127.0.0.1","::partial_absmax_bf16_vec_kernel(const uint4 *, const unsigned short *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2300704","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","12","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","40","40","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.44","33900936","33935648","33889824","542414976","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"17","59","python3.12","127.0.0.1","::final_scale_warp_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","7232","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","2684","10272","64","42944","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"18","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::bfloat16_copy_kernel_cuda(at::TensorIteratorBase &)::[lambda(float) (instance 1)], std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5120","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","1692","8480","64","27072","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"19","59","python3.12","127.0.0.1","void at::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, memory::LoadWithCast<1>, memory::StoreWithCast<1>>(int, T1, T2, T4, T5, T6, T7)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5344","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","8300","14624","704","132800","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"20","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(8288, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","86784","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","8288","1","1","8288","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","1060864","24","all","0","0","0","0","0","14.39","1066126","1072288","1062080","17058016","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"21","59","python3.12","127.0.0.1","void comfy::::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, 1, 1>(const T1 *, const float *, T2 *, T3 *, unsigned long, unsigned long, unsigned long, unsigned long, float)","1","7","(128, 1, 1)","(529536, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3287392","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","529536","1","1","529536","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","67780608","24","all","0","0","0","0","0","919.33","50886998","50920608","50861504","814191968","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"22","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","15520","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","2062","8480","64","32992","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"23","59","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 42, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","9250720","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","42","1","12432","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","4773888","24","all","0","0","0","0","0","259","827548744","827957600","826938848","13240779904","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"24","59","python3.12","127.0.0.1","_gate_add_kernel","1","7","(128, 1, 1)","(37810, 6, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5610528","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","6","1","226860","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","29038080","24","all","0","0","0","0","0","393.85","77251968","77629408","76679072","1236031488","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"25","59","python3.12","127.0.0.1","void at::::vectorized_layer_norm_kernel(int, T2, const T1 *, const T1 *, const T1 *, T2 *, T2 *, T1 *)","1","7","(32, 4, 1)","(37810, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3929888","1","32","4","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","1","1","37810","","0","0","24","24","12","28","12","336","2540","0","5264","956","4718592","0","0","0","0","38","40","32768","1048","1152","1024","24","0","48","1024","7","4839680","24","all","0","0","0","0","0","65.64","76551854","78109792","75792896","1224829664","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"26","59","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","9856","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","12340","19200","6816","197440","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"27","59","python3.12","127.0.0.1","void at::elementwise_kernel<128, 2, void at::gpu_kernel_impl_nocast(at::TensorIteratorBase &, const T1 &)::[lambda(int) (instance 1)]>(int, T3)","1","7","(128, 1, 1)","(63, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","4608","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","63","1","1","63","","0","0","24","24","21","32","12","336","2540","0","5264","956","4718592","0","0","0","0","18","24","32768","1024","1024","1024","0","0","48","1024","7","8064","24","all","0","0","0","0","0","0.11","12148","17952","5952","194368","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"28","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(6216, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","57088","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","6216","1","1","6216","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","795648","24","all","0","0","0","0","0","10.79","800050","805920","795904","12800800","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"29","59","python3.12","127.0.0.1","void ::partial_absmax_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","3997056","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","8","28","12","288","1722","0","5264","1256","4718592","0","0","0","0","60","64","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.67","27180280","27940640","26353792","434884480","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","32","66.67","8" +"30","59","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","21152","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","3322","10624","448","53152","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"31","59","python3.12","127.0.0.1","void ::quantize_nvfp4_modulated_bf16_kernel(const unsigned short *, const T1 *, const T1 *, const int *, const float *, unsigned char *, unsigned char *, long, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","2774720","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","6","16","6","512","2484","0","4976","1176","4718592","0","0","0","0","39","40","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","131.56","122062326","129937024","114884128","1952997216","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"32","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","6080","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","2108","7712","64","33728","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"33","59","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu","1","7","(384, 1, 1)","(296, 224, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","37024512","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","296","224","1","66304","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","25460736","24","all","0","0","0","0","0","1381.33","3344235266","3345719328","3342640864","53507764256","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"34","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::FillFunctor, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(16576, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","178816","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","16576","1","1","16576","","0","0","24","24","32","32","12","336","2540","0","5264","956","4718592","0","0","0","0","16","16","32768","1024","1024","1024","0","0","48","1024","7","2121728","24","all","0","0","0","0","0","28.78","2130438","2144672","2122560","34087008","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"35","59","python3.12","127.0.0.1","::partial_absmax_swiglu_bf16_kernel(const unsigned short *, float *, long, long)","1","7","(128, 1, 1)","(256, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","13130528","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","256","1","1","256","","0","0","24","24","16","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","28","32","32768","1152","1152","1024","0","128","48","1024","7","32768","24","all","0","0","0","0","0","0.44","135705878","135866208","135640448","2171294048","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"36","59","python3.12","127.0.0.1","::final_scale_bf16_compat_kernel(const float *, float *, long, float)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","14528","1","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","21","28","12","336","2540","0","5264","1364","4718592","0","0","0","0","24","24","32768","1152","1152","1024","0","128","48","1024","7","128","24","all","0","0","0","0","0","0.00","2942","9472","128","47072","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"37","59","python3.12","127.0.0.1","::quantize_nvfp4_swiglu_bf16_kernel(const unsigned short *, const float *, unsigned char *, unsigned char *, long, long, long, long, long)","1","7","(256, 1, 1)","(37888, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","10624672","0","256","1","1","256","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37888","1","1","37888","","0","0","24","24","5","16","6","472","2052","0","4976","1128","4718592","0","0","0","0","45","48","16384","1024","1024","1024","0","0","48","1024","7","9699328","24","all","0","0","0","0","0","157.87","391649550","394731200","390064192","6266392800","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","40","83.33","10" +"38","59","python3.12","127.0.0.1","void at::vectorized_elementwise_kernel<4, at::BinaryFunctor>, std::array>(int, T2, T3)","1","7","(128, 1, 1)","(1, 1, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","6048","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","1","1","1","1","","0","0","24","24","12","32","12","336","2540","0","5264","956","4718592","0","0","0","0","40","40","32768","1024","1024","1024","0","0","48","1024","7","128","24","all","0","0","0","0","0","0.00","1912","9440","64","30592","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" +"39","59","python3.12","127.0.0.1","cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k","1","7","(384, 1, 1)","(42, 296, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","54973312","8","384","1","1","384","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","42","296","1","12432","","0","0","3","24","1","1","4","300","157","0","2028","2388","4718592","0","0","0","0","168","168","102400","89088","89088","1024","88064","0","48","1024","7","4773888","24","all","0","0","0","0","0","259","1629999368","1630358560","1629644288","26079989888","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","12","25","3" +"40","59","python3.12","127.0.0.1","_gate_add_kernel","1","7","(128, 1, 1)","(37810, 6, 1)","0","12.1","0","1","432","1","0","1","1","1","0","1","0","1","0","443","2418000","1","12","1","0","1","1","1","No-CC","1","1","1","0","0","NVIDIA GB10","0","0","256","4","1","1","256","1","0","0","100","1","772935902","11794","32000","16","0","4","2500","16","161","4318","0","1","0","0","1","0","1","443","1","1","1","25165824","24","24","24","1","1","134217728","1024","1024","64","24","2418000","2147483647","65535","65535","4","1","8533000","18874368","2147483647","65536","65536","255","49152","101376","102400","1024","1536","48","12","2048","32768","32768","65536","32768","2048","32768","131072","16384","16384","16384","2046","32768","32768","2048","32768","268435456","32768","131072","32768","32768","65536","32768","2048","32768","65000","2097120","131072","32768","32768","131072","16384","32768","16384","8192","16384","8192","2046","32768","32768","4","8533000","1","1","0","0","0","0","48","6","4","1","0","1","1","1","0","15","1","0","1024","12","64","1","1","512","0","1","512","32","65536","130661769216","1","1","1","32","5603104","0","128","1","1","128","0","0","0","0","8","PolicySpread","0","1","0","CachePreferNone","1","37810","6","1","226860","","0","0","24","24","16","32","12","336","2540","0","5264","956","4718592","0","0","0","0","32","32","32768","1024","1024","1024","0","0","48","1024","7","29038080","24","all","0","0","0","0","0","393.85","77276300","77845184","76834432","1236420800","1","1","1","1","0","0","0","0","0","0","0","0","1","0","0","0","0","0","0","0","0","0","1","1","0","48","100","12" diff --git a/benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json b/benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json new file mode 100644 index 0000000..fb4db12 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json @@ -0,0 +1,31 @@ +{ + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "fused_elementwise": true, + "fused_nvfp4_modulation": true, + "fused_nvfp4_swiglu": true, + "nvfp4_scale_backend": "vortex", + "sage_qkv_layout": "strided_nhd", + "module_forward_checksum": 273326784.0, + "capture": "one warmed block between cudaProfilerStart/Stop" +} \ No newline at end of file diff --git a/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json b/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json new file mode 100644 index 0000000..fb4db12 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json @@ -0,0 +1,31 @@ +{ + "block_index": 24, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "fused_elementwise": true, + "fused_nvfp4_modulation": true, + "fused_nvfp4_swiglu": true, + "nvfp4_scale_backend": "vortex", + "sage_qkv_layout": "strided_nhd", + "module_forward_checksum": 273326784.0, + "capture": "one warmed block between cudaProfilerStart/Stop" +} \ No newline at end of file diff --git a/benchmarks/gb10-fully-fused-block24-fresh-torch.json b/benchmarks/gb10-fully-fused-block24-fresh-torch.json new file mode 100644 index 0000000..14b4b31 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-fresh-torch.json @@ -0,0 +1,1292 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 5, + "iterations": 10, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 10, + "mean_s": 0.0036379225028213115, + "p50_s": 0.003614433022448793, + "p90_s": 0.0037366025615483524, + "p95_s": 0.0037547897722106423, + "p99_s": 0.003769339540740475, + "min_s": 0.0035753760021179914, + "max_s": 0.003772976982872933 + }, + "modulate_msa": { + "count": 10, + "mean_s": 0.010608601296553388, + "p50_s": 0.010585986980004236, + "p90_s": 0.011015687783947214, + "p95_s": 0.011020101406029426, + "p99_s": 0.011023632303695194, + "min_s": 0.01028603402664885, + "max_s": 0.011024515028111637 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.382384940981865e-06, + "p50_s": 4.103989340364933e-06, + "p90_s": 4.502356750890611e-06, + "p95_s": 5.7551631471142146e-06, + "p99_s": 6.757408264093101e-06, + "min_s": 3.95200913771987e-06, + "max_s": 7.007969543337822e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 1.015679445117712e-05, + "p50_s": 8.024013368412852e-06, + "p90_s": 1.037599868141114e-05, + "p95_s": 1.9915992743335642e-05, + "p99_s": 2.754798799287528e-05, + "min_s": 7.760012522339821e-06, + "max_s": 2.945598680526018e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 10, + "mean_s": 3.4719996619969606e-06, + "p50_s": 3.416003892198205e-06, + "p90_s": 3.587204264476895e-06, + "p95_s": 3.7455960409715767e-06, + "p99_s": 3.872309462167323e-06, + "min_s": 3.328023012727499e-06, + "max_s": 3.903987817466259e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 10, + "mean_s": 0.001784054801100865, + "p50_s": 0.0017840564833022654, + "p90_s": 0.0018375497369561345, + "p95_s": 0.001851214884663932, + "p99_s": 0.0018621470028301699, + "min_s": 0.0017313450225628912, + "max_s": 0.0018648800323717296 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 10, + "mean_s": 1.0225595906376838e-05, + "p50_s": 9.207986295223236e-06, + "p90_s": 1.1627195635810492e-05, + "p95_s": 1.5061601880006484e-05, + "p99_s": 1.7809126875363292e-05, + "min_s": 8.128001354634762e-06, + "max_s": 1.849600812420249e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.002209685498382896, + "p50_s": 0.0021846969902981073, + "p90_s": 0.0022998010332230478, + "p95_s": 0.0023025010275887327, + "p99_s": 0.0023046610230812803, + "min_s": 0.0021451999782584608, + "max_s": 0.0023052010219544172 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.39361678622663e-06, + "p50_s": 6.1680038925260305e-06, + "p90_s": 6.588839460164307e-06, + "p95_s": 7.510429713875053e-06, + "p99_s": 8.247701916843653e-06, + "min_s": 5.936017259955406e-06, + "max_s": 8.432019967585802e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 10, + "mean_s": 0.026259651698637754, + "p50_s": 0.026372870983323082, + "p90_s": 0.026779549376806246, + "p95_s": 0.026793114203610458, + "p99_s": 0.02680396606505383, + "min_s": 0.025480758980847895, + "max_s": 0.02680667903041467 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 10, + "mean_s": 5.59519394300878e-06, + "p50_s": 5.456007784232497e-06, + "p90_s": 5.7215744163841006e-06, + "p95_s": 6.412793300114571e-06, + "p99_s": 6.965768407098949e-06, + "min_s": 5.264009814709425e-06, + "max_s": 7.104012183845043e-06 + }, + "attn_qkv_proj": { + "count": 10, + "mean_s": 0.03033423840533942, + "p50_s": 0.03052276000380516, + "p90_s": 0.030788672040216625, + "p95_s": 0.03079540401813574, + "p99_s": 0.030800789600471036, + "min_s": 0.02970914397155866, + "max_s": 0.030802135996054858 + }, + "attn_qkv_split_view": { + "count": 10, + "mean_s": 1.1774385347962379e-05, + "p50_s": 1.0199990356341004e-05, + "p90_s": 1.3715191744267938e-05, + "p95_s": 1.768957590684294e-05, + "p99_s": 2.0869083236902955e-05, + "min_s": 9.935989510267973e-06, + "max_s": 2.1663960069417953e-05 + }, + "attn_qk_rms_rope": { + "count": 10, + "mean_s": 0.012174534396035597, + "p50_s": 0.012181266996776685, + "p90_s": 0.012462468614103273, + "p95_s": 0.012474859817302786, + "p99_s": 0.012484772779862397, + "min_s": 0.011805346992332488, + "max_s": 0.012487251020502299 + }, + "attention_kernel": { + "count": 10, + "mean_s": 0.26265805189614183, + "p50_s": 0.26240472550853156, + "p90_s": 0.2634901970799547, + "p95_s": 0.26394581352651586, + "p99_s": 0.2643103066837648, + "min_s": 0.2618118610116653, + "max_s": 0.26440142997307703 + }, + "attn_output_reshape": { + "count": 10, + "mean_s": 4.099193029105664e-06, + "p50_s": 4.024012014269829e-06, + "p90_s": 4.249595804139972e-06, + "p95_s": 4.508779966272413e-06, + "p99_s": 4.716127295978368e-06, + "min_s": 3.840017598122358e-06, + "max_s": 4.767964128404856e-06 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.086404806002974e-06, + "p50_s": 4.064000677317381e-06, + "p90_s": 4.27677296102047e-06, + "p95_s": 4.370388342067599e-06, + "p99_s": 4.445280646905303e-06, + "min_s": 3.9199949242174625e-06, + "max_s": 4.464003723114729e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 8.564791642129421e-06, + "p50_s": 8.360017091035843e-06, + "p90_s": 8.947186870500445e-06, + "p95_s": 9.249590220861137e-06, + "p99_s": 9.49151290114969e-06, + "min_s": 8.192029781639576e-06, + "max_s": 9.551993571221828e-06 + }, + "linear.attn_out_proj.bias_cast": { + "count": 10, + "mean_s": 3.399985143914819e-06, + "p50_s": 3.3919932320713997e-06, + "p90_s": 3.489578375592828e-06, + "p95_s": 3.4967815736308694e-06, + "p99_s": 3.5025441320613027e-06, + "min_s": 3.263994585722685e-06, + "max_s": 3.503984771668911e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 10, + "mean_s": 0.002312088292092085, + "p50_s": 0.0023111284826882184, + "p90_s": 0.0023206335958093406, + "p95_s": 0.0023220447939820586, + "p99_s": 0.002323173752520233, + "min_s": 0.0022997119813226163, + "max_s": 0.002323455992154777 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 10, + "mean_s": 9.87689709290862e-06, + "p50_s": 9.71200643107295e-06, + "p90_s": 1.0727980406954884e-05, + "p95_s": 1.0835975990630686e-05, + "p99_s": 1.0922372457571327e-05, + "min_s": 9.34401759877801e-06, + "max_s": 1.0943971574306488e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.0028904856008011848, + "p50_s": 0.0028887049993500113, + "p90_s": 0.00291029930813238, + "p95_s": 0.002919062168803066, + "p99_s": 0.0029260724573396146, + "min_s": 0.002867089002393186, + "max_s": 0.002927825029473752 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.348802708089352e-06, + "p50_s": 6.263988325372338e-06, + "p90_s": 6.524822674691677e-06, + "p95_s": 6.726407445967197e-06, + "p99_s": 6.887675262987614e-06, + "min_s": 5.999987479299307e-06, + "max_s": 6.927992217242718e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 10, + "mean_s": 0.007921349199023098, + "p50_s": 0.00793309000437148, + "p90_s": 0.007959806796861812, + "p95_s": 0.007965408396557905, + "p99_s": 0.007969889676314778, + "min_s": 0.007834289979655296, + "max_s": 0.007971009996253997 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 10, + "mean_s": 5.4496107622981075e-06, + "p50_s": 5.368026904761791e-06, + "p90_s": 5.625589983537793e-06, + "p95_s": 5.740814958699047e-06, + "p99_s": 5.832994938828052e-06, + "min_s": 5.311972927302122e-06, + "max_s": 5.856039933860302e-06 + }, + "attn_out_proj": { + "count": 10, + "mean_s": 0.01320089278742671, + "p50_s": 0.013211563491495326, + "p90_s": 0.013256034994265066, + "p95_s": 0.01325833899609279, + "p99_s": 0.01326018219755497, + "min_s": 0.013092691951896995, + "max_s": 0.013260642997920513 + }, + "gate_msa": { + "count": 10, + "mean_s": 0.008729595987824723, + "p50_s": 0.00866641046013683, + "p90_s": 0.008878554130205885, + "p95_s": 0.008967942543677054, + "p99_s": 0.00903945327445399, + "min_s": 0.008640034007839859, + "max_s": 0.009057330957148224 + }, + "norm2": { + "count": 10, + "mean_s": 0.003563264914555475, + "p50_s": 0.0035559765237849206, + "p90_s": 0.0035853242268785836, + "p95_s": 0.0036025826178956773, + "p99_s": 0.0036163893307093534, + "min_s": 0.003548705019056797, + "max_s": 0.003619841008912772 + }, + "modulate_mlp": { + "count": 10, + "mean_s": 0.010596700402675196, + "p50_s": 0.010504154517548159, + "p90_s": 0.011042852600803598, + "p95_s": 0.01105070777994115, + "p99_s": 0.01105699192325119, + "min_s": 0.01029996998840943, + "max_s": 0.0110585629590787 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 10, + "mean_s": 4.132796311751008e-06, + "p50_s": 4.064029781147838e-06, + "p90_s": 4.33918321505189e-06, + "p95_s": 4.5695807784795754e-06, + "p99_s": 4.7538988292217255e-06, + "min_s": 3.936002030968666e-06, + "max_s": 4.799978341907263e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 10, + "mean_s": 7.998401997610927e-06, + "p50_s": 7.86400050856173e-06, + "p90_s": 8.32958030514419e-06, + "p95_s": 8.876787615008652e-06, + "p99_s": 9.314553462900221e-06, + "min_s": 7.3919654823839664e-06, + "max_s": 9.423994924873114e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 10, + "mean_s": 3.448012284934521e-06, + "p50_s": 3.4080003388226032e-06, + "p90_s": 3.6080251447856423e-06, + "p95_s": 3.6440149415284394e-06, + "p99_s": 3.6728067789226773e-06, + "min_s": 3.360037226229906e-06, + "max_s": 3.6800047382712364e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 10, + "mean_s": 0.0017999590025283396, + "p50_s": 0.001780224498361349, + "p90_s": 0.0018680152890738098, + "p95_s": 0.0018741996405879037, + "p99_s": 0.0018791471217991784, + "min_s": 0.0017290239920839667, + "max_s": 0.0018803839921019971 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 10, + "mean_s": 1.1595204705372453e-05, + "p50_s": 9.039998985826969e-06, + "p90_s": 1.3588822912424795e-05, + "p95_s": 2.3762410273775437e-05, + "p99_s": 3.1901280162855987e-05, + "min_s": 8.27200710773468e-06, + "max_s": 3.3935997635126114e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 10, + "mean_s": 0.002238086803117767, + "p50_s": 0.002225264994194731, + "p90_s": 0.0023277984117157755, + "p95_s": 0.0023365392175037415, + "p99_s": 0.002343531862134114, + "min_s": 0.002152128960005939, + "max_s": 0.002345280023291707 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 10, + "mean_s": 6.2112114392220976e-06, + "p50_s": 6.256013875827193e-06, + "p90_s": 6.483227480202913e-06, + "p95_s": 6.4976338762789965e-06, + "p99_s": 6.509158993139863e-06, + "min_s": 5.839974619448185e-06, + "max_s": 6.51204027235508e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 10, + "mean_s": 0.03520601270138286, + "p50_s": 0.0351380895008333, + "p90_s": 0.03551860520965419, + "p95_s": 0.035628779590479095, + "p99_s": 0.03571691909513902, + "min_s": 0.03475844202330336, + "max_s": 0.035738953971304 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 10, + "mean_s": 5.545601015910506e-06, + "p50_s": 5.504000000655651e-06, + "p90_s": 5.809619324281812e-06, + "p95_s": 5.816822522319853e-06, + "p99_s": 5.822585080750287e-06, + "min_s": 5.2480027079582214e-06, + "max_s": 5.824025720357895e-06 + }, + "mlp_fc1": { + "count": 10, + "mean_s": 0.03932547768927179, + "p50_s": 0.03926424199016765, + "p90_s": 0.03974540139315649, + "p95_s": 0.03980256220092997, + "p99_s": 0.03984829084714874, + "min_s": 0.03897794696968049, + "max_s": 0.03985972300870344 + }, + "mlp_swiglu": { + "count": 10, + "mean_s": 0.02429507830529474, + "p50_s": 0.024255126510979608, + "p90_s": 0.024437257228419183, + "p95_s": 0.024547863617772236, + "p99_s": 0.02463634872925468, + "min_s": 0.024114214000292122, + "max_s": 0.024658470007125288 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 10, + "mean_s": 4.006398376077413e-06, + "p50_s": 3.951980033889413e-06, + "p90_s": 4.184030694887042e-06, + "p95_s": 4.364005872048437e-06, + "p99_s": 4.507986013777554e-06, + "min_s": 3.824010491371155e-06, + "max_s": 4.543981049209833e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 10, + "mean_s": 7.497583283111453e-06, + "p50_s": 7.327966159209609e-06, + "p90_s": 8.059164974838495e-06, + "p95_s": 8.18159314803779e-06, + "p99_s": 8.279535686597228e-06, + "min_s": 7.167982403188944e-06, + "max_s": 8.304021321237087e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 10, + "mean_s": 3.446405753493309e-06, + "p50_s": 3.4240074455738068e-06, + "p90_s": 3.515236312523484e-06, + "p95_s": 3.565606311894953e-06, + "p99_s": 3.6059023113921285e-06, + "min_s": 3.3599790185689926e-06, + "max_s": 3.6159763112664223e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 10, + "mean_s": 0.004662481398554519, + "p50_s": 0.0046446814958471805, + "p90_s": 0.004776477930136025, + "p95_s": 0.004784527962328866, + "p99_s": 0.004790967988083139, + "min_s": 0.004539761983323842, + "max_s": 0.004792577994521707 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 10, + "mean_s": 1.0712019866332412e-05, + "p50_s": 9.048002539202571e-06, + "p90_s": 1.111680176109075e-05, + "p95_s": 1.866240054368971e-05, + "p99_s": 2.4698879569768907e-05, + "min_s": 8.144008461385965e-06, + "max_s": 2.62079993262887e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 10, + "mean_s": 0.005886719806585461, + "p50_s": 0.005925089004449546, + "p90_s": 0.005984025914222002, + "p95_s": 0.0060072454623877995, + "p99_s": 0.0060258211009204385, + "min_s": 0.005692336999345571, + "max_s": 0.006030465010553598 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 10, + "mean_s": 6.070389645174146e-06, + "p50_s": 6.079993909224868e-06, + "p90_s": 6.268831202760339e-06, + "p95_s": 6.398410187102855e-06, + "p99_s": 6.502073374576867e-06, + "min_s": 5.743990186601877e-06, + "max_s": 6.52798917144537e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 10, + "mean_s": 0.05435659209615551, + "p50_s": 0.05434621399035677, + "p90_s": 0.05462489890633151, + "p95_s": 0.05483235294814222, + "p99_s": 0.05499831618159078, + "min_s": 0.053905503009445965, + "max_s": 0.05503980698995292 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 10, + "mean_s": 5.41119952686131e-06, + "p50_s": 5.4480042308568954e-06, + "p90_s": 5.521601997315884e-06, + "p95_s": 5.60081098228693e-06, + "p99_s": 5.664178170263767e-06, + "min_s": 5.199981387704611e-06, + "max_s": 5.6800199672579765e-06 + }, + "mlp_fc2": { + "count": 10, + "mean_s": 0.06498188290861436, + "p50_s": 0.06497353702434339, + "p90_s": 0.06526047951192596, + "p95_s": 0.06543104073789437, + "p99_s": 0.0655674897186691, + "min_s": 0.06431148899719119, + "max_s": 0.06560160196386278 + }, + "gate_mlp": { + "count": 10, + "mean_s": 0.0101405818015337, + "p50_s": 0.008795610017841682, + "p90_s": 0.010404527210630472, + "p95_s": 0.016100218589417625, + "p99_s": 0.020656771692447368, + "min_s": 0.008680481987539679, + "max_s": 0.021795909968204796 + }, + "block_total": { + "count": 10, + "mean_s": 0.4943276042060461, + "p50_s": 0.492947026505135, + "p90_s": 0.4960365816077683, + "p95_s": 0.5006749098101864, + "p99_s": 0.5043855723721208, + "min_s": 0.4925488180015236, + "max_s": 0.5053132380126044 + } + }, + "module_forward": { + "count": 10, + "mean_s": 0.45834548950660975, + "p50_s": 0.45877519852365367, + "p90_s": 0.45995565400226046, + "p95_s": 0.46014774998766367, + "p99_s": 0.46030142677598634, + "min_s": 0.4553232230246067, + "max_s": 0.460339845973067 + }, + "module_forward_checksum": -895169280.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 2, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 16019156992 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1341.0399999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1330.4159999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 793.8730000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 646.9770000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 321.59999999999945, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 231.9360000000006, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 212.3199999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 206.91200000000026, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 200.52799999999934, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::contiguous", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 198.5599999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "aten::empty", + "count": 88, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 198.20800000000554, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 16019122176, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::clone", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 186.3679999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 178.73599999999988, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 177.96800000000076, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 135.51999999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 128.43200000000184, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 124.544000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [3, 5376], []]" + }, + { + "key": "aten::mul", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 90.81600000000162, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 4096, + "input_shapes": "[[], []]" + }, + { + "key": "aten::mean", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 84.9280000000008, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 28672, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 78.8159999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 73.10400000000027, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 58.36799999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::to", + "count": 18, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 51.45600000000195, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 45.904000000000906, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::fill_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 41.69599999999991, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 33.58399999999983, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty_like", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.839999999999918, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [], [], [], [], []]" + }, + { + "key": "aten::split", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.471999999999753, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], []]" + }, + { + "key": "aten::narrow", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.60799999999881, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.5600000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.487999999999829, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::reshape", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.78399999999965, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.67199999999957, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 12.128000000000156, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 12.064000000001215, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.840000000000146, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::squeeze", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.672000000000025, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.544000000001688, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::slice", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.920000000001437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.119999999999891, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.3119999999995855, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.688000000001466, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::view", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.623999999999796, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.608000000000175, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.303999999999178, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.079999999999927, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.983999999999469, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.2640000000001237, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.1359999999995125, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.087999999999738, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.207999999999629, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7759999999989304, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7600000000002183, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6640000000002146, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128]]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.551999999999225, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.4399999999995998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::alias", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3600000000014916, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.360000000000582, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.0080000000007203, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.975999999999658, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8320000000003347, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8160000000002583, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7039999999997235, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6560000000008586, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6080000000001746, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.5600000000004002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "[memory]", + "count": 74, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -16018849792, + "input_shapes": "[]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-fully-fused-block24-fresh.nsys-rep b/benchmarks/gb10-fully-fused-block24-fresh.nsys-rep new file mode 100644 index 0000000..3262f6e Binary files /dev/null and b/benchmarks/gb10-fully-fused-block24-fresh.nsys-rep differ diff --git a/benchmarks/gb10-fully-fused-block24-profile.json b/benchmarks/gb10-fully-fused-block24-profile.json new file mode 100644 index 0000000..16e9781 --- /dev/null +++ b/benchmarks/gb10-fully-fused-block24-profile.json @@ -0,0 +1,1292 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 10, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 10, + "mean_s": 0.003536642901599407, + "p50_s": 0.0034976669994648546, + "p90_s": 0.0035808175161946565, + "p95_s": 0.003707458774442784, + "p99_s": 0.003808771781041287, + "min_s": 0.0034645950072444975, + "max_s": 0.0038341000326909125 + }, + "modulate_msa": { + "count": 10, + "mean_s": 0.010376643878407777, + "p50_s": 0.01049173646606505, + "p90_s": 0.010621935367817059, + "p95_s": 0.010627220183960163, + "p99_s": 0.010631448036874644, + "min_s": 0.01000498398207128, + "max_s": 0.010632505000103265 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.6912056859582664e-06, + "p50_s": 4.264031304046512e-06, + "p90_s": 4.956789780408142e-06, + "p95_s": 6.742397090420122e-06, + "p99_s": 8.170882938429714e-06, + "min_s": 4.0800077840685844e-06, + "max_s": 8.52800440043211e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 1.067199045792222e-05, + "p50_s": 8.656003046780825e-06, + "p90_s": 1.1094362707808606e-05, + "p95_s": 2.0699159358628073e-05, + "p99_s": 2.838299667928368e-05, + "min_s": 8.01600981503725e-06, + "max_s": 3.0303956009447575e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 10, + "mean_s": 3.587285755202174e-06, + "p50_s": 3.54400253854692e-06, + "p90_s": 3.7352961953729388e-06, + "p95_s": 3.907622885890305e-06, + "p99_s": 4.045484238304198e-06, + "min_s": 3.375986125320196e-06, + "max_s": 4.079949576407671e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 10, + "mean_s": 0.001664134208112955, + "p50_s": 0.0016452169802505523, + "p90_s": 0.0017439107294194402, + "p95_s": 0.0017465958721004427, + "p99_s": 0.0017487439862452446, + "min_s": 0.0016137940110638738, + "max_s": 0.0017492810147814453 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 10, + "mean_s": 1.0867195669561625e-05, + "p50_s": 9.648007107898593e-06, + "p90_s": 1.2599985348060724e-05, + "p95_s": 1.580399402882903e-05, + "p99_s": 1.836720097344369e-05, + "min_s": 9.232026059180498e-06, + "max_s": 1.900800270959735e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.0021940851060207934, + "p50_s": 0.002150969987269491, + "p90_s": 0.00229719239869155, + "p95_s": 0.0023544612020486968, + "p99_s": 0.002400276244734414, + "min_s": 0.0021200490300543606, + "max_s": 0.0024117300054058433 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.774382200092077e-06, + "p50_s": 6.360001862049103e-06, + "p90_s": 7.612811168655753e-06, + "p95_s": 8.750418783165512e-06, + "p99_s": 9.660504874773325e-06, + "min_s": 6.175949238240719e-06, + "max_s": 9.888026397675276e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 10, + "mean_s": 0.02523876219638623, + "p50_s": 0.0252655814983882, + "p90_s": 0.02598772282944992, + "p95_s": 0.026068960392149166, + "p99_s": 0.02613395044230856, + "min_s": 0.024300965014845133, + "max_s": 0.02615019795484841 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 10, + "mean_s": 5.790404975414276e-06, + "p50_s": 5.527981556952e-06, + "p90_s": 5.9999991208314885e-06, + "p95_s": 7.080007344484326e-06, + "p99_s": 7.9440139234066e-06, + "min_s": 5.311972927302122e-06, + "max_s": 8.160015568137169e-06 + }, + "attn_qkv_proj": { + "count": 10, + "mean_s": 0.0291815816948656, + "p50_s": 0.029319992987439036, + "p90_s": 0.02994130760198459, + "p95_s": 0.029952906793914734, + "p99_s": 0.02996218614745885, + "min_s": 0.028194951999466866, + "max_s": 0.02996450598584488 + }, + "attn_qkv_split_view": { + "count": 10, + "mean_s": 1.3921584468334913e-05, + "p50_s": 1.2927979696542025e-05, + "p90_s": 1.685120514594018e-05, + "p95_s": 2.0033578039146953e-05, + "p99_s": 2.257947635371238e-05, + "min_s": 1.1119991540908813e-05, + "max_s": 2.3215950932353735e-05 + }, + "attn_qk_rms_rope": { + "count": 10, + "mean_s": 0.012174421816598623, + "p50_s": 0.01226081102504395, + "p90_s": 0.01244935499271378, + "p95_s": 0.012490539002465084, + "p99_s": 0.012523486210266128, + "min_s": 0.011791994038503617, + "max_s": 0.01253172301221639 + }, + "attention_kernel": { + "count": 10, + "mean_s": 0.2819813063077163, + "p50_s": 0.25794798001879826, + "p90_s": 0.3529046454816125, + "p95_s": 0.3818615222349762, + "p99_s": 0.40502702363766735, + "min_s": 0.25711314799264073, + "max_s": 0.41081839898834005 + }, + "attn_output_reshape": { + "count": 10, + "mean_s": 4.5040040276944636e-06, + "p50_s": 4.416011506691575e-06, + "p90_s": 4.788796650245785e-06, + "p95_s": 5.170408985577523e-06, + "p99_s": 5.475698853842915e-06, + "min_s": 4.064000677317381e-06, + "max_s": 5.552021320909262e-06 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.352000541985035e-06, + "p50_s": 4.296016413718462e-06, + "p90_s": 4.497583722695708e-06, + "p95_s": 4.864789661951362e-06, + "p99_s": 5.1585544133558866e-06, + "min_s": 4.064000677317381e-06, + "max_s": 5.231995601207018e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 9.830406634137035e-06, + "p50_s": 9.512004908174276e-06, + "p90_s": 1.074080355465412e-05, + "p95_s": 1.1842395178973672e-05, + "p99_s": 1.2723668478429319e-05, + "min_s": 8.383998647332191e-06, + "max_s": 1.2943986803293228e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 10, + "mean_s": 3.606401151046157e-06, + "p50_s": 3.6240089684724808e-06, + "p90_s": 3.7360237911343575e-06, + "p95_s": 3.772013587877154e-06, + "p99_s": 3.8008054252713917e-06, + "min_s": 3.4240074455738068e-06, + "max_s": 3.8080033846199512e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 10, + "mean_s": 0.0021395267103798687, + "p50_s": 0.0021357620134949684, + "p90_s": 0.0021502772287931293, + "p95_s": 0.002160587624530308, + "p99_s": 0.002168835941120051, + "min_s": 0.002125058032106608, + "max_s": 0.0021708980202674866 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 10, + "mean_s": 1.1635292321443557e-05, + "p50_s": 1.0199990356341004e-05, + "p90_s": 1.3747194316238162e-05, + "p95_s": 1.786559005267917e-05, + "p99_s": 2.1160306641831997e-05, + "min_s": 9.40798781812191e-06, + "max_s": 2.1983985789120197e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.002853197598597035, + "p50_s": 0.0028407784993760288, + "p90_s": 0.0028848740214016287, + "p95_s": 0.002903846013941802, + "p99_s": 0.002919023607973941, + "min_s": 0.0028314109658822417, + "max_s": 0.0029228180064819753 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.711995229125023e-06, + "p50_s": 6.647984264418483e-06, + "p90_s": 7.030414417386055e-06, + "p95_s": 7.275218376889824e-06, + "p99_s": 7.471061544492841e-06, + "min_s": 6.320013199001551e-06, + "max_s": 7.520022336393595e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 10, + "mean_s": 0.007781081803841516, + "p50_s": 0.007702854520175606, + "p90_s": 0.007810105266980827, + "p95_s": 0.008148016140330581, + "p99_s": 0.008418344839010387, + "min_s": 0.007674487016629428, + "max_s": 0.008485927013680339 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 10, + "mean_s": 5.652802065014839e-06, + "p50_s": 5.6480057537555695e-06, + "p90_s": 5.96162280999124e-06, + "p95_s": 6.004815804772079e-06, + "p99_s": 6.03937020059675e-06, + "min_s": 5.295965820550919e-06, + "max_s": 6.0480087995529175e-06 + }, + "attn_out_proj": { + "count": 10, + "mean_s": 0.012856716598616914, + "p50_s": 0.012750906986184418, + "p90_s": 0.012935022206511349, + "p95_s": 0.013290932588279247, + "p99_s": 0.013575660893693565, + "min_s": 0.012729579000733793, + "max_s": 0.013646842970047146 + }, + "gate_msa": { + "count": 10, + "mean_s": 0.008448739902814851, + "p50_s": 0.00843202299438417, + "p90_s": 0.008597137464676052, + "p95_s": 0.008597972249845043, + "p99_s": 0.008598640077980235, + "min_s": 0.008352135017048568, + "max_s": 0.008598807035014033 + }, + "norm2": { + "count": 10, + "mean_s": 0.0034746349963825198, + "p50_s": 0.0034624270047061145, + "p90_s": 0.003538892592769116, + "p95_s": 0.0035413837991654876, + "p99_s": 0.003543376764282584, + "min_s": 0.0034497790038585663, + "max_s": 0.0035438750055618584 + }, + "modulate_mlp": { + "count": 10, + "mean_s": 0.010302344901720061, + "p50_s": 0.0102713450032752, + "p90_s": 0.010716759436763824, + "p95_s": 0.010722224228084089, + "p99_s": 0.010726596061140299, + "min_s": 0.009972664003726095, + "max_s": 0.010727689019404352 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 10, + "mean_s": 4.3264124542474745e-06, + "p50_s": 4.25602775067091e-06, + "p90_s": 4.518410423770547e-06, + "p95_s": 4.691208596341312e-06, + "p99_s": 4.829447134397924e-06, + "min_s": 4.096014890819788e-06, + "max_s": 4.864006768912077e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 10, + "mean_s": 8.79999715834856e-06, + "p50_s": 8.639995940029621e-06, + "p90_s": 9.411235805600881e-06, + "p95_s": 1.0073615703731773e-05, + "p99_s": 1.0603519622236491e-05, + "min_s": 8.03195871412754e-06, + "max_s": 1.0735995601862669e-05 + }, + "linear.mlp_fc1.bias_cast": { + "count": 10, + "mean_s": 3.5087927244603633e-06, + "p50_s": 3.5119883250445127e-06, + "p90_s": 3.6495912354439497e-06, + "p95_s": 3.6567944334819913e-06, + "p99_s": 3.6625569919124246e-06, + "min_s": 3.3599790185689926e-06, + "max_s": 3.663997631520033e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 10, + "mean_s": 0.0016255454160273076, + "p50_s": 0.0016229460015892982, + "p90_s": 0.0016354218998458237, + "p95_s": 0.001644551966455765, + "p99_s": 0.001651856019743718, + "min_s": 0.0016106250113807619, + "max_s": 0.0016536820330657065 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 10, + "mean_s": 9.807990863919258e-06, + "p50_s": 9.544019121676683e-06, + "p90_s": 1.0929576819762588e-05, + "p95_s": 1.1368762352503835e-05, + "p99_s": 1.1720110778696835e-05, + "min_s": 8.719973266124725e-06, + "max_s": 1.1807947885245085e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 10, + "mean_s": 0.002176851505646482, + "p50_s": 0.0021523534960579127, + "p90_s": 0.0022936724068131297, + "p95_s": 0.0022943492193007843, + "p99_s": 0.0022948906692909074, + "min_s": 0.002131377987097949, + "max_s": 0.0022950260317884386 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 10, + "mean_s": 6.500800373032689e-06, + "p50_s": 6.4159976318478584e-06, + "p90_s": 6.894406396895647e-06, + "p95_s": 7.175200153142213e-06, + "p99_s": 7.399835158139467e-06, + "min_s": 5.856039933860302e-06, + "max_s": 7.455993909388781e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 10, + "mean_s": 0.03409755709581077, + "p50_s": 0.03422809299081564, + "p90_s": 0.03452030192129314, + "p95_s": 0.03457689344068058, + "p99_s": 0.03462216665619053, + "min_s": 0.03334671602351591, + "max_s": 0.03463348496006802 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 10, + "mean_s": 5.587207851931453e-06, + "p50_s": 5.592009983956814e-06, + "p90_s": 5.843234248459339e-06, + "p95_s": 5.857640644535422e-06, + "p99_s": 5.869165761396289e-06, + "min_s": 5.296024028211832e-06, + "max_s": 5.8720470406115055e-06 + }, + "mlp_fc1": { + "count": 10, + "mean_s": 0.037983797304332254, + "p50_s": 0.038089904526714236, + "p90_s": 0.03837030659778975, + "p95_s": 0.03842020980373491, + "p99_s": 0.03846013236849103, + "min_s": 0.03735007898649201, + "max_s": 0.03847011300968006 + }, + "mlp_swiglu": { + "count": 10, + "mean_s": 0.02382851129514165, + "p50_s": 0.023903323977719992, + "p90_s": 0.02399368488113396, + "p95_s": 0.02401542845473159, + "p99_s": 0.0240328233136097, + "min_s": 0.023509315971750766, + "max_s": 0.024037172028329223 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 10, + "mean_s": 4.140805685892701e-06, + "p50_s": 4.088011337444186e-06, + "p90_s": 4.2416038922965525e-06, + "p95_s": 4.392792470753193e-06, + "p99_s": 4.513743333518505e-06, + "min_s": 4.01597935706377e-06, + "max_s": 4.543981049209833e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 10, + "mean_s": 8.01280839368701e-06, + "p50_s": 7.863971404731274e-06, + "p90_s": 8.312013233080506e-06, + "p95_s": 8.996002725325523e-06, + "p99_s": 9.54319431912154e-06, + "min_s": 7.552036549896002e-06, + "max_s": 9.679992217570543e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 10, + "mean_s": 3.507203655317426e-06, + "p50_s": 3.5119883250445127e-06, + "p90_s": 3.576010931283235e-06, + "p95_s": 3.6120007280260323e-06, + "p99_s": 3.64079256542027e-06, + "min_s": 3.4080003388226032e-06, + "max_s": 3.6479905247688293e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 10, + "mean_s": 0.004368299985071644, + "p50_s": 0.004383971972856671, + "p90_s": 0.004443581588566304, + "p95_s": 0.004447440773947164, + "p99_s": 0.004450528122251853, + "min_s": 0.0042894280049949884, + "max_s": 0.004451299959328026 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 10, + "mean_s": 9.504001354798675e-06, + "p50_s": 9.360024705529213e-06, + "p90_s": 1.0358402505517006e-05, + "p95_s": 1.0963209206238387e-05, + "p99_s": 1.1447054566815497e-05, + "min_s": 8.463975973427296e-06, + "max_s": 1.1568015906959772e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 10, + "mean_s": 0.005924072209745645, + "p50_s": 0.005892501038033515, + "p90_s": 0.006118534551933408, + "p95_s": 0.006128189770970493, + "p99_s": 0.006135913946200162, + "min_s": 0.005749636969994754, + "max_s": 0.006137844990007579 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 10, + "mean_s": 6.358395330607891e-06, + "p50_s": 6.351969204843044e-06, + "p90_s": 6.553635466843843e-06, + "p95_s": 6.74081384204328e-06, + "p99_s": 6.89055654220283e-06, + "min_s": 5.871988832950592e-06, + "max_s": 6.927992217242718e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 10, + "mean_s": 0.05625267179566436, + "p50_s": 0.05627037599333562, + "p90_s": 0.05668957601301372, + "p95_s": 0.05669458000920713, + "p99_s": 0.056698583206161855, + "min_s": 0.055724094971083105, + "max_s": 0.05669958400540054 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 10, + "mean_s": 5.647999932989478e-06, + "p50_s": 5.527981556952e-06, + "p90_s": 5.806365516036748e-06, + "p95_s": 6.375182420015334e-06, + "p99_s": 6.830235943198204e-06, + "min_s": 5.296024028211832e-06, + "max_s": 6.943999323993921e-06 + }, + "mlp_fc2": { + "count": 10, + "mean_s": 0.06662256080890075, + "p50_s": 0.06662932099425234, + "p90_s": 0.06703189701656811, + "p95_s": 0.06705320902692619, + "p99_s": 0.06707025863521267, + "min_s": 0.0659938960452564, + "max_s": 0.06707452103728428 + }, + "gate_mlp": { + "count": 10, + "mean_s": 0.01003243929008022, + "p50_s": 0.008418151992373168, + "p90_s": 0.010617988393642003, + "p95_s": 0.017215404182206823, + "p99_s": 0.022493336813058706, + "min_s": 0.008402118983212858, + "max_s": 0.02381281997077167 + }, + "block_total": { + "count": 10, + "mean_s": 0.510889746597968, + "p50_s": 0.48557983900536783, + "p90_s": 0.5824499189096968, + "p95_s": 0.6108186564553761, + "p99_s": 0.6335136464919197, + "min_s": 0.483186541008763, + "max_s": 0.6391873940010555 + } + }, + "module_forward": { + "count": 10, + "mean_s": 0.4524989926023409, + "p50_s": 0.4523211540072225, + "p90_s": 0.4534270477830432, + "p95_s": 0.4535232613969129, + "p99_s": 0.4536002322880086, + "min_s": 0.4518661299953237, + "max_s": 0.4536194750107825 + }, + "module_forward_checksum": -23765568.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 8009578496 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1857.8570000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1847.825, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 966.9929999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 728.1770000000006, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "aten::contiguous", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 447.7280000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "aten::clone", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 438.51199999999926, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 437.3929999999996, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 44, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 406.1919999999975, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009561088, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 388.7049999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 314.2399999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 111.39199999999983, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [3, 5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 109.80799999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 96.46399999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 92.30400000000009, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 88.17599999999948, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 83.29599999999937, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 81.71200000000044, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 80.55999999999949, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 76.44799999999941, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 58.944000000001324, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "aten::_to_copy", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.712000000000444, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 33.90399999999954, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::split", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 32.55999999999949, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 30.048999999999978, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::to", + "count": 9, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 27.99999999999909, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::empty_like", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 26.623999999999796, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[3, 5376], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 23.200000000000728, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::narrow", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.592000000000553, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.384000000000015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.1279999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::squeeze", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.936000000000604, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], []]" + }, + { + "key": "aten::reshape", + "count": 5, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.615999999999985, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.351999999999862, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.288000000000011, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810], []]" + }, + { + "key": "aten::empty_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.608000000001084, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 1024, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.527999999999338, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.431999999999789, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::slice", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.11200000000008, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.711999999999534, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.520000000000437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.519999999999413, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376], [], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.1840000000001965, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.6479999999992287, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.6879999999991924, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 5, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.496000000001004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.4639999999990323, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.319999999999709, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.9520000000002256, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.5679999999993015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.4880000000002838, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2960000000002765, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1520000000000437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.0399999999999636, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8640000000004875, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8000000000001819, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7839999999996508, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128]]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7200000000002547, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.5280000000002474, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.4639999999999418, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.44800000000032014, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.431999999999789, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.431999999999789, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.38400000000001455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.35199999999986176, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.28800000000046566, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.28799999999955617, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "[memory]", + "count": 36, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -8009273344, + "input_shapes": "[]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-fully-fused-fresh-nsight-summary.json b/benchmarks/gb10-fully-fused-fresh-nsight-summary.json new file mode 100644 index 0000000..5a4a0cf --- /dev/null +++ b/benchmarks/gb10-fully-fused-fresh-nsight-summary.json @@ -0,0 +1,125 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "block": 24, + "attention": "sage2", + "seed": 440420 + }, + "configuration": { + "H3_NVFP4_SCALE_BACKEND": "vortex", + "H3_NVFP4_SCALE_VERSION": "1", + "H3_FUSED_ELEMENTWISE": "1", + "H3_NVFP4_MODULATE_FUSION": "1", + "H3_NVFP4_SWIGLU_FUSION": "1", + "H3_SAGE_QKV_LAYOUT": "strided_nhd" + }, + "block_24": { + "uninstrumented_module_p50_ms": 458.7751985236537, + "nsight_gpu_span_ms": 465.779616, + "kernel_time_ms": 465.695712, + "kernel_launches": 41, + "inter_kernel_idle_ms": 0.083904, + "average_inter_kernel_idle_us": 2.098, + "maximum_inter_kernel_idle_us": 3.36, + "components": { + "sage2": {"milliseconds": 268.224256, "percent": 57.59646247290334}, + "nvfp4_gemms": {"milliseconds": 125.113088, "percent": 26.865844966165376}, + "nvfp4_packing": {"milliseconds": 41.526688, "percent": 8.917129131736562}, + "norm_and_rope": {"milliseconds": 19.863552, "percent": 4.265349988878575}, + "remaining_gate_add": {"milliseconds": 10.940192, "percent": 2.349214673464719}, + "other": {"milliseconds": 0.027936, "percent": 0.005998732851428} + }, + "producer_detail": { + "modulated_qkv_and_fc1_ms": 13.041792, + "swiglu_fc2_ms": 23.043104, + "attention_output_pack_ms": 5.126112, + "scale_finalize_and_zero_fill_ms": 0.31568, + "standalone_modulation_kernels": 0, + "standalone_swiglu_kernels": 0, + "gate_add_kernels": 2 + }, + "allocation_and_conversion": { + "positive_self_allocated_bytes_per_block": 8009578496, + "aten_empty_calls_per_block": 44, + "large_layout_copy_kernels": 0, + "explicit_cuda_memcpy_ops": 0, + "remaining_contiguous_calls_per_block": 4, + "remaining_contiguous_shape": [3, 5376], + "remaining_to_copy_calls_per_block": 2, + "note": "Remaining contiguous and to-copy operations are small AdaLN/scalar conversions, not full activation layout materializations." + } + }, + "one_warmed_sampling_step": { + "elapsed_seconds": 23.707971603027545, + "gpu_span_seconds": 23.706977664, + "kernel_time_seconds": 23.693982688, + "kernel_launches": 2694, + "inter_kernel_idle_ms": 12.994976, + "average_inter_kernel_idle_us": 4.825, + "maximum_inter_kernel_idle_ms": 6.395168, + "peak_allocated_bytes": 15570401280, + "peak_reserved_bytes": 18538823680, + "explicit_cuda_memcpy_ops": 9, + "explicit_cuda_memcpy_bytes": 15282118, + "explicit_cuda_memcpy_ms": 0.169856, + "components": { + "sage2": {"milliseconds": 13760.566304, "percent": 58.07620645797613}, + "nvfp4_gemms": {"milliseconds": 6269.804512, "percent": 26.46158982455656}, + "nvfp4_packing": {"milliseconds": 2093.052672, "percent": 8.833688702997335}, + "norm_and_rope": {"milliseconds": 987.609696, "percent": 4.168187801117043}, + "remaining_gate_add": {"milliseconds": 545.150976, "percent": 2.3007992500817345}, + "other": {"milliseconds": 37.798528, "percent": 0.159528965271231} + }, + "gpu_metrics": { + "gr_active_average_percent": 99.57, + "sms_active_average_percent": 99.27, + "sm_issue_average_percent": 25.63, + "tensor_active_average_percent": 53.72 + } + }, + "memory_traffic": { + "measurement_note": "GB10 Nsight exposes no direct physical DRAM-byte counter. L2 requested bytes are measured with lts__t_bytes.sum. The off-chip proxy is 32 bytes times warm-cache L2 read/write miss sectors with Nsight Compute cache control disabled.", + "l2_requested_bytes": { + "total": 320998821280, + "sage2": 166126147104, + "nvfp4_gemms": 132957604576, + "nvfp4_packing": 14638661888, + "norm_and_rope": 4802883392, + "remaining_gate_add": 2472452288 + }, + "off_chip_request_proxy_bytes": { + "total": 42116618176, + "sage2": 6357343328, + "nvfp4_gemms": 23690363008, + "nvfp4_packing": 7685899232, + "norm_and_rope": 2753536064, + "remaining_gate_add": 1629476544 + }, + "off_chip_request_proxy_percent": { + "sage2": 15.094619661610697, + "nvfp4_gemms": 56.249442699793654, + "nvfp4_packing": 18.249089230957726, + "norm_and_rope": 6.537885004188424, + "remaining_gate_add": 3.868963403449499 + } + }, + "decision": { + "time_bottleneck": "Sage2, led by the 246.629 ms attention mainloop in block 24.", + "memory_traffic_bottleneck": "The four NVFP4 GEMMs, with 56.25% of the warm-cache off-chip request proxy.", + "cpu_launch_bottleneck": false, + "next_kernel": "No implementation target is selected solely from the old profile. Any next target must use this fresh split between Sage2 compute time and NVFP4 GEMM memory traffic." + }, + "artifacts": [ + "benchmarks/gb10-fully-fused-block24-fresh.nsys-rep", + "benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep", + "benchmarks/gb10-fully-fused-block24-fresh-capture.json", + "benchmarks/gb10-fully-fused-block24-fresh-torch.json", + "benchmarks/gb10-fully-fused-one-step-fresh.json", + "benchmarks/gb10-fully-fused-block24-fresh-l2.csv", + "benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv" + ] +} diff --git a/benchmarks/gb10-fully-fused-one-step-fresh.json b/benchmarks/gb10-fully-fused-one-step-fresh.json new file mode 100644 index 0000000..3773329 --- /dev/null +++ b/benchmarks/gb10-fully-fused-one-step-fresh.json @@ -0,0 +1,24 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 1, + "seed": 440420, + "text_tokens": 100, + "warmup_runs": 1, + "cuda_profiler_capture": true, + "elapsed_seconds": 23.707971603027545, + "stage_trace": [], + "checksums": [ + -197605.3125, + 528.5980224609375 + ], + "peak_allocated_bytes": 15570401280, + "peak_reserved_bytes": 18538823680, + "measurement_policy": "uninstrumented production sampling wall time inside a whole-run NVTX range" +} diff --git a/benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep b/benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep new file mode 100644 index 0000000..3c49b4c Binary files /dev/null and b/benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep differ diff --git a/benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json b/benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json new file mode 100644 index 0000000..bf0c2a5 --- /dev/null +++ b/benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json @@ -0,0 +1,1532 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 5, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 5, + "mean_s": 0.00370076420949772, + "p50_s": 0.0036445120349526405, + "p90_s": 0.003916181833483279, + "p95_s": 0.004001954442355782, + "p99_s": 0.004070572529453784, + "min_s": 0.00354526296723634, + "max_s": 0.004087727051228285 + }, + "modulate_msa": { + "count": 5, + "mean_s": 0.010961576015688479, + "p50_s": 0.010747502034064382, + "p90_s": 0.01159105401020497, + "p95_s": 0.011800702009350062, + "p99_s": 0.011968420408666134, + "min_s": 0.010485582984983921, + "max_s": 0.012010350008495152 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 5, + "mean_s": 5.1711918786168095e-06, + "p50_s": 4.815985448658466e-06, + "p90_s": 6.588792894035578e-06, + "p95_s": 7.094396278262138e-06, + "p99_s": 7.4988789856433865e-06, + "min_s": 4.031986463814974e-06, + "max_s": 7.599999662488699e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 5, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 5, + "mean_s": 1.361599424853921e-05, + "p50_s": 9.135983418673277e-06, + "p90_s": 2.324479864910245e-05, + "p95_s": 2.690240507945418e-05, + "p99_s": 2.982849022373557e-05, + "min_s": 8.032016921788454e-06, + "max_s": 3.056001150980592e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 5, + "mean_s": 3.5776058211922647e-06, + "p50_s": 3.5199918784201145e-06, + "p90_s": 3.856001421809197e-06, + "p95_s": 3.9359903894364836e-06, + "p99_s": 3.999981563538312e-06, + "min_s": 3.296008799225092e-06, + "max_s": 4.01597935706377e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 5, + "mean_s": 0.0017301246174611152, + "p50_s": 0.0017411670414730906, + "p90_s": 0.00177472640061751, + "p95_s": 0.0017832832061685621, + "p99_s": 0.0017901286506094038, + "min_s": 0.0016634880448691547, + "max_s": 0.0017918400117196143 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 5, + "mean_s": 1.5584006905555726e-05, + "p50_s": 1.7023994587361813e-05, + "p90_s": 1.9052799325436352e-05, + "p95_s": 1.94304040633142e-05, + "p99_s": 1.9732487853616476e-05, + "min_s": 1.1184019967913628e-05, + "max_s": 1.9808008801192045e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 5, + "mean_s": 0.002322953403927386, + "p50_s": 0.0022722400026395917, + "p90_s": 0.002535033598542213, + "p95_s": 0.0025973887997679415, + "p99_s": 0.0026472729607485233, + "min_s": 0.0021412789938040078, + "max_s": 0.002659744000993669 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 5, + "mean_s": 6.976001895964146e-06, + "p50_s": 7.071997970342636e-06, + "p90_s": 7.951981388032436e-06, + "p95_s": 8.207978680729866e-06, + "p99_s": 8.412776514887811e-06, + "min_s": 6.0480087995529175e-06, + "max_s": 8.463975973427296e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 5, + "mean_s": 0.03267693240195513, + "p50_s": 0.02632073196582496, + "p90_s": 0.04515579420840368, + "p95_s": 0.0506150125991553, + "p99_s": 0.054982387311756614, + "min_s": 0.025907867995556444, + "max_s": 0.05607423098990694 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 5, + "mean_s": 6.319989915937186e-06, + "p50_s": 6.608024705201387e-06, + "p90_s": 7.007969543337822e-06, + "p95_s": 7.135968189686537e-06, + "p99_s": 7.2383671067655085e-06, + "min_s": 5.231995601207018e-06, + "max_s": 7.263966836035252e-06 + }, + "attn_qkv_proj": { + "count": 5, + "mean_s": 0.03682621520711109, + "p50_s": 0.030483131005894393, + "p90_s": 0.04947682461934165, + "p95_s": 0.0550504078157246, + "p99_s": 0.05950927437283098, + "min_s": 0.029794570989906788, + "max_s": 0.06062399101210758 + }, + "attn_qkv_split_view": { + "count": 5, + "mean_s": 1.826880034059286e-05, + "p50_s": 2.0159990526735783e-05, + "p90_s": 2.2012810222804547e-05, + "p95_s": 2.2102415096014737e-05, + "p99_s": 2.217409899458289e-05, + "min_s": 1.3535958714783192e-05, + "max_s": 2.219201996922493e-05 + }, + "attn_qk_rms_rope": { + "count": 5, + "mean_s": 0.012522334605455399, + "p50_s": 0.01253089401870966, + "p90_s": 0.01304144160822034, + "p95_s": 0.013201063813176005, + "p99_s": 0.01332876157714054, + "min_s": 0.011644751008134335, + "max_s": 0.013360686018131673 + }, + "attn_q_transpose_contiguous": { + "count": 5, + "mean_s": 0.010179851192515344, + "p50_s": 0.00496414297958836, + "p90_s": 0.020518390822689983, + "p95_s": 0.02548829743172973, + "p99_s": 0.029464222718961538, + "min_s": 0.004930094990413636, + "max_s": 0.030458204040769488 + }, + "attn_k_transpose_contiguous": { + "count": 5, + "mean_s": 0.010719105391763151, + "p50_s": 0.004932318988721818, + "p90_s": 0.022215538995806132, + "p95_s": 0.027844894991721953, + "p99_s": 0.032348379788454625, + "min_s": 0.004929247021209449, + "max_s": 0.03347425098763779 + }, + "attn_v_transpose_contiguous": { + "count": 5, + "mean_s": 0.008646603603847326, + "p50_s": 0.005328848026692867, + "p90_s": 0.01569883298361674, + "p95_s": 0.01915314298821613, + "p99_s": 0.021916590991895645, + "min_s": 0.004975487012416124, + "max_s": 0.022607452992815524 + }, + "attention_kernel": { + "count": 5, + "mean_s": 0.26568923201411965, + "p50_s": 0.2572870340081863, + "p90_s": 0.28179349241545426, + "p95_s": 0.28840694121317934, + "p99_s": 0.2936977002513595, + "min_s": 0.2569375950261019, + "max_s": 0.2950203900109045 + }, + "attn_output_reshape": { + "count": 5, + "mean_s": 0.004784690390806645, + "p50_s": 0.004718751995824277, + "p90_s": 0.004893067781813443, + "p95_s": 0.004894261376466601, + "p99_s": 0.00489521625218913, + "min_s": 0.004707070998847485, + "max_s": 0.0048954549711197615 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 5, + "mean_s": 4.870397970080376e-06, + "p50_s": 4.5120250433683395e-06, + "p90_s": 5.759997293353081e-06, + "p95_s": 5.999987479299307e-06, + "p99_s": 6.1919796280562874e-06, + "min_s": 4.239962436258793e-06, + "max_s": 6.239977665245533e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 5, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 5, + "mean_s": 1.5878397971391677e-05, + "p50_s": 1.0415969882160425e-05, + "p90_s": 2.6598421391099693e-05, + "p95_s": 3.005122998729348e-05, + "p99_s": 3.2813476864248514e-05, + "min_s": 9.215960744768381e-06, + "max_s": 3.350403858348727e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 5, + "mean_s": 3.6063836887478827e-06, + "p50_s": 3.439956344664097e-06, + "p90_s": 3.920029848814011e-06, + "p95_s": 4.032033029943705e-06, + "p99_s": 4.121635574847459e-06, + "min_s": 3.4239492379128933e-06, + "max_s": 4.144036211073399e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 5, + "mean_s": 0.0022977533983066676, + "p50_s": 0.002296319988090545, + "p90_s": 0.0023102719802409414, + "p95_s": 0.0023118719807825984, + "p99_s": 0.002313151981215924, + "min_s": 0.002284880029037595, + "max_s": 0.0023134719813242555 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 5, + "mean_s": 1.2838398106396198e-05, + "p50_s": 1.1263997294008732e-05, + "p90_s": 1.6198388766497374e-05, + "p95_s": 1.7027172725647687e-05, + "p99_s": 1.769019989296794e-05, + "min_s": 1.0271964129060507e-05, + "max_s": 1.7855956684798002e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 5, + "mean_s": 0.002899941592477262, + "p50_s": 0.002914238953962922, + "p90_s": 0.002939288597553968, + "p95_s": 0.002945931791327894, + "p99_s": 0.002951246346347034, + "min_s": 0.002855567028746009, + "max_s": 0.002952574985101819 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 5, + "mean_s": 7.366400677710772e-06, + "p50_s": 6.735965143889189e-06, + "p90_s": 8.963211439549922e-06, + "p95_s": 9.4336224719882e-06, + "p99_s": 9.809951297938823e-06, + "min_s": 6.176007445901632e-06, + "max_s": 9.90403350442648e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 5, + "mean_s": 0.008071339805610478, + "p50_s": 0.007845567015465349, + "p90_s": 0.008501621428877115, + "p95_s": 0.008666610228829086, + "p99_s": 0.008798601268790661, + "min_s": 0.00783475098432973, + "max_s": 0.008831599028781056 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 5, + "mean_s": 5.833595059812069e-06, + "p50_s": 5.615991540253162e-06, + "p90_s": 6.400002166628838e-06, + "p95_s": 6.608001422137022e-06, + "p99_s": 6.77440082654357e-06, + "min_s": 5.424022674560547e-06, + "max_s": 6.8160006776452065e-06 + }, + "attn_out_proj": { + "count": 5, + "mean_s": 0.013364516384899616, + "p50_s": 0.013153581996448338, + "p90_s": 0.01383550518658012, + "p95_s": 0.013990103593096138, + "p99_s": 0.014113782318308949, + "min_s": 0.013065422012005001, + "max_s": 0.014144701999612153 + }, + "gate_msa": { + "count": 5, + "mean_s": 0.008748206798918546, + "p50_s": 0.008709535002708435, + "p90_s": 0.008930900797713548, + "p95_s": 0.008934161404613405, + "p99_s": 0.00893676989013329, + "min_s": 0.008583375019952655, + "max_s": 0.008937422011513263 + }, + "norm2": { + "count": 5, + "mean_s": 0.0035379068111069502, + "p50_s": 0.0035068950382992625, + "p90_s": 0.003613071620929986, + "p95_s": 0.0036341758328489963, + "p99_s": 0.003651059202384204, + "min_s": 0.003484688000753522, + "max_s": 0.0036552800447680056 + }, + "modulate_mlp": { + "count": 5, + "mean_s": 0.010690554999746383, + "p50_s": 0.010643246001563966, + "p90_s": 0.011026267206761985, + "p95_s": 0.011032692599110307, + "p99_s": 0.011037832912988961, + "min_s": 0.010254477965645492, + "max_s": 0.011039117991458625 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 5, + "mean_s": 4.796788562089205e-06, + "p50_s": 4.767964128404856e-06, + "p90_s": 5.372788291424513e-06, + "p95_s": 5.574396345764399e-06, + "p99_s": 5.735682789236307e-06, + "min_s": 4.127970896661282e-06, + "max_s": 5.776004400104284e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 5, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 5, + "mean_s": 8.927995804697274e-06, + "p50_s": 8.736038580536842e-06, + "p90_s": 9.878387209028005e-06, + "p95_s": 1.0147178545594216e-05, + "p99_s": 1.0362211614847183e-05, + "min_s": 7.96798849478364e-06, + "max_s": 1.0415969882160425e-05 + }, + "linear.mlp_fc1.bias_cast": { + "count": 5, + "mean_s": 3.6192010156810285e-06, + "p50_s": 3.6800047382712364e-06, + "p90_s": 3.6992132663726807e-06, + "p95_s": 3.705616109073162e-06, + "p99_s": 3.710738383233547e-06, + "min_s": 3.375986125320196e-06, + "max_s": 3.7120189517736435e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 5, + "mean_s": 0.0017399997916072607, + "p50_s": 0.0017451830208301544, + "p90_s": 0.0017586175817996264, + "p95_s": 0.0017593887750990689, + "p99_s": 0.0017600057297386229, + "min_s": 0.001707071962300688, + "max_s": 0.0017601599683985114 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 5, + "mean_s": 1.1795188765972852e-05, + "p50_s": 1.0351999662816525e-05, + "p90_s": 1.6096001490950585e-05, + "p95_s": 1.7888005822896956e-05, + "p99_s": 1.9321609288454058e-05, + "min_s": 9.007984772324562e-06, + "max_s": 1.968001015484333e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 5, + "mean_s": 0.0022628605947829783, + "p50_s": 0.0021891199867241085, + "p90_s": 0.002411609189584851, + "p95_s": 0.002436860592570156, + "p99_s": 0.0024570617149583993, + "min_s": 0.002152992004994303, + "max_s": 0.0024621119955554605 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 5, + "mean_s": 6.5312138758599755e-06, + "p50_s": 6.560003384947777e-06, + "p90_s": 6.992032285779714e-06, + "p95_s": 7.072032894939184e-06, + "p99_s": 7.13603338226676e-06, + "min_s": 6.095971912145615e-06, + "max_s": 7.152033504098654e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 5, + "mean_s": 0.0716201048111543, + "p50_s": 0.07167574996128678, + "p90_s": 0.07178817201638595, + "p95_s": 0.07180112900678068, + "p99_s": 0.07181149459909647, + "min_s": 0.07133280503330752, + "max_s": 0.07181408599717543 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 5, + "mean_s": 5.792011506855488e-06, + "p50_s": 5.6800199672579765e-06, + "p90_s": 6.160023622214794e-06, + "p95_s": 6.192026194185018e-06, + "p99_s": 6.217628251761198e-06, + "min_s": 5.4559786804020405e-06, + "max_s": 6.224028766155243e-06 + }, + "mlp_fc1": { + "count": 5, + "mean_s": 0.07571263879071921, + "p50_s": 0.07571077399188653, + "p90_s": 0.07579628340899944, + "p95_s": 0.07580330419586971, + "p99_s": 0.07580892082536593, + "min_s": 0.07558842096477747, + "max_s": 0.07581032498273998 + }, + "mlp_swiglu": { + "count": 5, + "mean_s": 0.024767270009033383, + "p50_s": 0.024613836023490876, + "p90_s": 0.025068361801095305, + "p95_s": 0.025185571413021534, + "p99_s": 0.025279339102562515, + "min_s": 0.024597676005214453, + "max_s": 0.025302781024947762 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 5, + "mean_s": 4.336016718298197e-06, + "p50_s": 4.096014890819788e-06, + "p90_s": 4.9184192903339865e-06, + "p95_s": 5.123228766024112e-06, + "p99_s": 5.287076346576214e-06, + "min_s": 3.936002030968666e-06, + "max_s": 5.328038241714239e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 5, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 5, + "mean_s": 8.10560304671526e-06, + "p50_s": 7.872004061937332e-06, + "p90_s": 8.53120582178235e-06, + "p95_s": 8.553615771234035e-06, + "p99_s": 8.571543730795383e-06, + "min_s": 7.760012522339821e-06, + "max_s": 8.57602572068572e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 5, + "mean_s": 3.6800047382712364e-06, + "p50_s": 3.503984771668911e-06, + "p90_s": 4.086398985236883e-06, + "p95_s": 4.259194247424602e-06, + "p99_s": 4.397430457174778e-06, + "min_s": 3.4240074455738068e-06, + "max_s": 4.431989509612322e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 5, + "mean_s": 0.004378968803212047, + "p50_s": 0.004356223973445594, + "p90_s": 0.0044430166250094775, + "p95_s": 0.004468475829344243, + "p99_s": 0.0044888431928120556, + "min_s": 0.004333471006248146, + "max_s": 0.0044939350336790085 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 5, + "mean_s": 9.596801828593015e-06, + "p50_s": 9.232026059180498e-06, + "p90_s": 1.0777614079415798e-05, + "p95_s": 1.1180818546563387e-05, + "p99_s": 1.1503382120281458e-05, + "min_s": 8.751987479627132e-06, + "max_s": 1.1584023013710976e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 5, + "mean_s": 0.0058938198140822354, + "p50_s": 0.0058150230324827135, + "p90_s": 0.006084485398605466, + "p95_s": 0.006107090192381292, + "p99_s": 0.006125174027401954, + "min_s": 0.005752607015892863, + "max_s": 0.006129694986157119 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 5, + "mean_s": 6.204796954989434e-06, + "p50_s": 6.191956344991922e-06, + "p90_s": 6.3615967519581315e-06, + "p95_s": 6.396800745278597e-06, + "p99_s": 6.4249639399349686e-06, + "min_s": 6.0480087995529175e-06, + "max_s": 6.432004738599062e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 5, + "mean_s": 0.053530597186181694, + "p50_s": 0.05350962496595457, + "p90_s": 0.05387938078492879, + "p95_s": 0.05388907039305195, + "p99_s": 0.053896822079550474, + "min_s": 0.0530537529848516, + "max_s": 0.053898760001175106 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 5, + "mean_s": 5.484803114086389e-06, + "p50_s": 5.4879928939044476e-06, + "p90_s": 5.6319870054721825e-06, + "p95_s": 5.679985042661429e-06, + "p99_s": 5.718383472412825e-06, + "min_s": 5.328038241714239e-06, + "max_s": 5.727983079850674e-06 + }, + "mlp_fc2": { + "count": 5, + "mean_s": 0.06388071699766443, + "p50_s": 0.06395071098813787, + "p90_s": 0.06436533080413936, + "p95_s": 0.0644813884049654, + "p99_s": 0.06457423448562623, + "min_s": 0.06331373500870541, + "max_s": 0.06459744600579143 + }, + "gate_mlp": { + "count": 5, + "mean_s": 0.011803047789726406, + "p50_s": 0.009327662992291152, + "p90_s": 0.017453504574950787, + "p95_s": 0.02013837477425113, + "p99_s": 0.02228627093369141, + "min_s": 0.008719118020962924, + "max_s": 0.022823244973551482 + }, + "block_total": { + "count": 5, + "mean_s": 0.5766410392010585, + "p50_s": 0.5561805270262994, + "p90_s": 0.6259831115952693, + "p95_s": 0.6395874358015136, + "p99_s": 0.6504708951665089, + "min_s": 0.543343793018721, + "max_s": 0.6531917600077577 + } + }, + "module_forward": { + "count": 5, + "mean_s": 0.5217330385930836, + "p50_s": 0.5216766290250234, + "p90_s": 0.5225755279883743, + "p95_s": 0.5228595179971307, + "p99_s": 0.523086710004136, + "min_s": 0.5204838289646432, + "max_s": 0.5231435080058873 + }, + "module_forward_checksum": 333782528.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 2, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 22523446272 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1042.591999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -606208, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1023.7440000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1021.6320000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 644.1119999999996, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "aten::zeros", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 518.3519999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 499.7760000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 465.85599999999977, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::zero_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 430.86400000000094, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::contiguous", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 420.71900000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], []]" + }, + { + "key": "aten::clone", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 409.96700000000055, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], []]" + }, + { + "key": "aten::_to_copy", + "count": 16, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 297.3599999999992, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 257.03999999999905, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "aten::empty", + "count": 76, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 220.49600000000498, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 20354950144, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 219.36000000000058, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 216.04799999999977, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 209.45600000000013, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "aten::to", + "count": 24, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 203.599999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 16, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 199.64800000000014, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 192.84800000000087, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 180.46399999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -1024, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::to", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 119.04000000000133, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 116.27199999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "aten::mul", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 111.32800000000043, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 4096, + "input_shapes": "[[], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 108.91200000000026, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], [1, 56, 37810, 128], [1, 56, 128, 37824], [1, 56, 37810, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::mean", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 104.86400000000049, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 28672, + "input_shapes": "[[1, 56, 37810, 128], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 93.5039999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 77.1359999999986, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], [1, 56, 37810, 128], []]" + }, + { + "key": "aten::silu", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 74.30399999999827, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2168455168, + "input_shapes": "[[37810, 14336]]" + }, + { + "key": "aten::fill_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 54.400000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::mul_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.34399999999914, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [37810, 14336]]" + }, + { + "key": "aten::empty_strided", + "count": 16, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.072000000001935, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8192, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 47.35999999999967, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::split", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 38.35199999999941, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], []]" + }, + { + "key": "aten::empty_like", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 28.943999999999505, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], [], [], [], [], []]" + }, + { + "key": "aten::clone", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 26.35200000000077, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::narrow", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 22.192000000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.54400000000078, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::chunk", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.43199999999979, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::reshape", + "count": 16, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 19.968000000000302, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::transpose", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.064000000000306, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.871999999999844, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.391999999999825, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.31999999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.984000000000378, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::reshape", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.423999999998841, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::split", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.072000000001935, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.495999999998276, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::narrow", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.720000000000255, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::squeeze", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.256000000000313, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 1, 128], []]" + }, + { + "key": "aten::slice", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.95199999999977, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.791999999999007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::slice", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.752000000001317, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::empty_like", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.11200000000008, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.0479999999990355, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.000000000000455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810], []]" + }, + { + "key": "aten::slice", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.839999999999236, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.584000000000287, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 16, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.359999999999218, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.104000000001179, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.191999999999098, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.11200000000008, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.967999999999847, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::as_strided", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.936000000001968, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.8239999999998417, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::transpose", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4559999999992215, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.119999999999891, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.768000000000029, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.4799999999995634, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.2880000000004657, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::_unsafe_view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.256000000000313, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.1279999999997017, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128]]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.063999999999851, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6800000000012005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.359999999998763, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::alias", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.295999999999367, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2479999999995925, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1359999999976935, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.087999999999738, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8159999999998035, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 37810, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7999999999997272, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7520000000004075, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7520000000004075, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 56, 1, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7200000000011642, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7039999999997235, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.5600000000004002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.543999999999869, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.5119999999997162, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "[memory]", + "count": 74, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -22522835968, + "input_shapes": "[]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json b/benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json new file mode 100644 index 0000000..b90402f --- /dev/null +++ b/benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json @@ -0,0 +1,77 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "packed_tokens": 37810, + "steps": 12, + "seed": 440420, + "feature_flag": "H3_FUSED_ELEMENTWISE=1", + "isolated_cuda_tests": { + "modulation_bit_exact": true, + "gate_add_bit_exact": true + }, + "real_block_results": [ + { + "block": 0, + "eager_p50_seconds": 0.5083600514917634, + "fused_p50_seconds": 0.4901100995193701, + "speedup": 1.0372364331816264, + "latency_reduction": 0.03589965796651318, + "max_abs_difference": 0.0, + "checksum": -8279139.0 + }, + { + "block": 24, + "eager_p50_seconds": 0.5089578525221441, + "fused_p50_seconds": 0.4891547074948903, + "speedup": 1.040484420826024, + "latency_reduction": 0.03890920422804989, + "max_abs_difference": 0.0, + "checksum": 89071408.0 + }, + { + "block": 49, + "eager_p50_seconds": 0.5089952205016743, + "fused_p50_seconds": 0.4891957634827122, + "speedup": 1.0404734842305352, + "latency_reduction": 0.03889910203763303, + "max_abs_difference": 0.0, + "checksum": -40617144320.0 + } + ], + "sampling_12_step": { + "eager_seconds": 310.109976747015, + "fused_seconds": 301.04568115097936, + "speedup": 1.0301093693202321, + "latency_reduction": 0.029229293720628102, + "eager_checksums": [ + -91481.7890625, + 678.401611328125 + ], + "fused_checksums": [ + -91481.7890625, + 678.401611328125 + ] + }, + "sampling_2_step_memory": { + "eager_peak_allocated_bytes": 19278355968, + "fused_peak_allocated_bytes": 18869530112, + "allocated_bytes_saved": 408825856, + "eager_peak_reserved_bytes": 21976055808, + "fused_peak_reserved_bytes": 21846032384, + "reserved_bytes_saved": 130023424 + }, + "validation": { + "deployed_tests": "32 passed", + "block_parity": "torch.equal at blocks 0, 24, and 49", + "sampling_parity": "identical video and audio checksums after 12 steps", + "generic_default": "eager path remains default unless H3_FUSED_ELEMENTWISE is enabled", + "spark_default": "enabled in Spark compose files" + } +} diff --git a/benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json b/benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json new file mode 100644 index 0000000..eddc89f --- /dev/null +++ b/benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json @@ -0,0 +1,9100 @@ +{ + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 10, + "profiler_iterations": 2, + "quantizers": [ + "comfy", + "vortex_recalculate", + "vortex_native" + ], + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "results": [ + { + "name": "attn_qkv_proj", + "quantizer": "comfy", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 21504, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 21504, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 21504, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 6.0496153309941295e-06, + "p50_s": 4.536035703495145e-06, + "p90_s": 9.27842338569462e-06, + "p95_s": 1.3015206786803892e-05, + "p99_s": 1.6004633507691324e-05, + "min_s": 3.936002030968666e-06, + "max_s": 1.675199018791318e-05 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 1.2190302368253469e-05, + "p50_s": 9.69597022049129e-06, + "p90_s": 1.4918379019945852e-05, + "p95_s": 2.5099195772781942e-05, + "p99_s": 3.324384917505086e-05, + "min_s": 7.95203959569335e-06, + "max_s": 3.5280012525618076e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 3.7487887311726807e-06, + "p50_s": 3.6879791878163815e-06, + "p90_s": 4.092749441042542e-06, + "p95_s": 4.22238081227988e-06, + "p99_s": 4.326085909269751e-06, + "min_s": 3.503984771668911e-06, + "max_s": 4.352012183517218e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.007912381808273495, + "p50_s": 0.007679425994865596, + "p90_s": 0.008507974818348885, + "p95_s": 0.00887430993025191, + "p99_s": 0.009167378019774334, + "min_s": 0.007318171032238752, + "max_s": 0.009240645042154938 + }, + "linear": { + "count": 10, + "mean_s": 0.07953937528654934, + "p50_s": 0.04432033549528569, + "p90_s": 0.1047707318968604, + "p95_s": 0.2624061949405583, + "p99_s": 0.3885145653755172, + "min_s": 0.023811994993593544, + "max_s": 0.42004165798425674 + }, + "slice_reshape": { + "count": 10, + "mean_s": 7.45270517654717e-06, + "p50_s": 6.400019628927112e-06, + "p90_s": 9.331200271844862e-06, + "p95_s": 1.2369587784633033e-05, + "p99_s": 1.4800297794863581e-05, + "min_s": 5.6480057537555695e-06, + "max_s": 1.5407975297421217e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 1626132480, + "separate_materialization_bytes": 2147045376, + "epilogue_candidate": "QKV views/layout and Q/K normalization/RoPE" + }, + "profiler_summary": { + "device_kernel_launches": 0, + "unique_device_kernels": 0, + "positive_self_device_allocated_bytes": 4297373696, + "negative_self_device_allocated_bytes": -4297373696, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_qkv_proj-comfy.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3996.745999999999, + "self_cpu_time_total_us": 814.9270000000015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3158.5549999999985, + "self_cpu_time_total_us": 3140.0429999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [0]]" + }, + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2818.124, + "self_cpu_time_total_us": 1045.0059999999976, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1944.1740000000027, + "self_cpu_time_total_us": 756.0640000000021, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 230834176, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1757.6620000000003, + "self_cpu_time_total_us": 1659.4219999999987, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1179.0390000000007, + "self_cpu_time_total_us": 1172.287000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1153.9020000000019, + "self_cpu_time_total_us": 24.9120000000039, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1122.221999999998, + "self_cpu_time_total_us": 1046.7179999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::amax", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 685.6779999999999, + "self_cpu_time_total_us": 680.2539999999972, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[37810, 5376], [], []]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 334.9119999999966, + "self_cpu_time_total_us": 17.247999999995955, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 317.66400000000067, + "self_cpu_time_total_us": 208.70400000000154, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::div", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 85.58400000000256, + "self_cpu_time_total_us": 85.58400000000256, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 84.95999999999913, + "self_cpu_time_total_us": 84.95999999999913, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 75.50400000000081, + "self_cpu_time_total_us": 75.50400000000081, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 67.07199999999648, + "self_cpu_time_total_us": 67.07199999999648, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "aten::empty", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 60.55999999999767, + "self_cpu_time_total_us": 60.55999999999767, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3484303360, + "self_device_memory_usage_bytes": 3484303360, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24.0, + "self_cpu_time_total_us": 24.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::resize_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.511999999998807, + "self_cpu_time_total_us": 18.511999999998807, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 813066240, + "flops": 0, + "input_shapes": "[[0], [], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.247999999999593, + "self_cpu_time_total_us": 12.175999999999476, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.45600000000195, + "self_cpu_time_total_us": 12.864000000005035, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.072000000003754, + "self_cpu_time_total_us": 15.072000000003754, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.7519999999985885, + "self_cpu_time_total_us": 6.7519999999985885, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.424000000002707, + "self_cpu_time_total_us": 5.424000000002707, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.072000000000116, + "self_cpu_time_total_us": 5.072000000000116, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.600000000002183, + "self_cpu_time_total_us": 3.600000000002183, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.1679999999978463, + "self_cpu_time_total_us": 1.8399999999965075, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.591999999996915, + "self_cpu_time_total_us": 2.591999999996915, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.5280000000020664, + "self_cpu_time_total_us": 2.5280000000020664, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.2079999999969004, + "self_cpu_time_total_us": 2.2079999999969004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3280000000013388, + "self_cpu_time_total_us": 1.3280000000013388, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "[memory]", + "count": 18, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -4297372672, + "self_device_memory_usage_bytes": -4297372672, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_qkv_proj", + "quantizer": "vortex_recalculate", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 21504, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 21504, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 21504, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 6.164796650409698e-06, + "p50_s": 5.215988494455814e-06, + "p90_s": 8.035183418542146e-06, + "p95_s": 8.409592555835842e-06, + "p99_s": 8.7091198656708e-06, + "min_s": 4.57599526271224e-06, + "max_s": 8.78400169312954e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 0.00016132930177263914, + "p50_s": 1.0535994078963995e-05, + "p90_s": 0.00017684289487078733, + "p95_s": 0.0008282039430923744, + "p99_s": 0.0013492927816696467, + "min_s": 8.000002708286047e-06, + "max_s": 0.0014795649913139641 + }, + "bias_cast": { + "count": 10, + "mean_s": 5.227205110713839e-06, + "p50_s": 4.3920299503952265e-06, + "p90_s": 5.606387276202437e-06, + "p95_s": 9.163195500150314e-06, + "p99_s": 1.200864207930863e-05, + "min_s": 4.096014890819788e-06, + "max_s": 1.2720003724098206e-05 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0016794118040706963, + "p50_s": 0.0016737655096221715, + "p90_s": 0.001700576167786494, + "p95_s": 0.0017489025805843992, + "p99_s": 0.0017875637108227239, + "min_s": 0.0016442370251752436, + "max_s": 0.001797228993382305 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 2.4196697631850837e-05, + "p50_s": 1.6175967175513506e-05, + "p90_s": 3.5485514672473066e-05, + "p95_s": 4.8294765292666825e-05, + "p99_s": 5.854216578882188e-05, + "min_s": 1.1936004739254713e-05, + "max_s": 6.110401591286063e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.4910398749634624e-05, + "p50_s": 1.364797935821116e-05, + "p90_s": 1.9180780509486794e-05, + "p95_s": 2.0174376550130544e-05, + "p99_s": 2.096925338264555e-05, + "min_s": 1.0959978681057692e-05, + "max_s": 2.1167972590774298e-05 + }, + "ck_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0032863131840713323, + "p50_s": 0.002224964991910383, + "p90_s": 0.0034175936831161344, + "p95_s": 0.00809333081706426, + "p99_s": 0.011833920524222778, + "min_s": 0.0021520599839277565, + "max_s": 0.012769067951012403 + }, + "params_wrap": { + "count": 10, + "mean_s": 1.0996783385053276e-05, + "p50_s": 8.327973773702979e-06, + "p90_s": 1.9836780847981568e-05, + "p95_s": 2.3206384503282598e-05, + "p99_s": 2.5902067427523434e-05, + "min_s": 6.83200778439641e-06, + "max_s": 2.657598815858364e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.0050613199127838016, + "p50_s": 0.003989738033851609, + "p90_s": 0.0053149498358834495, + "p95_s": 0.009906519434298378, + "p99_s": 0.013579775113030339, + "min_s": 0.003893512999638915, + "max_s": 0.014498089032713324 + }, + "linear": { + "count": 10, + "mean_s": 0.03375178879359737, + "p50_s": 0.034510504483478144, + "p90_s": 0.04217453677556478, + "p95_s": 0.04223313039110508, + "p99_s": 0.04228000528353732, + "min_s": 0.02445064898347482, + "max_s": 0.04229172400664538 + }, + "slice_reshape": { + "count": 10, + "mean_s": 8.408003486692905e-06, + "p50_s": 7.824011845514178e-06, + "p90_s": 1.0484817903488874e-05, + "p95_s": 1.08664040453732e-05, + "p99_s": 1.1171672958880663e-05, + "min_s": 6.607966497540474e-06, + "max_s": 1.1247990187257528e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 1626132480, + "separate_materialization_bytes": 2147045376, + "epilogue_candidate": "QKV views/layout and Q/K normalization/RoPE" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 3485617152, + "negative_self_device_allocated_bytes": -3485617152, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_qkv_proj-vortex_recalculate.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 525.5680000000002, + "self_cpu_time_total_us": 122.27200000000039, + "device_time_total_us": 50787.75, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3254779904, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 389.1359999999995, + "self_cpu_time_total_us": 292.04799999999886, + "device_time_total_us": 50787.75, + "self_device_time_total_us": 50783.78200000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3254779904, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 50783.78200000001, + "self_device_time_total_us": 50783.78200000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 518.3830000000003, + "self_cpu_time_total_us": 398.92799999999943, + "device_time_total_us": 4210.547999999997, + "self_device_time_total_us": 4165.332, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 230834176, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4165.332, + "self_device_time_total_us": 4165.332, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3297.079, + "self_device_time_total_us": 3297.079, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 62.128000000000156, + "self_cpu_time_total_us": 16.40000000000009, + "device_time_total_us": 45.21599999999671, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.952000000000226, + "self_cpu_time_total_us": 10.559999999999945, + "device_time_total_us": 45.21599999999671, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 29.39200000000028, + "self_cpu_time_total_us": 13.232000000000426, + "device_time_total_us": 45.21599999999671, + "self_device_time_total_us": 45.21599999999671, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 45.21599999999671, + "self_device_time_total_us": 45.21599999999671, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 188.20600000000013, + "self_cpu_time_total_us": 27.840000000000146, + "device_time_total_us": 5.503000000006523, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 110.94199999999955, + "self_cpu_time_total_us": 69.98299999999927, + "device_time_total_us": 5.503000000006523, + "self_device_time_total_us": 5.503000000006523, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.264000000000124, + "self_cpu_time_total_us": 22.463999999999942, + "device_time_total_us": 3.9679999999962092, + "self_device_time_total_us": 3.9679999999962092, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.9679999999962092, + "self_device_time_total_us": 3.9679999999962092, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.871999999997115, + "self_device_time_total_us": 3.871999999997115, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 36.751000000000204, + "self_cpu_time_total_us": 3.088000000000193, + "device_time_total_us": 2.9750000000008185, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9750000000008185, + "self_device_time_total_us": 2.9750000000008185, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 168.75099999999918, + "self_cpu_time_total_us": 14.20799999999906, + "device_time_total_us": 2.5280000000057044, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.5280000000057044, + "self_device_time_total_us": 2.5280000000057044, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 56534.981, + "self_cpu_time_total_us": 56534.981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1254.8939999999998, + "self_cpu_time_total_us": 1254.8939999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 209.16600000000062, + "self_cpu_time_total_us": 209.16600000000062, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.42400000000043, + "self_cpu_time_total_us": 49.42400000000043, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34.943999999999505, + "self_cpu_time_total_us": 34.943999999999505, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3485614080, + "self_device_memory_usage_bytes": 3485614080, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.12800000000061, + "self_cpu_time_total_us": 8.896000000000186, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.696000000000367, + "self_cpu_time_total_us": 17.696000000000367, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.65599999999995, + "self_cpu_time_total_us": 16.65599999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.16000000000031, + "self_cpu_time_total_us": 10.176000000000386, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.791999999999916, + "self_cpu_time_total_us": 8.415999999999826, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.232000000000426, + "self_cpu_time_total_us": 9.232000000000426, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.000000000000455, + "self_cpu_time_total_us": 8.000000000000455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.37600000000009, + "self_cpu_time_total_us": 5.37600000000009, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.9839999999999236, + "self_cpu_time_total_us": 3.9839999999999236, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.5359999999996035, + "self_cpu_time_total_us": 2.207999999999629, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.4720000000002074, + "self_cpu_time_total_us": 1.4720000000002074, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3600000000001273, + "self_cpu_time_total_us": 1.3600000000001273, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3279999999999745, + "self_cpu_time_total_us": 1.3279999999999745, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -3485616128, + "self_device_memory_usage_bytes": -3485616128, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_qkv_proj", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 21504, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 21504, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 21504, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.945604061707854e-06, + "p50_s": 5.079986294731498e-06, + "p90_s": 7.5663847383111715e-06, + "p95_s": 7.631213520653546e-06, + "p99_s": 7.683076546527445e-06, + "min_s": 4.688045009970665e-06, + "max_s": 7.69604230299592e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 1.253918744623661e-05, + "p50_s": 9.055976988747716e-06, + "p90_s": 1.8667196854948994e-05, + "p95_s": 2.426159335300325e-05, + "p99_s": 2.873711055144668e-05, + "min_s": 8.256000000983477e-06, + "max_s": 2.985598985105753e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.419207107275724e-06, + "p50_s": 4.344008630141616e-06, + "p90_s": 4.886416718363762e-06, + "p95_s": 4.915229510515928e-06, + "p99_s": 4.938279744237661e-06, + "min_s": 3.936002030968666e-06, + "max_s": 4.944042302668095e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.001660768303554505, + "p50_s": 0.0016563970129936934, + "p90_s": 0.0016738977981731295, + "p95_s": 0.0016784913954325021, + "p99_s": 0.00168216627324, + "min_s": 0.0016487810062244534, + "max_s": 0.0016830849926918745 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 0.0012001517054159195, + "p50_s": 1.4680001186206937e-05, + "p90_s": 0.0021871404955163564, + "p95_s": 0.006389521749224504, + "p99_s": 0.00975142675219104, + "min_s": 1.1808006092905998e-05, + "max_s": 0.010591903002932668 + }, + "scale_to_device": { + "count": 10, + "mean_s": 2.060000551864505e-05, + "p50_s": 1.736002741381526e-05, + "p90_s": 3.288962761871517e-05, + "p95_s": 3.667680721264331e-05, + "p99_s": 3.970655088778585e-05, + "min_s": 1.0911957360804081e-05, + "max_s": 4.0463986806571484e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0022807597066275775, + "p50_s": 0.0022093004954513162, + "p90_s": 0.002431947988225147, + "p95_s": 0.002448867986095138, + "p99_s": 0.0024624039843911304, + "min_s": 0.002181821037083864, + "max_s": 0.0024657879839651287 + }, + "params_wrap": { + "count": 10, + "mean_s": 1.1233595432713628e-05, + "p50_s": 7.1919639594852924e-06, + "p90_s": 2.6147201424464583e-05, + "p95_s": 2.702559868339449e-05, + "p99_s": 2.7728316490538417e-05, + "min_s": 6.783986464142799e-06, + "max_s": 2.79039959423244e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.005210591707145796, + "p50_s": 0.004148697014898062, + "p90_s": 0.006150649720802899, + "p95_s": 0.010348617378622284, + "p99_s": 0.013706991504877808, + "min_s": 0.0038902339874766767, + "max_s": 0.014546585036441684 + }, + "linear": { + "count": 10, + "mean_s": 0.026462141395313665, + "p50_s": 0.025728374515892938, + "p90_s": 0.027108474890701468, + "p95_s": 0.03137550492538138, + "p99_s": 0.034789128953125324, + "min_s": 0.024055432993918657, + "max_s": 0.03564253496006131 + }, + "slice_reshape": { + "count": 10, + "mean_s": 7.641606498509645e-06, + "p50_s": 7.071997970342636e-06, + "p90_s": 9.409559424966574e-06, + "p95_s": 9.77679155766964e-06, + "p99_s": 1.0070577263832093e-05, + "min_s": 6.384041626006365e-06, + "max_s": 1.0144023690372705e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 1626132480, + "separate_materialization_bytes": 2147045376, + "epilogue_candidate": "QKV views/layout and Q/K normalization/RoPE" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 3484306432, + "negative_self_device_allocated_bytes": -3484306432, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_qkv_proj-vortex_native.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 641.3119999999994, + "self_cpu_time_total_us": 113.00799999999936, + "device_time_total_us": 51407.981999999996, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 517.6480000000001, + "self_cpu_time_total_us": 422.4639999999981, + "device_time_total_us": 51407.981999999996, + "self_device_time_total_us": 51404.11, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 51404.11, + "self_device_time_total_us": 51404.11, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4352.473000000004, + "self_device_time_total_us": 4352.473000000004, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3293.211000000001, + "self_device_time_total_us": 3293.211000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 58.43199999999979, + "self_cpu_time_total_us": 11.695999999999458, + "device_time_total_us": 45.59900000000107, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 41.76000000000022, + "self_cpu_time_total_us": 13.728000000000065, + "device_time_total_us": 45.59900000000107, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 28.032000000000153, + "self_cpu_time_total_us": 13.071999999999662, + "device_time_total_us": 45.59900000000107, + "self_device_time_total_us": 45.59900000000107, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 45.59900000000107, + "self_device_time_total_us": 45.59900000000107, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 184.27199999999948, + "self_cpu_time_total_us": 35.91999999999939, + "device_time_total_us": 5.5359999999996035, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 80.25599999999986, + "self_cpu_time_total_us": 45.23199999999906, + "device_time_total_us": 5.5359999999996035, + "self_device_time_total_us": 5.5359999999996035, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.000000000004093, + "self_device_time_total_us": 4.000000000004093, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 36.48000000000002, + "self_cpu_time_total_us": 20.864000000000487, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 38.81600000000026, + "self_cpu_time_total_us": 2.9280000000003383, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 2.9760000000005675, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 162.99199999999917, + "self_cpu_time_total_us": 14.607999999999606, + "device_time_total_us": 2.559999999999036, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.559999999999036, + "self_device_time_total_us": 2.559999999999036, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 57783.187, + "self_cpu_time_total_us": 57783.187, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1088.8139999999999, + "self_cpu_time_total_us": 1088.8139999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 162.4160000000004, + "self_cpu_time_total_us": 162.4160000000004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 68.09600000000023, + "self_cpu_time_total_us": 68.09600000000023, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 35.36000000000058, + "self_cpu_time_total_us": 35.36000000000058, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3484303360, + "self_device_memory_usage_bytes": 3484303360, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.50400000000036, + "self_cpu_time_total_us": 17.50400000000036, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.024000000000342, + "self_cpu_time_total_us": 15.024000000000342, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.376000000000317, + "self_cpu_time_total_us": 8.1760000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.655999999999949, + "self_cpu_time_total_us": 7.983999999999924, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.135999999999967, + "self_cpu_time_total_us": 5.8400000000001455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.176000000000386, + "self_cpu_time_total_us": 6.176000000000386, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.200000000000017, + "self_cpu_time_total_us": 5.200000000000017, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.7599999999997635, + "self_cpu_time_total_us": 2.6559999999994943, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.2959999999998217, + "self_cpu_time_total_us": 3.2959999999998217, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.6720000000000255, + "self_cpu_time_total_us": 2.6720000000000255, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6960000000003674, + "self_cpu_time_total_us": 1.6960000000003674, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6640000000002146, + "self_cpu_time_total_us": 1.6640000000002146, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1040000000002692, + "self_cpu_time_total_us": 1.1040000000002692, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -3484305408, + "self_device_memory_usage_bytes": -3484305408, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_out_proj", + "quantizer": "comfy", + "module": { + "class": "Nvfp4Linear", + "in_features": 7168, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 3584 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 448 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 7168 + ], + "input_is_contiguous": true, + "input_stride": [ + 7168, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 4.9375870730727914e-06, + "p50_s": 4.639994585886598e-06, + "p90_s": 5.185569170862435e-06, + "p95_s": 6.4167659729719135e-06, + "p99_s": 7.4017234146595e-06, + "min_s": 4.496017936617136e-06, + "max_s": 7.647962775081396e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 7.991993334144354e-06, + "p50_s": 7.855996955186129e-06, + "p90_s": 8.633575635030866e-06, + "p95_s": 8.820780203677713e-06, + "p99_s": 8.970543858595192e-06, + "min_s": 7.520022336393595e-06, + "max_s": 9.007984772324562e-06 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.151801113039255e-06, + "p50_s": 4.120025550946593e-06, + "p90_s": 4.21190052293241e-06, + "p95_s": 4.3779407860711215e-06, + "p99_s": 4.510772996582091e-06, + "min_s": 4.047004040330648e-06, + "max_s": 4.543981049209833e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.009642405400518328, + "p50_s": 0.009569192508934066, + "p90_s": 0.009928448009304702, + "p95_s": 0.010009808000177145, + "p99_s": 0.010074895992875098, + "min_s": 0.009391825005877763, + "max_s": 0.010091167991049588 + }, + "linear": { + "count": 10, + "mean_s": 0.008343223499832676, + "p50_s": 0.008321458473801613, + "p90_s": 0.008729003596818075, + "p95_s": 0.008753238795907236, + "p99_s": 0.008772626955178565, + "min_s": 0.007807716028764844, + "max_s": 0.008777473994996399 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.433599628508091e-06, + "p50_s": 6.351998308673501e-06, + "p90_s": 6.64321705698967e-06, + "p95_s": 6.873614620417356e-06, + "p99_s": 7.057932671159506e-06, + "min_s": 6.143993232399225e-06, + "max_s": 7.104012183845043e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 7168 + ], + "input_bytes": 542044160, + "packed_qdata_bytes": 135561216, + "block_scale_bytes": 16945152, + "output_bytes": 406533120, + "separate_materialization_bytes": 1101083648, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 18, + "unique_device_kernels": 9, + "positive_self_device_allocated_bytes": 2202529792, + "negative_self_device_allocated_bytes": -2202529792, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_out_proj-comfy.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 693.0239999999999, + "self_cpu_time_total_us": 138.5279999999998, + "device_time_total_us": 17302.301, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 547.3919999999998, + "self_cpu_time_total_us": 401.4880000000003, + "device_time_total_us": 17302.301, + "self_device_time_total_us": 17298.301, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 17298.301, + "self_device_time_total_us": 17298.301, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1081.39, + "self_cpu_time_total_us": 9.600000000000364, + "device_time_total_us": 13704.701000000001, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1084088320, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168]]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1052.1739999999998, + "self_cpu_time_total_us": 43.455999999999676, + "device_time_total_us": 13704.701000000001, + "self_device_time_total_us": 9117.385, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1084088320, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [0]]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::AbsFunctor, std::array >(int, at::native::AbsFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 9117.385, + "self_device_time_total_us": 9117.385, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 592.575, + "self_cpu_time_total_us": 478.4950000000001, + "device_time_total_us": 5686.86, + "self_device_time_total_us": 5609.068, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 305070080, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 5609.068, + "self_device_time_total_us": 5609.068, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 946.062, + "self_cpu_time_total_us": 946.062, + "device_time_total_us": 4587.316000000001, + "self_device_time_total_us": 4587.316000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::amax", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 179.82400000000007, + "self_cpu_time_total_us": 84.25600000000009, + "device_time_total_us": 4413.910999999998, + "self_device_time_total_us": 4413.910999999998, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[37810, 7168], [], []]" + }, + { + "key": "void at::native::reduce_kernel<512, 1, at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4> >(at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4413.046999999997, + "self_device_time_total_us": 4413.046999999997, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 65.2159999999999, + "self_cpu_time_total_us": 13.135999999999967, + "device_time_total_us": 77.79199999999946, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 33947648, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 46.83199999999988, + "self_cpu_time_total_us": 9.375999999999749, + "device_time_total_us": 77.79199999999946, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.45600000000013, + "self_cpu_time_total_us": 15.10400000000027, + "device_time_total_us": 77.79199999999946, + "self_device_time_total_us": 77.79199999999946, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 77.79199999999946, + "self_device_time_total_us": 77.79199999999946, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 82.12800000000016, + "self_cpu_time_total_us": 25.184000000000196, + "device_time_total_us": 4.0, + "self_device_time_total_us": 4.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.0, + "self_device_time_total_us": 4.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::div", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 69.36000000000013, + "self_cpu_time_total_us": 43.96799999999985, + "device_time_total_us": 3.7119999999995343, + "self_device_time_total_us": 3.7119999999995343, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BUnaryFunctor >, std::array >(int, at::native::BUnaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.7119999999995343, + "self_device_time_total_us": 3.7119999999995343, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 105.87200000000007, + "self_cpu_time_total_us": 10.192000000000235, + "device_time_total_us": 2.4629999999988286, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 95.67999999999984, + "self_cpu_time_total_us": 21.695999999999913, + "device_time_total_us": 2.4629999999988286, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 63.67999999999961, + "self_cpu_time_total_us": 34.991999999999734, + "device_time_total_us": 2.4629999999988286, + "self_device_time_total_us": 2.4629999999988286, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.4629999999988286, + "self_device_time_total_us": 2.4629999999988286, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Memset (Device)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.864000000001397, + "self_device_time_total_us": 0.864000000001397, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34483.321, + "self_cpu_time_total_us": 34483.321, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 227.69599999999969, + "self_cpu_time_total_us": 227.69599999999969, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaMemsetAsync", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 61.4079999999999, + "self_cpu_time_total_us": 61.4079999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 46.09600000000046, + "self_cpu_time_total_us": 46.09600000000046, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1118437376, + "self_device_memory_usage_bytes": 1118437376, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24.032000000000153, + "self_cpu_time_total_us": 24.032000000000153, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::resize_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 23.408000000000072, + "self_cpu_time_total_us": 23.408000000000072, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1084088320, + "self_device_memory_usage_bytes": 1084088320, + "flops": 0, + "input_shapes": "[[0], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 21.55199999999968, + "self_cpu_time_total_us": 21.55199999999968, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.760000000000076, + "self_cpu_time_total_us": 8.65600000000029, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::empty_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.304000000000315, + "self_cpu_time_total_us": 10.304000000000315, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.399999999999181, + "self_cpu_time_total_us": 8.399999999999181, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.104000000000269, + "self_cpu_time_total_us": 5.871999999999844, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.816000000000258, + "self_cpu_time_total_us": 4.656000000000404, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.736000000000104, + "self_cpu_time_total_us": 6.736000000000104, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.103999999999786, + "self_cpu_time_total_us": 5.103999999999786, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.119999999999891, + "self_cpu_time_total_us": 2.4320000000002437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.9600000000000364, + "self_cpu_time_total_us": 2.9600000000000364, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.1599999999998545, + "self_cpu_time_total_us": 2.1599999999998545, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6640000000002146, + "self_cpu_time_total_us": 1.6640000000002146, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.536000000000115, + "self_cpu_time_total_us": 1.536000000000115, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2320000000004256, + "self_cpu_time_total_us": 1.2320000000004256, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6879999999996471, + "self_cpu_time_total_us": 0.6879999999996471, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 18, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -2202528768, + "self_device_memory_usage_bytes": -2202528768, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_out_proj", + "quantizer": "vortex_recalculate", + "module": { + "class": "Nvfp4Linear", + "in_features": 7168, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 3584 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 448 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 7168 + ], + "input_is_contiguous": true, + "input_stride": [ + 7168, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 6.518402369692922e-06, + "p50_s": 4.640023689717054e-06, + "p90_s": 8.460803655907508e-06, + "p95_s": 1.348640944343059e-05, + "p99_s": 1.7506894073449077e-05, + "min_s": 4.431989509612322e-06, + "max_s": 1.8512015230953693e-05 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.603202877566218e-06, + "p50_s": 8.216011337935925e-06, + "p90_s": 9.257608326151965e-06, + "p95_s": 1.0740812285803255e-05, + "p99_s": 1.1927375453524292e-05, + "min_s": 7.679976988583803e-06, + "max_s": 1.222401624545455e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.187191370874643e-06, + "p50_s": 4.10401844419539e-06, + "p90_s": 4.600000102072954e-06, + "p95_s": 4.635989898815752e-06, + "p99_s": 4.664781736209988e-06, + "min_s": 3.96795803681016e-06, + "max_s": 4.671979695558548e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0021354953991249205, + "p50_s": 0.0021257479675114155, + "p90_s": 0.0021507832396309825, + "p95_s": 0.0021839896420715378, + "p99_s": 0.002210554764023982, + "min_s": 0.002113804977852851, + "max_s": 0.002217196044512093 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 1.7780804773792626e-05, + "p50_s": 1.193603384308517e-05, + "p90_s": 1.879042829386888e-05, + "p95_s": 4.351523530203843e-05, + "p99_s": 6.329508090857418e-05, + "min_s": 1.1280004400759935e-05, + "max_s": 6.824004231020808e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.2153509305790067e-05, + "p50_s": 1.0727497283369303e-05, + "p90_s": 1.528642023913562e-05, + "p95_s": 1.841121993493288e-05, + "p99_s": 2.09110596915707e-05, + "min_s": 1.0400020983070135e-05, + "max_s": 2.1536019630730152e-05 + }, + "ck_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0028809011972043665, + "p50_s": 0.0028607480053324252, + "p90_s": 0.002952268690569326, + "p95_s": 0.002984316347283311, + "p99_s": 0.0030099544726544993, + "min_s": 0.00283020397182554, + "max_s": 0.003016364003997296 + }, + "params_wrap": { + "count": 10, + "mean_s": 8.480006363242864e-06, + "p50_s": 6.720016244798899e-06, + "p90_s": 9.32640978135168e-06, + "p95_s": 1.637519744690506e-05, + "p99_s": 2.2014227579347792e-05, + "min_s": 6.4639607444405556e-06, + "max_s": 2.3423985112458467e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.005091450706822797, + "p50_s": 0.005055840010754764, + "p90_s": 0.005223735910840332, + "p95_s": 0.005299623461905867, + "p99_s": 0.005360333502758294, + "min_s": 0.005004151025786996, + "max_s": 0.005375511012971401 + }, + "linear": { + "count": 10, + "mean_s": 0.008391089201904834, + "p50_s": 0.0083667790167965, + "p90_s": 0.008696587570011616, + "p95_s": 0.008707358787069098, + "p99_s": 0.008715975760715082, + "min_s": 0.008024482987821102, + "max_s": 0.008718130004126579 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.70079025439918e-06, + "p50_s": 6.407994078472257e-06, + "p90_s": 7.02879624441266e-06, + "p95_s": 8.058408275246618e-06, + "p99_s": 8.882097899913788e-06, + "min_s": 6.239977665245533e-06, + "max_s": 9.08802030608058e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 7168 + ], + "input_bytes": 542044160, + "packed_qdata_bytes": 135561216, + "block_scale_bytes": 16945152, + "output_bytes": 406533120, + "separate_materialization_bytes": 1101083648, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1120353280, + "negative_self_device_allocated_bytes": -1120353280, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_out_proj-vortex_recalculate.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 314.89499999999987, + "self_cpu_time_total_us": 56.24000000000001, + "device_time_total_us": 16426.848, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 815280128, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 250.41499999999985, + "self_cpu_time_total_us": 190.55899999999997, + "device_time_total_us": 16426.848, + "self_device_time_total_us": 16423.04, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 815280128, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 16423.04, + "self_device_time_total_us": 16423.04, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 257.75900000000024, + "self_cpu_time_total_us": 188.3510000000001, + "device_time_total_us": 5623.03, + "self_device_time_total_us": 5545.046, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 305070080, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 5545.046, + "self_device_time_total_us": 5545.046, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4233.208000000001, + "self_device_time_total_us": 4233.208000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 42.25599999999997, + "self_cpu_time_total_us": 6.33600000000024, + "device_time_total_us": 77.98399999999947, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 33947648, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 30.86399999999992, + "self_cpu_time_total_us": 11.935999999999922, + "device_time_total_us": 77.98399999999947, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.927999999999997, + "self_cpu_time_total_us": 7.295999999999935, + "device_time_total_us": 77.98399999999947, + "self_device_time_total_us": 77.98399999999947, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 77.98399999999947, + "self_device_time_total_us": 77.98399999999947, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 157.34400000000016, + "self_cpu_time_total_us": 22.51199999999983, + "device_time_total_us": 5.503999999998541, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 97.58400000000023, + "self_cpu_time_total_us": 75.34400000000045, + "device_time_total_us": 5.503999999998541, + "self_device_time_total_us": 5.503999999998541, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.903999999999087, + "self_device_time_total_us": 3.903999999999087, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25.104000000000042, + "self_cpu_time_total_us": 13.984000000000037, + "device_time_total_us": 3.8080000000018117, + "self_device_time_total_us": 3.8080000000018117, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.8080000000018117, + "self_device_time_total_us": 3.8080000000018117, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 26.65600000000012, + "self_cpu_time_total_us": 1.9040000000000532, + "device_time_total_us": 3.0399999999999636, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.0399999999999636, + "self_device_time_total_us": 3.0399999999999636, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 140.36800000000025, + "self_cpu_time_total_us": 7.776000000000153, + "device_time_total_us": 2.4639999999985776, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.4639999999985776, + "self_device_time_total_us": 2.4639999999985776, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25379.704, + "self_cpu_time_total_us": 25379.704, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 106.83200000000016, + "self_cpu_time_total_us": 106.83200000000016, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 62.54400000000001, + "self_cpu_time_total_us": 62.54400000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.248000000000104, + "self_cpu_time_total_us": 37.248000000000104, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25.43999999999994, + "self_cpu_time_total_us": 25.43999999999994, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1120350208, + "self_device_memory_usage_bytes": 1120350208, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.727999999999952, + "self_cpu_time_total_us": 7.231999999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.103999999999928, + "self_cpu_time_total_us": 11.103999999999928, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.240000000000009, + "self_cpu_time_total_us": 6.336000000000013, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.279999999999973, + "self_cpu_time_total_us": 7.279999999999973, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.223999999999819, + "self_cpu_time_total_us": 4.30399999999986, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.495999999999981, + "self_cpu_time_total_us": 4.495999999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.400000000000091, + "self_cpu_time_total_us": 4.400000000000091, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.3840000000000146, + "self_cpu_time_total_us": 1.6479999999999109, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.919999999999959, + "self_cpu_time_total_us": 1.919999999999959, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.9039999999999964, + "self_cpu_time_total_us": 1.9039999999999964, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1359999999999673, + "self_cpu_time_total_us": 1.1359999999999673, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.928000000000111, + "self_cpu_time_total_us": 0.928000000000111, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7360000000001037, + "self_cpu_time_total_us": 0.7360000000001037, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1120352256, + "self_device_memory_usage_bytes": -1120352256, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_out_proj", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 7168, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 3584 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 448 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 7168 + ], + "input_is_contiguous": true, + "input_stride": [ + 7168, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.027191946282983e-06, + "p50_s": 4.735979018732905e-06, + "p90_s": 5.1952141802757965e-06, + "p95_s": 6.613615551032122e-06, + "p99_s": 7.748336647637189e-06, + "min_s": 4.4800108298659325e-06, + "max_s": 8.032016921788454e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.164806058630347e-06, + "p50_s": 7.89598561823368e-06, + "p90_s": 8.761591743677855e-06, + "p95_s": 9.38080484047532e-06, + "p99_s": 9.876175317913293e-06, + "min_s": 7.744005415588617e-06, + "max_s": 1.0000017937272787e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.150398308411241e-06, + "p50_s": 4.1199964471161366e-06, + "p90_s": 4.235201049596071e-06, + "p95_s": 4.3575768359005455e-06, + "p99_s": 4.4554774649441245e-06, + "min_s": 4.01597935706377e-06, + "max_s": 4.479952622205019e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.002151049306849018, + "p50_s": 0.0021311805176083, + "p90_s": 0.0021834448620211334, + "p95_s": 0.0022133604245027525, + "p99_s": 0.002237292874488048, + "min_s": 0.0021157090086489916, + "max_s": 0.002243275986984372 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 1.2465595500543713e-05, + "p50_s": 1.2080010492354631e-05, + "p90_s": 1.4244805788621306e-05, + "p95_s": 1.4554386143572627e-05, + "p99_s": 1.4802050427533687e-05, + "min_s": 1.1231983080506325e-05, + "max_s": 1.486396649852395e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.192319905385375e-05, + "p50_s": 1.0927993571385741e-05, + "p90_s": 1.4393642777577042e-05, + "p95_s": 1.616481749806553e-05, + "p99_s": 1.758175727445632e-05, + "min_s": 1.0415969882160425e-05, + "max_s": 1.793599221855402e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0029934655001852663, + "p50_s": 0.002858611987903714, + "p90_s": 0.0032465396041516216, + "p95_s": 0.003295938795781694, + "p99_s": 0.0033354581490857528, + "min_s": 0.0028280760161578655, + "max_s": 0.0033453379874117672 + }, + "params_wrap": { + "count": 10, + "mean_s": 6.678409408777952e-06, + "p50_s": 6.6159991547465324e-06, + "p90_s": 6.9615547545254225e-06, + "p95_s": 6.968784146010876e-06, + "p99_s": 6.974567659199238e-06, + "min_s": 6.384041626006365e-06, + "max_s": 6.976013537496328e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.005205234803725034, + "p50_s": 0.005049952014815062, + "p90_s": 0.0054687846451997755, + "p95_s": 0.005527795810485258, + "p99_s": 0.0055750047427136455, + "min_s": 0.005013736023101956, + "max_s": 0.005586806975770742 + }, + "linear": { + "count": 10, + "mean_s": 0.008270908409031108, + "p50_s": 0.008207883016439155, + "p90_s": 0.00870191397261806, + "p95_s": 0.008760846007498912, + "p99_s": 0.008807991635403597, + "min_s": 0.007719588000327349, + "max_s": 0.008819778042379767 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.51999143883586e-06, + "p50_s": 6.471993401646614e-06, + "p90_s": 6.943999323993921e-06, + "p95_s": 6.943999323993921e-06, + "p99_s": 6.943999323993921e-06, + "min_s": 6.2239705584943295e-06, + "max_s": 6.943999323993921e-06 + } + }, + "reference_diff": { + "max": 1408.0, + "mean": 51.06270217895508 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 7168 + ], + "input_bytes": 542044160, + "packed_qdata_bytes": 135561216, + "block_scale_bytes": 16945152, + "output_bytes": 406533120, + "separate_materialization_bytes": 1101083648, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1118440448, + "negative_self_device_allocated_bytes": -1118440448, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/attn_out_proj-vortex_native.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 696.5589999999997, + "self_cpu_time_total_us": 127.80799999999988, + "device_time_total_us": 16839.242999999995, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 554.9269999999998, + "self_cpu_time_total_us": 438.75199999999984, + "device_time_total_us": 16839.242999999995, + "self_device_time_total_us": 16835.371, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 16835.371, + "self_device_time_total_us": 16835.371, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 5570.564, + "self_device_time_total_us": 5570.564, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4346.603000000001, + "self_device_time_total_us": 4346.603000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.80799999999988, + "self_cpu_time_total_us": 9.055999999999813, + "device_time_total_us": 77.72800000000143, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 33947648, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 36.320000000000164, + "self_cpu_time_total_us": 10.368000000000166, + "device_time_total_us": 77.72800000000143, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25.951999999999998, + "self_cpu_time_total_us": 13.023999999999887, + "device_time_total_us": 77.72800000000143, + "self_device_time_total_us": 77.72800000000143, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 77.72800000000143, + "self_device_time_total_us": 77.72800000000143, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 172.28800000000012, + "self_cpu_time_total_us": 24.944000000000074, + "device_time_total_us": 5.503999999999905, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 97.18400000000008, + "self_cpu_time_total_us": 66.70400000000018, + "device_time_total_us": 5.503999999999905, + "self_device_time_total_us": 5.503999999999905, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.904000000000906, + "self_device_time_total_us": 3.904000000000906, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34.38300000000004, + "self_cpu_time_total_us": 20.75200000000018, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 32.624000000000024, + "self_cpu_time_total_us": 2.7519999999999527, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 2.9760000000005675, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 151.61599999999987, + "self_cpu_time_total_us": 9.199999999999818, + "device_time_total_us": 2.527999999999338, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.527999999999338, + "self_device_time_total_us": 2.527999999999338, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25651.799, + "self_cpu_time_total_us": 25651.799, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 116.63799999999958, + "self_cpu_time_total_us": 116.63799999999958, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 64.56, + "self_cpu_time_total_us": 64.56, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 50.15999999999997, + "self_cpu_time_total_us": 50.15999999999997, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 44.35200000000009, + "self_cpu_time_total_us": 44.35200000000009, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 32.27199999999971, + "self_cpu_time_total_us": 32.27199999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1118437376, + "self_device_memory_usage_bytes": 1118437376, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.687999999999874, + "self_cpu_time_total_us": 16.687999999999874, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.496000000000066, + "self_cpu_time_total_us": 8.448000000000178, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.82400000000007, + "self_cpu_time_total_us": 9.072000000000116, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.384000000000242, + "self_cpu_time_total_us": 2.576000000000249, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.047999999999888, + "self_cpu_time_total_us": 6.047999999999888, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.751999999999953, + "self_cpu_time_total_us": 4.751999999999953, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.8079999999999927, + "self_cpu_time_total_us": 3.8079999999999927, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.6960000000000264, + "self_cpu_time_total_us": 3.6960000000000264, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.0399999999999636, + "self_cpu_time_total_us": 1.9200000000000728, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6320000000002892, + "self_cpu_time_total_us": 1.6320000000002892, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1520000000000437, + "self_cpu_time_total_us": 1.1520000000000437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1199999999998909, + "self_cpu_time_total_us": 1.1199999999998909, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1118439424, + "self_device_memory_usage_bytes": -1118439424, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc1", + "quantizer": "comfy", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 28672, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 28672, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 28672, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 4.872021963819861e-06, + "p50_s": 4.6000059228390455e-06, + "p90_s": 5.0144211854785676e-06, + "p95_s": 6.123216007836161e-06, + "p99_s": 7.010251865722239e-06, + "min_s": 4.496017936617136e-06, + "max_s": 7.232010830193758e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.105597225949169e-06, + "p50_s": 7.927999831736088e-06, + "p90_s": 8.56962869875133e-06, + "p95_s": 9.044804028235375e-06, + "p99_s": 9.424944291822614e-06, + "min_s": 7.744005415588617e-06, + "max_s": 9.519979357719421e-06 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.192005144432187e-06, + "p50_s": 4.1120219975709915e-06, + "p90_s": 4.332786193117499e-06, + "p95_s": 4.53439715784043e-06, + "p99_s": 4.695685929618775e-06, + "min_s": 4.047993570566177e-06, + "max_s": 4.736008122563362e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.007388551312033087, + "p50_s": 0.007319628493860364, + "p90_s": 0.007553186418954283, + "p95_s": 0.007642171217594295, + "p99_s": 0.007713359056506306, + "min_s": 0.007295236049685627, + "max_s": 0.0077311560162343085 + }, + "linear": { + "count": 10, + "mean_s": 0.03401212958851829, + "p50_s": 0.03394539398141205, + "p90_s": 0.03457325681229122, + "p95_s": 0.03510591190715786, + "p99_s": 0.03553203598305117, + "min_s": 0.03306085895746946, + "max_s": 0.0356385670020245 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.496004061773419e-06, + "p50_s": 6.440008291974664e-06, + "p90_s": 6.731203757226467e-06, + "p95_s": 6.925611523911356e-06, + "p99_s": 7.081137737259269e-06, + "min_s": 6.127986125648022e-06, + "max_s": 7.120019290596247e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 2168176640, + "separate_materialization_bytes": 2689089536, + "epilogue_candidate": "BF16 split and SwiGLU input" + }, + "profiler_summary": { + "device_kernel_launches": 18, + "unique_device_kernels": 9, + "positive_self_device_allocated_bytes": 5381863424, + "negative_self_device_allocated_bytes": -5381863424, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc1-comfy.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1490.0310000000009, + "self_cpu_time_total_us": 347.7600000000002, + "device_time_total_us": 67700.774, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1135.6949999999997, + "self_cpu_time_total_us": 1005.2789999999968, + "device_time_total_us": 67700.774, + "self_device_time_total_us": 67696.902, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 67696.902, + "self_device_time_total_us": 67696.902, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2874.414999999999, + "self_cpu_time_total_us": 606.847999999999, + "device_time_total_us": 10481.243000000002, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2247.343, + "self_cpu_time_total_us": 731.3599999999997, + "device_time_total_us": 10481.243000000002, + "self_device_time_total_us": 6973.885000000002, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [0]]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::AbsFunctor, std::array >(int, at::native::AbsFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 6973.885000000002, + "self_device_time_total_us": 6973.885000000002, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1524.686999999999, + "self_cpu_time_total_us": 476.7679999999991, + "device_time_total_us": 4195.678000000007, + "self_device_time_total_us": 4150.206000000004, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 230834176, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4150.206000000004, + "self_device_time_total_us": 4150.206000000004, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 829.2470000000003, + "self_cpu_time_total_us": 829.2470000000003, + "device_time_total_us": 3507.358, + "self_device_time_total_us": 3507.358, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::amax", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 165.39200000000073, + "self_cpu_time_total_us": 74.08000000000084, + "device_time_total_us": 3495.5510000000013, + "self_device_time_total_us": 3495.5510000000013, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[37810, 5376], [], []]" + }, + { + "key": "void at::native::reduce_kernel<512, 1, at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4> >(at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3494.6870000000035, + "self_device_time_total_us": 3494.6870000000035, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 991.9030000000002, + "self_cpu_time_total_us": 14.78399999999965, + "device_time_total_us": 45.47200000000339, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 971.103000000001, + "self_cpu_time_total_us": 916.6550000000007, + "device_time_total_us": 45.47200000000339, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 54.44800000000032, + "self_cpu_time_total_us": 19.360000000000582, + "device_time_total_us": 45.47200000000339, + "self_device_time_total_us": 45.47200000000339, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 45.47200000000339, + "self_device_time_total_us": 45.47200000000339, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 47.823999999998705, + "self_cpu_time_total_us": 20.89599999999882, + "device_time_total_us": 3.872000000004846, + "self_device_time_total_us": 3.872000000004846, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.872000000004846, + "self_device_time_total_us": 3.872000000004846, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::div", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 50.25600000000031, + "self_cpu_time_total_us": 29.088000000000648, + "device_time_total_us": 3.6159999999999854, + "self_device_time_total_us": 3.6159999999999854, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BUnaryFunctor >, std::array >(int, at::native::BUnaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.6159999999999854, + "self_device_time_total_us": 3.6159999999999854, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 96.75199999999859, + "self_cpu_time_total_us": 11.21600000000035, + "device_time_total_us": 2.5279999999984284, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 85.53599999999824, + "self_cpu_time_total_us": 23.519999999999527, + "device_time_total_us": 2.5279999999984284, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 55.23199999999906, + "self_cpu_time_total_us": 28.575999999999112, + "device_time_total_us": 2.5279999999984284, + "self_device_time_total_us": 2.5279999999984284, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.5279999999984284, + "self_device_time_total_us": 2.5279999999984284, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Memset (Device)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.863999999997759, + "self_device_time_total_us": 0.863999999997759, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 78306.49100000001, + "self_cpu_time_total_us": 78306.49100000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 867.0880000000016, + "self_cpu_time_total_us": 867.0880000000016, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 333.87200000000166, + "self_cpu_time_total_us": 330.12800000000107, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::empty", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 53.36000000000058, + "self_cpu_time_total_us": 53.36000000000058, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4568793088, + "self_device_memory_usage_bytes": 4568793088, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cudaMemsetAsync", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.08799999999974, + "self_cpu_time_total_us": 39.08799999999974, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 29.98400000000038, + "self_cpu_time_total_us": 29.98400000000038, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.496000000001004, + "self_cpu_time_total_us": 20.496000000001004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.67200000000048, + "self_cpu_time_total_us": 14.67200000000048, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::resize_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.256000000000313, + "self_cpu_time_total_us": 10.256000000000313, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813066240, + "self_device_memory_usage_bytes": 813066240, + "flops": 0, + "input_shapes": "[[0], [], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.263999999999214, + "self_cpu_time_total_us": 6.143999999998414, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::empty_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.783999999999651, + "self_cpu_time_total_us": 6.783999999999651, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.576000000000931, + "self_cpu_time_total_us": 5.040000000002692, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.87999999999829, + "self_cpu_time_total_us": 4.87999999999829, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.7440000000005966, + "self_cpu_time_total_us": 3.7440000000005966, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.6480000000010477, + "self_cpu_time_total_us": 3.6480000000010477, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.1200000000008004, + "self_cpu_time_total_us": 3.1200000000008004, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.040000000000873, + "self_cpu_time_total_us": 2.2400000000016007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.5359999999982392, + "self_cpu_time_total_us": 1.5359999999982392, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.375999999999749, + "self_cpu_time_total_us": 1.375999999999749, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2639999999992142, + "self_cpu_time_total_us": 1.2639999999992142, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7999999999992724, + "self_cpu_time_total_us": 0.7999999999992724, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "[memory]", + "count": 18, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -5381862400, + "self_device_memory_usage_bytes": -5381862400, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc1", + "quantizer": "vortex_recalculate", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 28672, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 28672, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 28672, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 4.996801726520061e-06, + "p50_s": 4.631991032510996e-06, + "p90_s": 5.161581793799995e-06, + "p95_s": 6.788797327317294e-06, + "p99_s": 8.090569754131138e-06, + "min_s": 4.447996616363525e-06, + "max_s": 8.416012860834599e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.19360138848424e-06, + "p50_s": 8.16801912151277e-06, + "p90_s": 8.572766091674566e-06, + "p95_s": 9.206385584548114e-06, + "p99_s": 9.713281178846957e-06, + "min_s": 7.503957021981478e-06, + "max_s": 9.840005077421665e-06 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.1727907955646515e-06, + "p50_s": 4.160014213994145e-06, + "p90_s": 4.3039792217314245e-06, + "p95_s": 4.447990795597434e-06, + "p99_s": 4.563200054690241e-06, + "min_s": 3.936002030968666e-06, + "max_s": 4.592002369463444e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0016323688847478478, + "p50_s": 0.0016255035006906837, + "p90_s": 0.0016488469729665667, + "p95_s": 0.0016816789662698283, + "p99_s": 0.0017079445609124377, + "min_s": 0.0016072000144049525, + "max_s": 0.00171451095957309 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 1.3878295430913567e-05, + "p50_s": 1.2615957530215383e-05, + "p90_s": 1.8737633945420383e-05, + "p95_s": 1.9536822219379242e-05, + "p99_s": 2.0176172838546336e-05, + "min_s": 1.1696014553308487e-05, + "max_s": 2.0336010493338108e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.1468795128166675e-05, + "p50_s": 1.0991992894560099e-05, + "p90_s": 1.2360006803646683e-05, + "p95_s": 1.3619990204460917e-05, + "p99_s": 1.4627976925112308e-05, + "min_s": 1.0575982742011547e-05, + "max_s": 1.4879973605275154e-05 + }, + "ck_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0021427478175610306, + "p50_s": 0.002115183015121147, + "p90_s": 0.0022523318126332014, + "p95_s": 0.002252965405932628, + "p99_s": 0.0022534722805721687, + "min_s": 0.0021083989995531738, + "max_s": 0.0022535989992320538 + }, + "params_wrap": { + "count": 10, + "mean_s": 6.8128050770610574e-06, + "p50_s": 6.791990017518401e-06, + "p90_s": 7.233611540868878e-06, + "p95_s": 7.24081473890692e-06, + "p99_s": 7.246577297337353e-06, + "min_s": 6.336020305752754e-06, + "max_s": 7.2480179369449615e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.003836804605089128, + "p50_s": 0.0038021259824745357, + "p90_s": 0.003966951585607603, + "p95_s": 0.004002778799622319, + "p99_s": 0.0040314405708340925, + "min_s": 0.003776589990593493, + "max_s": 0.004038606013637036 + }, + "linear": { + "count": 10, + "mean_s": 0.034088603203417736, + "p50_s": 0.034148512000683695, + "p90_s": 0.034465921606170014, + "p95_s": 0.03457616078958381, + "p99_s": 0.03466435213631485, + "min_s": 0.033573855995200574, + "max_s": 0.034686399972997606 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.470386870205402e-06, + "p50_s": 6.511982064694166e-06, + "p90_s": 6.7823741119354965e-06, + "p95_s": 6.919182487763464e-06, + "p99_s": 7.0286291884258385e-06, + "min_s": 6.127986125648022e-06, + "max_s": 7.0559908635914326e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 2168176640, + "separate_materialization_bytes": 2689089536, + "epilogue_candidate": "BF16 split and SwiGLU input" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 4568796160, + "negative_self_device_allocated_bytes": -4568796160, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc1-vortex_recalculate.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3680.063, + "self_cpu_time_total_us": 1429.4720000000016, + "device_time_total_us": 68406.907, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2217.3909999999996, + "self_cpu_time_total_us": 1829.5510000000013, + "device_time_total_us": 68406.907, + "self_device_time_total_us": 68403.547, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 68403.547, + "self_device_time_total_us": 68403.547, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1975.7920000000013, + "self_cpu_time_total_us": 491.7920000000049, + "device_time_total_us": 4257.5679999999975, + "self_device_time_total_us": 4212.2239999999965, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 230834176, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4212.2239999999965, + "self_device_time_total_us": 4212.2239999999965, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3363.5510000000068, + "self_device_time_total_us": 3363.5510000000068, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1429.2639999999974, + "self_cpu_time_total_us": 525.0559999999969, + "device_time_total_us": 45.34400000000096, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 899.2639999999992, + "self_cpu_time_total_us": 825.616, + "device_time_total_us": 45.34400000000096, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 73.64799999999923, + "self_cpu_time_total_us": 28.67199999999866, + "device_time_total_us": 45.34400000000096, + "self_device_time_total_us": 45.34400000000096, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 45.34400000000096, + "self_device_time_total_us": 45.34400000000096, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1569.7269999999999, + "self_cpu_time_total_us": 1278.831000000001, + "device_time_total_us": 5.504000000000815, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 172.47999999999956, + "self_cpu_time_total_us": 88.46399999999812, + "device_time_total_us": 5.504000000000815, + "self_device_time_total_us": 5.504000000000815, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.9679999999971187, + "self_device_time_total_us": 3.9679999999971187, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 315.9359999999997, + "self_cpu_time_total_us": 249.74399999999878, + "device_time_total_us": 3.36000000000422, + "self_device_time_total_us": 3.36000000000422, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.36000000000422, + "self_device_time_total_us": 3.36000000000422, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 44.91199999999844, + "self_cpu_time_total_us": 4.287999999998647, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 2.9760000000005675, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1554.5429999999997, + "self_cpu_time_total_us": 25.4399999999996, + "device_time_total_us": 2.5280000000002474, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.5280000000002474, + "self_device_time_total_us": 2.5280000000002474, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 69911.369, + "self_cpu_time_total_us": 69911.369, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1079.4230000000034, + "self_cpu_time_total_us": 1079.4230000000034, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 929.1040000000003, + "self_cpu_time_total_us": 929.1040000000003, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 407.0560000000005, + "self_cpu_time_total_us": 400.9440000000013, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 118.41599999999926, + "self_cpu_time_total_us": 118.41599999999926, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.47200000000157, + "self_cpu_time_total_us": 37.47200000000157, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4568793088, + "self_device_memory_usage_bytes": 4568793088, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 33.19999999999891, + "self_cpu_time_total_us": 26.591999999998734, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 21.375999999998385, + "self_cpu_time_total_us": 21.375999999998385, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 21.023999999999432, + "self_cpu_time_total_us": 21.023999999999432, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.935999999999694, + "self_cpu_time_total_us": 15.935999999999694, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.072000000000116, + "self_cpu_time_total_us": 7.775999999998021, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.2960000000020955, + "self_cpu_time_total_us": 7.2960000000020955, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.608000000000175, + "self_cpu_time_total_us": 6.608000000000175, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.495999999999185, + "self_cpu_time_total_us": 4.31999999999789, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.1119999999991705, + "self_cpu_time_total_us": 6.1119999999991705, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.992000000000189, + "self_cpu_time_total_us": 2.992000000000189, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.9440000000013242, + "self_cpu_time_total_us": 2.9440000000013242, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.176000000001295, + "self_cpu_time_total_us": 2.176000000001295, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -4568795136, + "self_device_memory_usage_bytes": -4568795136, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc1", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 28672, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 28672, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 28672, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.4367876145988704e-06, + "p50_s": 4.743982572108507e-06, + "p90_s": 7.131200982257723e-06, + "p95_s": 7.973608444444833e-06, + "p99_s": 8.647534414194525e-06, + "min_s": 4.4959597289562225e-06, + "max_s": 8.816015906631947e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 9.887997293844819e-06, + "p50_s": 8.920003892853856e-06, + "p90_s": 1.2116803554818033e-05, + "p95_s": 1.4730411930941038e-05, + "p99_s": 1.6821298631839455e-05, + "min_s": 7.792026735842228e-06, + "max_s": 1.7344020307064056e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.305597394704819e-06, + "p50_s": 4.223984433338046e-06, + "p90_s": 4.609598545357585e-06, + "p95_s": 4.760813317261636e-06, + "p99_s": 4.881785134784878e-06, + "min_s": 4.047993570566177e-06, + "max_s": 4.9120280891656876e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0016478794103022664, + "p50_s": 0.0016467270033899695, + "p90_s": 0.0016853383334819226, + "p95_s": 0.0016977731749648227, + "p99_s": 0.001707721048151143, + "min_s": 0.0016089430428110063, + "max_s": 0.001710208016447723 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 0.00011893280898220838, + "p50_s": 1.4992023352533579e-05, + "p90_s": 0.0001227600208949294, + "p95_s": 0.0005900040006963526, + "p99_s": 0.0009637991845374928, + "min_s": 1.1680007446557283e-05, + "max_s": 0.0010572479804977775 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.5512004029005767e-05, + "p50_s": 1.1624011676758528e-05, + "p90_s": 2.5340763386338946e-05, + "p95_s": 2.6550376787781715e-05, + "p99_s": 2.7518067508935928e-05, + "min_s": 1.0560033842921257e-05, + "max_s": 2.775999018922448e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.00223893690854311, + "p50_s": 0.002212615014286712, + "p90_s": 0.002293387893587351, + "p95_s": 0.0023610539501532912, + "p99_s": 0.0024151867954060437, + "min_s": 0.0021939509897492826, + "max_s": 0.0024287200067192316 + }, + "params_wrap": { + "count": 10, + "mean_s": 9.187200339511037e-06, + "p50_s": 6.959977326914668e-06, + "p90_s": 9.856035467237227e-06, + "p95_s": 1.964803668670354e-05, + "p99_s": 2.748163766227663e-05, + "min_s": 6.320013199001551e-06, + "max_s": 2.944003790616989e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.004063829098595306, + "p50_s": 0.0039235510339494795, + "p90_s": 0.00427848370745778, + "p95_s": 0.004657296853838488, + "p99_s": 0.004960347370943054, + "min_s": 0.003862606012262404, + "max_s": 0.005036110000219196 + }, + "linear": { + "count": 10, + "mean_s": 0.051964699605014174, + "p50_s": 0.051965390506666154, + "p90_s": 0.07021930979099125, + "p95_s": 0.07034799542161636, + "p99_s": 0.07045094392611645, + "min_s": 0.03307627700269222, + "max_s": 0.07047668105224147 + }, + "slice_reshape": { + "count": 10, + "mean_s": 7.37918890081346e-06, + "p50_s": 6.959977326914668e-06, + "p90_s": 8.595181861892343e-06, + "p95_s": 9.113576379604637e-06, + "p99_s": 9.528291993774474e-06, + "min_s": 6.175949238240719e-06, + "max_s": 9.631970897316933e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12708864, + "output_bytes": 2168176640, + "separate_materialization_bytes": 2689089536, + "epilogue_candidate": "BF16 split and SwiGLU input" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 4568796160, + "negative_self_device_allocated_bytes": -4568796160, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc1-vortex_native.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 727.5519999999997, + "self_cpu_time_total_us": 141.0880000000002, + "device_time_total_us": 67877.84599999999, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 569.2799999999997, + "self_cpu_time_total_us": 453.4079999999999, + "device_time_total_us": 67877.84599999999, + "self_device_time_total_us": 67873.97399999999, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 67873.97399999999, + "self_device_time_total_us": 67873.97399999999, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4261.194000000005, + "self_device_time_total_us": 4261.194000000005, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3239.6230000000014, + "self_device_time_total_us": 3239.6230000000014, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 62.19199999999978, + "self_cpu_time_total_us": 15.568000000000666, + "device_time_total_us": 45.792000000002645, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 27492352, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 41.90399999999954, + "self_cpu_time_total_us": 10.67199999999957, + "device_time_total_us": 45.792000000002645, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.23199999999997, + "self_cpu_time_total_us": 12.672000000000025, + "device_time_total_us": 45.792000000002645, + "self_device_time_total_us": 45.792000000002645, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 45.792000000002645, + "self_device_time_total_us": 45.792000000002645, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 273.9520000000002, + "self_cpu_time_total_us": 34.40000000000077, + "device_time_total_us": 5.472000000000662, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 140.5119999999995, + "self_cpu_time_total_us": 56.6400000000001, + "device_time_total_us": 5.472000000000662, + "self_device_time_total_us": 5.472000000000662, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.9370000000012624, + "self_device_time_total_us": 3.9370000000012624, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 42.14399999999978, + "self_cpu_time_total_us": 24.1279999999997, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 41.69600000000014, + "self_cpu_time_total_us": 3.423999999999978, + "device_time_total_us": 2.9760000000010223, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000010223, + "self_device_time_total_us": 2.9760000000010223, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 256.0639999999999, + "self_cpu_time_total_us": 20.383999999999844, + "device_time_total_us": 2.49599999999964, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.49599999999964, + "self_device_time_total_us": 2.49599999999964, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 73904.807, + "self_cpu_time_total_us": 73904.807, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1172.5439999999999, + "self_cpu_time_total_us": 1172.5439999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 197.6479999999999, + "self_cpu_time_total_us": 197.6479999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 99.03999999999996, + "self_cpu_time_total_us": 99.03999999999996, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34.95999999999913, + "self_cpu_time_total_us": 34.95999999999913, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4568793088, + "self_device_memory_usage_bytes": 4568793088, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.695999999999913, + "self_cpu_time_total_us": 31.695999999999913, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.18399999999974, + "self_cpu_time_total_us": 12.495999999999185, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.088000000000193, + "self_cpu_time_total_us": 17.088000000000193, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.832000000000136, + "self_cpu_time_total_us": 9.040000000000362, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.431999999999789, + "self_cpu_time_total_us": 6.431999999999789, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.320000000000164, + "self_cpu_time_total_us": 3.6159999999999854, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.791999999999774, + "self_cpu_time_total_us": 5.791999999999774, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.688000000000557, + "self_cpu_time_total_us": 4.688000000000557, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.6480000000001382, + "self_cpu_time_total_us": 1.8080000000004475, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.9440000000004147, + "self_cpu_time_total_us": 2.9440000000004147, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.7040000000001783, + "self_cpu_time_total_us": 2.7040000000001783, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.8399999999996908, + "self_cpu_time_total_us": 1.8399999999996908, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3600000000001273, + "self_cpu_time_total_us": 1.3600000000001273, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -4568795136, + "self_device_memory_usage_bytes": -4568795136, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc2", + "quantizer": "comfy", + "module": { + "class": "Nvfp4Linear", + "in_features": 14336, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 7168 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 896 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 14336 + ], + "input_is_contiguous": true, + "input_stride": [ + 14336, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.179201252758503e-06, + "p50_s": 4.592002369463444e-06, + "p90_s": 5.366385448724029e-06, + "p95_s": 7.843185449019069e-06, + "p99_s": 9.824625449255109e-06, + "min_s": 4.511966835707426e-06, + "max_s": 1.0319985449314117e-05 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.53920355439186e-06, + "p50_s": 8.416012860834599e-06, + "p90_s": 8.804787648841739e-06, + "p95_s": 9.618382318876683e-06, + "p99_s": 1.0269258054904639e-05, + "min_s": 7.935974281281233e-06, + "max_s": 1.0431976988911629e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.128011642023921e-06, + "p50_s": 4.056026227772236e-06, + "p90_s": 4.284823080524802e-06, + "p95_s": 4.486434045247733e-06, + "p99_s": 4.647722817026079e-06, + "min_s": 3.999972250312567e-06, + "max_s": 4.688045009970665e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.01925712469383143, + "p50_s": 0.019156786001985893, + "p90_s": 0.019557130028260872, + "p95_s": 0.019652889997814782, + "p99_s": 0.01972949797345791, + "min_s": 0.01911408902378753, + "max_s": 0.019748649967368692 + }, + "linear": { + "count": 10, + "mean_s": 0.0551715788024012, + "p50_s": 0.055235853971680626, + "p90_s": 0.05537108918651938, + "p95_s": 0.055614175589289515, + "p99_s": 0.05580864471150562, + "min_s": 0.054014783003367484, + "max_s": 0.05585726199205965 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.69279252178967e-06, + "p50_s": 6.6559878177940845e-06, + "p90_s": 6.865593604743481e-06, + "p95_s": 7.16079375706613e-06, + "p99_s": 7.3969538789242504e-06, + "min_s": 6.4159976318478584e-06, + "max_s": 7.455993909388781e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 14336 + ], + "input_bytes": 1084088320, + "packed_qdata_bytes": 271122432, + "block_scale_bytes": 33890304, + "output_bytes": 406533120, + "separate_materialization_bytes": 1795634176, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 18, + "unique_device_kernels": 9, + "positive_self_device_allocated_bytes": 3591688192, + "negative_self_device_allocated_bytes": -3591688192, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc2-comfy.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 684.2070000000003, + "self_cpu_time_total_us": 126.79999999999927, + "device_time_total_us": 110675.38, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 546.335, + "self_cpu_time_total_us": 425.0710000000008, + "device_time_total_us": 110675.38, + "self_device_time_total_us": 110671.54, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 110671.54000000001, + "self_device_time_total_us": 110671.54000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1181.8869999999997, + "self_cpu_time_total_us": 15.695999999999685, + "device_time_total_us": 26738.947999999997, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2168176640, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336]]" + }, + { + "key": "aten::abs", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1146.1910000000005, + "self_cpu_time_total_us": 998.768, + "device_time_total_us": 26738.947999999997, + "self_device_time_total_us": 17820.685999999998, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2168176640, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [0]]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::AbsFunctor, std::array >(int, at::native::AbsFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 17820.685999999998, + "self_device_time_total_us": 17820.685999999998, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 616.7360000000003, + "self_cpu_time_total_us": 492.43200000000024, + "device_time_total_us": 11358.48500000001, + "self_device_time_total_us": 11129.75000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 610140160, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 11129.75000000001, + "self_device_time_total_us": 11129.75000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 63.29500000000007, + "self_cpu_time_total_us": 63.29500000000007, + "device_time_total_us": 8918.261999999999, + "self_device_time_total_us": 8918.261999999999, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::amax", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 528.1280000000006, + "self_cpu_time_total_us": 350.1120000000001, + "device_time_total_us": 8901.593000000004, + "self_device_time_total_us": 8901.593000000004, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[37810, 14336], [], []]" + }, + { + "key": "void at::native::reduce_kernel<512, 1, at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4> >(at::native::ReduceOp >, unsigned int, c10::BFloat16, 4, 4>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 8900.761000000006, + "self_device_time_total_us": 8900.761000000006, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 69.60000000000036, + "self_cpu_time_total_us": 18.736000000000786, + "device_time_total_us": 228.73500000000058, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 67895296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 46.7199999999998, + "self_cpu_time_total_us": 14.831999999999425, + "device_time_total_us": 228.73500000000058, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.888000000000375, + "self_cpu_time_total_us": 17.135999999999967, + "device_time_total_us": 228.73500000000058, + "self_device_time_total_us": 228.73500000000058, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 228.73500000000058, + "self_device_time_total_us": 228.73500000000058, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 52.159999999999854, + "self_cpu_time_total_us": 29.040000000000873, + "device_time_total_us": 3.8400000000074215, + "self_device_time_total_us": 3.8400000000074215, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.8400000000074215, + "self_device_time_total_us": 3.8400000000074215, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::div", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 67.35999999999967, + "self_cpu_time_total_us": 45.2800000000002, + "device_time_total_us": 3.679999999993015, + "self_device_time_total_us": 3.679999999993015, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BUnaryFunctor >, std::array >(int, at::native::BUnaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.679999999993015, + "self_device_time_total_us": 3.679999999993015, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 155.19999999999982, + "self_cpu_time_total_us": 11.456000000000131, + "device_time_total_us": 2.463999999999942, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 143.7439999999997, + "self_cpu_time_total_us": 31.615999999999985, + "device_time_total_us": 2.463999999999942, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 97.23199999999997, + "self_cpu_time_total_us": 36.95999999999958, + "device_time_total_us": 2.463999999999942, + "self_device_time_total_us": 2.463999999999942, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.463999999999942, + "self_device_time_total_us": 2.463999999999942, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Memset (Device)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.8319999999985157, + "self_device_time_total_us": 0.8319999999985157, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 144756.801, + "self_cpu_time_total_us": 144756.801, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 267.56799999999976, + "self_cpu_time_total_us": 267.56799999999976, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaMemsetAsync", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 85.11999999999989, + "self_cpu_time_total_us": 85.11999999999989, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 52.351999999998895, + "self_cpu_time_total_us": 52.351999999998895, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1423507456, + "self_device_memory_usage_bytes": 1423507456, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::resize_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.648000000000366, + "self_cpu_time_total_us": 39.648000000000366, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2168176640, + "self_device_memory_usage_bytes": 2168176640, + "flops": 0, + "input_shapes": "[[0], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24.720000000000255, + "self_cpu_time_total_us": 24.720000000000255, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 19.071999999999207, + "self_cpu_time_total_us": 19.071999999999207, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.32799999999952, + "self_cpu_time_total_us": 9.135999999999058, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::empty_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.89599999999973, + "self_cpu_time_total_us": 14.89599999999973, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.344000000000023, + "self_cpu_time_total_us": 8.079999999999757, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.903999999999996, + "self_cpu_time_total_us": 11.903999999999996, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.16800000000012, + "self_cpu_time_total_us": 11.16800000000012, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.072000000001026, + "self_cpu_time_total_us": 7.792000000000371, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.431999999999789, + "self_cpu_time_total_us": 3.5039999999999054, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.192000000000462, + "self_cpu_time_total_us": 6.192000000000462, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.264000000000266, + "self_cpu_time_total_us": 5.264000000000266, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.488000000000426, + "self_cpu_time_total_us": 3.488000000000426, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4400000000005093, + "self_cpu_time_total_us": 3.4400000000005093, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.280000000000655, + "self_cpu_time_total_us": 3.280000000000655, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.9600000000000364, + "self_cpu_time_total_us": 2.9600000000000364, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.9279999999998836, + "self_cpu_time_total_us": 2.9279999999998836, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 18, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -3591687168, + "self_device_memory_usage_bytes": -3591687168, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc2", + "quantizer": "vortex_recalculate", + "module": { + "class": "Nvfp4Linear", + "in_features": 14336, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 7168 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 896 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 14336 + ], + "input_is_contiguous": true, + "input_stride": [ + 14336, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 4.913605516776443e-06, + "p50_s": 4.656030796468258e-06, + "p90_s": 5.064037395641207e-06, + "p95_s": 6.180009222589431e-06, + "p99_s": 7.072786684148015e-06, + "min_s": 4.4800108298659325e-06, + "max_s": 7.295981049537659e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.223997429013253e-06, + "p50_s": 8.096016244962811e-06, + "p90_s": 8.585641626268624e-06, + "p95_s": 9.06084314920008e-06, + "p99_s": 9.441004367545247e-06, + "min_s": 7.647962775081396e-06, + "max_s": 9.536044672131538e-06 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.2159983422607185e-06, + "p50_s": 4.0959566831588745e-06, + "p90_s": 4.289590287953615e-06, + "p95_s": 4.800781607627867e-06, + "p99_s": 5.209734663367272e-06, + "min_s": 4.00003045797348e-06, + "max_s": 5.311972927302122e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0042167027073446665, + "p50_s": 0.004208286991342902, + "p90_s": 0.004253056499874219, + "p95_s": 0.004259255257784389, + "p99_s": 0.004264214264112525, + "min_s": 0.00419579102890566, + "max_s": 0.004265454015694559 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 1.2134405551478267e-05, + "p50_s": 1.1976022506132722e-05, + "p90_s": 1.271362998522818e-05, + "p95_s": 1.2972814147360623e-05, + "p99_s": 1.3180161477066575e-05, + "min_s": 1.1487980373203754e-05, + "max_s": 1.3231998309493065e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.143038971349597e-05, + "p50_s": 1.084798714146018e-05, + "p90_s": 1.190401380881667e-05, + "p95_s": 1.399199827574193e-05, + "p99_s": 1.5662385849282145e-05, + "min_s": 1.0607996955513954e-05, + "max_s": 1.6079982742667198e-05 + }, + "ck_quantize_nvfp4": { + "count": 10, + "mean_s": 0.005730823596240953, + "p50_s": 0.005727157986257225, + "p90_s": 0.005760842794552445, + "p95_s": 0.0057634924189187585, + "p99_s": 0.00576561211841181, + "min_s": 0.005701629968825728, + "max_s": 0.005766142043285072 + }, + "params_wrap": { + "count": 10, + "mean_s": 6.742402911186219e-06, + "p50_s": 6.688002031296492e-06, + "p90_s": 6.943981861695647e-06, + "p95_s": 7.159999222494661e-06, + "p99_s": 7.332813111133874e-06, + "min_s": 6.479967851191759e-06, + "max_s": 7.376016583293676e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.01000769910751842, + "p50_s": 0.010017916531069204, + "p90_s": 0.010028161690570414, + "p95_s": 0.010029766825027764, + "p99_s": 0.010031050932593644, + "min_s": 0.009965389035642147, + "max_s": 0.010031371959485114 + }, + "linear": { + "count": 10, + "mean_s": 0.05473228610935621, + "p50_s": 0.05463681402034126, + "p90_s": 0.05536423642770387, + "p95_s": 0.055373805234557955, + "p99_s": 0.05538146028004121, + "min_s": 0.05412913399050012, + "max_s": 0.055383374041412026 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.51679583825171e-06, + "p50_s": 6.4480118453502655e-06, + "p90_s": 6.679981015622616e-06, + "p95_s": 6.859982386231422e-06, + "p99_s": 7.003983482718468e-06, + "min_s": 6.336020305752754e-06, + "max_s": 7.039983756840229e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 14336 + ], + "input_bytes": 1084088320, + "packed_qdata_bytes": 271122432, + "block_scale_bytes": 33890304, + "output_bytes": 406533120, + "separate_materialization_bytes": 1795634176, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1423510528, + "negative_self_device_allocated_bytes": -1423510528, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc2-vortex_recalculate.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 231.60000000000002, + "self_cpu_time_total_us": 31.79200000000003, + "device_time_total_us": 109667.01, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 192.38400000000001, + "self_cpu_time_total_us": 139.2159999999999, + "device_time_total_us": 109667.01, + "self_device_time_total_us": 109663.04200000002, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 109663.04200000002, + "self_device_time_total_us": 109663.04200000002, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 153.48799999999994, + "self_cpu_time_total_us": 98.11199999999985, + "device_time_total_us": 11601.821000000004, + "self_device_time_total_us": 11372.093000000003, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 610140160, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [], [], [], []]" + }, + { + "key": "void comfy::(anonymous namespace)::quantize_nvfp4_kernel<__nv_bfloat16, __nv_fp4x2_e2m1, __nv_fp8_e4m3, true, true>(__nv_bfloat16 const*, float const*, __nv_fp4x2_e2m1*, __nv_fp8_e4m3*, unsigned long, unsigned long, unsigned long, unsigned long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 11372.093000000003, + "self_device_time_total_us": 11372.093000000003, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 8416.318000000005, + "self_device_time_total_us": 8416.318000000005, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34.81599999999992, + "self_cpu_time_total_us": 5.711999999999989, + "device_time_total_us": 229.72800000000188, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 67895296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 25.19999999999999, + "self_cpu_time_total_us": 6.064000000000021, + "device_time_total_us": 229.72800000000188, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 19.135999999999967, + "self_cpu_time_total_us": 5.7439999999999145, + "device_time_total_us": 229.72800000000188, + "self_device_time_total_us": 229.72800000000188, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 229.72800000000188, + "self_device_time_total_us": 229.72800000000188, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 67.92000000000002, + "self_cpu_time_total_us": 12.048000000000059, + "device_time_total_us": 5.5039999999926295, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.567999999999984, + "self_cpu_time_total_us": 17.52000000000004, + "device_time_total_us": 5.5039999999926295, + "self_device_time_total_us": 5.5039999999926295, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.064000000002125, + "self_device_time_total_us": 4.064000000002125, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.831999999999937, + "self_cpu_time_total_us": 10.287999999999954, + "device_time_total_us": 3.9679999999880238, + "self_device_time_total_us": 3.9679999999880238, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.9679999999880238, + "self_device_time_total_us": 3.9679999999880238, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 23.85600000000005, + "self_cpu_time_total_us": 2.1440000000000623, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 2.9760000000005675, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 52.95999999999998, + "self_cpu_time_total_us": 6.751999999999953, + "device_time_total_us": 2.527999999992062, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.527999999992062, + "self_device_time_total_us": 2.527999999992062, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 129138.23, + "self_cpu_time_total_us": 129138.23, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 87.95199999999986, + "self_cpu_time_total_us": 87.95199999999986, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 64.624, + "self_cpu_time_total_us": 64.624, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.767999999999972, + "self_cpu_time_total_us": 20.767999999999972, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1423507456, + "self_device_memory_usage_bytes": 1423507456, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 18.303999999999974, + "self_cpu_time_total_us": 18.303999999999974, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.960000000000036, + "self_cpu_time_total_us": 14.960000000000036, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.696000000000012, + "self_cpu_time_total_us": 5.744000000000042, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.423999999999978, + "self_cpu_time_total_us": 6.447999999999979, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.760000000000048, + "self_cpu_time_total_us": 4.048000000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4720000000000937, + "self_cpu_time_total_us": 3.4720000000000937, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.5120000000000573, + "self_cpu_time_total_us": 1.8400000000001455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.2880000000000678, + "self_cpu_time_total_us": 2.2880000000000678, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.9519999999999698, + "self_cpu_time_total_us": 1.9519999999999698, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.712000000000046, + "self_cpu_time_total_us": 1.712000000000046, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1200000000001182, + "self_cpu_time_total_us": 1.1200000000001182, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.9759999999999991, + "self_cpu_time_total_us": 0.9759999999999991, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.9279999999999973, + "self_cpu_time_total_us": 0.9279999999999973, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6719999999999118, + "self_cpu_time_total_us": 0.6719999999999118, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1423509504, + "self_device_memory_usage_bytes": -1423509504, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc2", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 14336, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 7168 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 896 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 14336 + ], + "input_is_contiguous": true, + "input_stride": [ + 14336, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 4.836788866668939e-06, + "p50_s": 4.592002369463444e-06, + "p90_s": 5.012779729440807e-06, + "p95_s": 5.97038597334176e-06, + "p99_s": 6.736470968462527e-06, + "min_s": 4.415982402861118e-06, + "max_s": 6.927992217242718e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 8.192012319341303e-06, + "p50_s": 8.055998478084803e-06, + "p90_s": 8.614402031525969e-06, + "p95_s": 9.075223351828752e-06, + "p99_s": 9.443880408070982e-06, + "min_s": 7.823982741683722e-06, + "max_s": 9.536044672131538e-06 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.124792758375406e-06, + "p50_s": 4.0800077840685844e-06, + "p90_s": 4.228804027661681e-06, + "p95_s": 4.322393215261399e-06, + "p99_s": 4.397264565341175e-06, + "min_s": 3.999972250312567e-06, + "max_s": 4.415982402861118e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.004220321995671838, + "p50_s": 0.004209934995742515, + "p90_s": 0.004265264485729858, + "p95_s": 0.004276431250036694, + "p99_s": 0.004285364661482162, + "min_s": 0.004186158999800682, + "max_s": 0.00428759801434353 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 1.2371200136840344e-05, + "p50_s": 1.1800002539530396e-05, + "p90_s": 1.2766424333676694e-05, + "p95_s": 1.5135228750295932e-05, + "p99_s": 1.703027228359133e-05, + "min_s": 1.1391995940357447e-05, + "max_s": 1.7504033166915178e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.1740811169147492e-05, + "p50_s": 1.096801133826375e-05, + "p90_s": 1.2745597632601854e-05, + "p95_s": 1.530880981590598e-05, + "p99_s": 1.7359379562549293e-05, + "min_s": 1.057604094967246e-05, + "max_s": 1.787202199921012e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.005958578817080706, + "p50_s": 0.005956517998129129, + "p90_s": 0.00597937082638964, + "p95_s": 0.0059890044096391655, + "p99_s": 0.0059967112762387845, + "min_s": 0.005931598017923534, + "max_s": 0.005998637992888689 + }, + "params_wrap": { + "count": 10, + "mean_s": 6.75680348649621e-06, + "p50_s": 6.704009138047695e-06, + "p90_s": 7.038423791527748e-06, + "p95_s": 7.175205973908305e-06, + "p99_s": 7.284631719812751e-06, + "min_s": 6.52798917144537e-06, + "max_s": 7.311988156288862e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.010239630402065814, + "p50_s": 0.010224309022305533, + "p90_s": 0.010317761672195048, + "p95_s": 0.010330742830410598, + "p99_s": 0.010341127756983043, + "min_s": 0.010191901004873216, + "max_s": 0.010343723988626152 + }, + "linear": { + "count": 10, + "mean_s": 0.054607604414923114, + "p50_s": 0.054261310026049614, + "p90_s": 0.055418743623886255, + "p95_s": 0.05567892282269895, + "p99_s": 0.05588706618174911, + "min_s": 0.05413044604938477, + "max_s": 0.055939102021511644 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.523198680952191e-06, + "p50_s": 6.432004738599062e-06, + "p90_s": 6.795220542699098e-06, + "p95_s": 7.061607902869582e-06, + "p99_s": 7.27471779100597e-06, + "min_s": 6.304006092250347e-06, + "max_s": 7.327995263040066e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 14336 + ], + "input_bytes": 1084088320, + "packed_qdata_bytes": 271122432, + "block_scale_bytes": 33890304, + "output_bytes": 406533120, + "separate_materialization_bytes": 1795634176, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1423510528, + "negative_self_device_allocated_bytes": -1423510528, + "trace_path": "/output/h3-blackwell-runtime/benchmarks/traces/gb10-nvfp4-component2/mlp_fc2-vortex_native.json", + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 682.9750000000001, + "self_cpu_time_total_us": 108.75199999999995, + "device_time_total_us": 109631.81999999998, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 562.2230000000002, + "self_cpu_time_total_us": 466.54300000000023, + "device_time_total_us": 109631.81999999998, + "self_device_time_total_us": 109628.044, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 109628.044, + "self_device_time_total_us": 109628.044, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 12120.637000000006, + "self_device_time_total_us": 12120.637000000006, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 8392.425000000003, + "self_device_time_total_us": 8392.425000000003, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 52.2879999999999, + "self_cpu_time_total_us": 10.62399999999991, + "device_time_total_us": 252.70399999999972, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 67895296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 37.039999999999964, + "self_cpu_time_total_us": 9.631999999999948, + "device_time_total_us": 252.70399999999972, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 27.408000000000015, + "self_cpu_time_total_us": 12.912000000000035, + "device_time_total_us": 252.70399999999972, + "self_device_time_total_us": 252.70399999999972, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 252.70399999999972, + "self_device_time_total_us": 252.70399999999972, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 216.4319999999998, + "self_cpu_time_total_us": 27.855999999999938, + "device_time_total_us": 5.471999999992477, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 136.63999999999993, + "self_cpu_time_total_us": 102.27200000000022, + "device_time_total_us": 5.471999999992477, + "self_device_time_total_us": 5.471999999992477, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.936000000004242, + "self_device_time_total_us": 3.936000000004242, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 44.59199999999976, + "self_cpu_time_total_us": 26.0799999999997, + "device_time_total_us": 3.775999999992564, + "self_device_time_total_us": 3.775999999992564, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.775999999992564, + "self_device_time_total_us": 3.775999999992564, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 34.94400000000019, + "self_cpu_time_total_us": 2.496000000000322, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.9760000000005675, + "self_device_time_total_us": 2.9760000000005675, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 195.51999999999978, + "self_cpu_time_total_us": 11.53599999999986, + "device_time_total_us": 2.495999999991909, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.495999999991909, + "self_device_time_total_us": 2.495999999991909, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 129225.382, + "self_cpu_time_total_us": 129225.382, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 142.22399999999948, + "self_cpu_time_total_us": 142.22399999999948, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 62.70400000000001, + "self_cpu_time_total_us": 62.70400000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 51.93599999999992, + "self_cpu_time_total_us": 51.93599999999992, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 33.408000000000015, + "self_cpu_time_total_us": 33.408000000000015, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1423507456, + "self_device_memory_usage_bytes": 1423507456, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 16.368000000000166, + "self_cpu_time_total_us": 16.368000000000166, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.784000000000049, + "self_cpu_time_total_us": 8.656000000000205, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 13.199999999999818, + "self_cpu_time_total_us": 13.199999999999818, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 12.0, + "self_cpu_time_total_us": 9.200000000000273, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.127999999999844, + "self_cpu_time_total_us": 6.127999999999844, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.296000000000049, + "self_cpu_time_total_us": 5.296000000000049, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.616000000000099, + "self_cpu_time_total_us": 2.240000000000123, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.799999999999727, + "self_cpu_time_total_us": 2.799999999999727, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.5920000000000982, + "self_cpu_time_total_us": 1.7440000000001419, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7440000000001419, + "self_cpu_time_total_us": 1.7440000000001419, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.456000000000131, + "self_cpu_time_total_us": 1.456000000000131, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3759999999999764, + "self_cpu_time_total_us": 1.3759999999999764, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.8479999999999563, + "self_cpu_time_total_us": 0.8479999999999563, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1423509504, + "self_device_memory_usage_bytes": -1423509504, + "flops": 0, + "input_shapes": "[]" + } + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json b/benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json new file mode 100644 index 0000000..8c40719 --- /dev/null +++ b/benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 320, + 192 + ], + "frames": 22, + "steps": 1, + "feature": "lora_producer_fusion", + "seed": 440420, + "baseline_seconds": 3.3671130429720506, + "candidate_seconds": 0.32701997098047286, + "improvement_percent": 90.28782322402155, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + 911.742431640625, + 45.64133834838867 + ], + "candidate_checksums": [ + 911.742431640625, + 45.64133834838867 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json b/benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json new file mode 100644 index 0000000..3a3fbad --- /dev/null +++ b/benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 4, + "feature": "lora_producer_fusion", + "seed": 440420, + "baseline_seconds": 131.10563724400708, + "candidate_seconds": 135.1392796059954, + "improvement_percent": -3.076635335276312, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + -135415.75, + 1111.0089111328125 + ], + "candidate_checksums": [ + -135415.75, + 1111.0089111328125 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json b/benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json new file mode 100644 index 0000000..fa1835a --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json @@ -0,0 +1,26 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 12, + "seed": 440420, + "baseline_seconds": 285.99590051098494, + "candidate_seconds": 284.4078275830252, + "improvement_percent": 0.5552782138213641, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + -91481.7890625, + 678.401611328125 + ], + "candidate_checksums": [ + -91481.7890625, + 678.401611328125 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json b/benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json new file mode 100644 index 0000000..0f7187b --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json @@ -0,0 +1,26 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 2, + "seed": 440420, + "baseline_seconds": 47.66834012803156, + "candidate_seconds": 47.41923755296739, + "improvement_percent": 0.5225744684944189, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + -276716.375, + 727.1075439453125 + ], + "candidate_checksums": [ + -276716.375, + 727.1075439453125 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json b/benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json new file mode 100644 index 0000000..a3f1779 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 0, + "feature": "modulate_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": 2624555264.0, + "candidate_checksum": 2624555264.0, + "baseline": { + "mean_s": 0.4574462401214987, + "p50_s": 0.45709861497743987, + "min_s": 0.45598882500780746, + "max_s": 0.46002755500376225 + }, + "ring": { + "mean_s": 0.4554249476204859, + "p50_s": 0.4558210720133502, + "min_s": 0.4539778209873475, + "max_s": 0.4566468709963374 + }, + "p50_improvement_percent": 0.2794895723218782 +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json b/benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json new file mode 100644 index 0000000..518b066 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "feature": "modulate_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 6, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -168955712.0, + "candidate_checksum": -168955712.0, + "baseline": { + "mean_s": 0.4642839096680594, + "p50_s": 0.4646846964897122, + "min_s": 0.4618866750388406, + "max_s": 0.4662657259614207 + }, + "ring": { + "mean_s": 0.4616632471734192, + "p50_s": 0.461888676014496, + "min_s": 0.45918042300036177, + "max_s": 0.46357238601194695 + }, + "p50_improvement_percent": 0.6017027236613792 +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json b/benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json new file mode 100644 index 0000000..0aa1690 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 49, + "feature": "modulate_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -2184959754240.0, + "candidate_checksum": -2184959754240.0, + "baseline": { + "mean_s": 0.4588216957417899, + "p50_s": 0.4588029959995765, + "min_s": 0.45711035997373983, + "max_s": 0.4599904809729196 + }, + "ring": { + "mean_s": 0.4552568598664948, + "p50_s": 0.4551602325227577, + "min_s": 0.4534932450042106, + "max_s": 0.4569288089987822 + }, + "p50_improvement_percent": 0.7939711616055223 +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json b/benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json new file mode 100644 index 0000000..2b34d4a --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json @@ -0,0 +1,38 @@ +{ + "device": "NVIDIA GB10", + "deployment": { + "compose_project": "h3-blackwell-runtime", + "service": "h3-hot-runtime", + "image": "sha256:5f879c43374bcedb89745971d7d95d82afc8fcf9c41d30f11b257c2863b9fe28", + "feature_flag": "H3_NVFP4_MODULATE_FUSION=1", + "state": "running", + "health_ready": true + }, + "startup": { + "warmup_request_seconds": 30.185598155134358, + "note": "First startup warmup includes one-time CUDA extension compilation." + }, + "resident_smoke": { + "prompt": "A small orange cat walks across a sunlit wooden floor, natural motion, static camera.", + "output": "/output/h3-blackwell-runtime/deployed-nvfp4-modulate-fusion-smoke.mp4", + "width": 320, + "height": 192, + "frames": 22, + "steps": 1, + "seed": 440420, + "attention": "sage2", + "text_conditioned_seconds": 1.4317435850389302, + "sampled_seconds": 0.227207261021249, + "vae_decoded_seconds": 0.28855295496759936, + "request_seconds": 2.1424294949974865, + "wall_seconds": 2.1437057349830866, + "media_validation": { + "codec": "h264", + "width": 320, + "height": 192, + "frames": 22, + "duration_seconds": 0.916667 + } + }, + "result": "pass" +} diff --git a/benchmarks/gb10-nvfp4-modulate-fusion-summary.json b/benchmarks/gb10-nvfp4-modulate-fusion-summary.json new file mode 100644 index 0000000..9d5a6f5 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-fusion-summary.json @@ -0,0 +1,45 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "seed": 440420, + "attention": "sage2" + }, + "feature_flag": "H3_NVFP4_MODULATE_FUSION=1", + "implementation": "Exact BF16 AdaLN modulation recomputed in Vortex global-scale and native QDATA/SFA pack passes; Comfy NVFP4 GEMMs unchanged", + "producer_validation": { + "blocks": [0, 24, 49], + "rows": 37810, + "tensor_scale_equal": true, + "qdata_differences": 0, + "sfa_differences": 0, + "block_24_producer_p50_ms": 6.018611951731145 + }, + "alternating_block_p50_improvement_percent": { + "block_0": 0.2794895723218782, + "block_24": 0.6017027236613792, + "block_49": 0.7939711616055223 + }, + "warmed_trajectory": { + "two_step": { + "baseline_seconds": 47.66834012803156, + "candidate_seconds": 47.41923755296739, + "improvement_percent": 0.5225744684944189, + "video_equal": true, + "audio_equal": true + }, + "twelve_step": { + "baseline_seconds": 285.99590051098494, + "candidate_seconds": 284.4078275830252, + "improvement_percent": 0.5552782138213641, + "video_checksum": -91481.7890625, + "audio_checksum": 678.401611328125, + "video_equal": true, + "audio_equal": true + } + }, + "decision": "Enable for Spark single-GPU deployments. Distributed, chunked-MLP, full-precision, pre-scale, autograd, and active-LoRA cases retain materialized fallback." +} diff --git a/benchmarks/gb10-nvfp4-modulate-producer-block0-full.json b/benchmarks/gb10-nvfp4-modulate-producer-block0-full.json new file mode 100644 index 0000000..01991d0 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-producer-block0-full.json @@ -0,0 +1,14 @@ +{ + "device": "NVIDIA GB10", + "block_index": 0, + "rows": 37810, + "width": 5376, + "scale_equal": true, + "scale_reference": 0.00390625, + "scale_fused": 0.00390625, + "qdata_equal": true, + "qdata_differences": 0, + "sfa_equal": true, + "sfa_differences": 0, + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json b/benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json new file mode 100644 index 0000000..6ddc292 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json @@ -0,0 +1,14 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "rows": 2048, + "width": 5376, + "scale_equal": true, + "scale_reference": 0.0205078125, + "scale_fused": 0.0205078125, + "qdata_equal": true, + "qdata_differences": 0, + "sfa_equal": true, + "sfa_differences": 0, + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json b/benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json new file mode 100644 index 0000000..a9a9397 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json @@ -0,0 +1,15 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "rows": 37810, + "width": 5376, + "scale_equal": true, + "scale_reference": 0.0205078125, + "scale_fused": 0.0205078125, + "qdata_equal": true, + "qdata_differences": 0, + "sfa_equal": true, + "sfa_differences": 0, + "fused_producer_p50_ms": 6.018611951731145, + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-producer-block24-full.json b/benchmarks/gb10-nvfp4-modulate-producer-block24-full.json new file mode 100644 index 0000000..bb0174e --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-producer-block24-full.json @@ -0,0 +1,14 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "rows": 37810, + "width": 5376, + "scale_equal": true, + "scale_reference": 0.0205078125, + "scale_fused": 0.0205078125, + "qdata_equal": true, + "qdata_differences": 0, + "sfa_equal": true, + "sfa_differences": 0, + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-modulate-producer-block49-full.json b/benchmarks/gb10-nvfp4-modulate-producer-block49-full.json new file mode 100644 index 0000000..e9c98b8 --- /dev/null +++ b/benchmarks/gb10-nvfp4-modulate-producer-block49-full.json @@ -0,0 +1,14 @@ +{ + "device": "NVIDIA GB10", + "block_index": 49, + "rows": 37810, + "width": 5376, + "scale_equal": true, + "scale_reference": 0.0625, + "scale_fused": 0.0625, + "qdata_equal": true, + "qdata_differences": 0, + "sfa_equal": true, + "sfa_differences": 0, + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json b/benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json new file mode 100644 index 0000000..db860b0 --- /dev/null +++ b/benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json @@ -0,0 +1,2816 @@ +{ + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 10, + "profiler_iterations": 2, + "quantizers": [ + "vortex_native" + ], + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "results": [ + { + "name": "attn_qkv_proj", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 21504, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 21504, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 21504, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 6.052799290046096e-06, + "p50_s": 5.256006261333823e-06, + "p90_s": 9.011215297505258e-06, + "p95_s": 9.889612556435166e-06, + "p99_s": 1.0592330363579095e-05, + "min_s": 4.047993570566177e-06, + "max_s": 1.0768009815365076e-05 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 1.6932794824242593e-05, + "p50_s": 1.1727999662980437e-05, + "p90_s": 3.8022396620363e-05, + "p95_s": 3.869918291456997e-05, + "p99_s": 3.924061194993555e-05, + "min_s": 7.951981388032436e-06, + "max_s": 3.937596920877695e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 3.7984049413353204e-06, + "p50_s": 3.816006937995553e-06, + "p90_s": 3.96320829167962e-06, + "p95_s": 4.013604484498501e-06, + "p99_s": 4.053921438753605e-06, + "min_s": 3.5040429793298244e-06, + "max_s": 4.064000677317381e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0024285033985506745, + "p50_s": 0.0025771994842216372, + "p90_s": 0.002898441592697054, + "p95_s": 0.0030190847872290758, + "p99_s": 0.0031155993428546936, + "min_s": 0.0016482880455441773, + "max_s": 0.003139727981761098 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 2.7616001898422837e-05, + "p50_s": 2.454398781992495e-05, + "p90_s": 5.4996821563690905e-05, + "p95_s": 5.516241653822362e-05, + "p99_s": 5.5294892517849804e-05, + "min_s": 1.02399499155581e-05, + "max_s": 5.532801151275635e-05 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.5809608157724142e-05, + "p50_s": 1.5712023014202714e-05, + "p90_s": 2.2017589071765545e-05, + "p95_s": 2.2168803843669595e-05, + "p99_s": 2.2289775661192834e-05, + "min_s": 9.231967851519585e-06, + "max_s": 2.2320018615573645e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.0028375787020195275, + "p50_s": 0.002920447994256392, + "p90_s": 0.0030217359191738067, + "p95_s": 0.003200979460962116, + "p99_s": 0.0033443742943927647, + "min_s": 0.002199806971475482, + "max_s": 0.0033802230027504265 + }, + "params_wrap": { + "count": 10, + "mean_s": 7.174396887421608e-06, + "p50_s": 6.7280197981745005e-06, + "p90_s": 8.566409815102815e-06, + "p95_s": 9.24319610930979e-06, + "p99_s": 9.784625144675375e-06, + "min_s": 6.127986125648022e-06, + "max_s": 9.91998240351677e-06 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.005351344519294798, + "p50_s": 0.0055619110353291035, + "p90_s": 0.006045710993930697, + "p95_s": 0.006317583000054583, + "p99_s": 0.006535080604953692, + "min_s": 0.003900239011272788, + "max_s": 0.0065894550061784685 + }, + "linear": { + "count": 10, + "mean_s": 0.18048321470851078, + "p50_s": 0.08908659202279523, + "p90_s": 0.3484924691321794, + "p95_s": 0.38921985907072654, + "p99_s": 0.4218017710215645, + "min_s": 0.02535125898430124, + "max_s": 0.42994724900927395 + }, + "slice_reshape": { + "count": 10, + "mean_s": 1.6507197869941592e-05, + "p50_s": 7.480004569515586e-06, + "p90_s": 2.6808015536516882e-05, + "p95_s": 5.6436000159010215e-05, + "p99_s": 8.0138387857005e-05, + "min_s": 5.759997293353081e-06, + "max_s": 8.606398478150368e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12730368, + "output_bytes": 1626132480, + "separate_materialization_bytes": 2147066880, + "epilogue_candidate": "QKV views/layout and Q/K normalization/RoPE" + }, + "profiler_summary": { + "device_kernel_launches": 0, + "runtime_kernel_launches": 0, + "unique_device_kernels": 0, + "positive_self_device_allocated_bytes": 3482274816, + "negative_self_device_allocated_bytes": -3482274816, + "profiled_iterations": 2, + "trace_path": null, + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3133.4399999999987, + "self_cpu_time_total_us": 797.7919999999976, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2144.351999999999, + "self_cpu_time_total_us": 1963.8880000000026, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3253469184, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1378.5920000000042, + "self_cpu_time_total_us": 874.9120000000039, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1296.912000000002, + "self_cpu_time_total_us": 19.023999999999432, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1074.0629999999983, + "self_cpu_time_total_us": 364.5109999999986, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 25460736, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 919.5669999999991, + "self_cpu_time_total_us": 911.1669999999995, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 691.119999999999, + "self_cpu_time_total_us": 636.5280000000021, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 343.82400000000416, + "self_cpu_time_total_us": 343.82400000000416, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 191.2960000000021, + "self_cpu_time_total_us": 184.49600000000282, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 159.85599999999613, + "self_cpu_time_total_us": 159.85599999999613, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 108.78399999999965, + "self_cpu_time_total_us": 8.079999999998108, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 105.44000000000233, + "self_cpu_time_total_us": 105.44000000000233, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 101.19999999999345, + "self_cpu_time_total_us": 101.19999999999345, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 3482271744, + "self_device_memory_usage_bytes": 3482271744, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 54.591999999996915, + "self_cpu_time_total_us": 54.591999999996915, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.711999999999534, + "self_cpu_time_total_us": 8.272000000000844, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.11200000000099, + "self_cpu_time_total_us": 10.11200000000099, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.399999999999636, + "self_cpu_time_total_us": 8.399999999999636, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.351999999998952, + "self_cpu_time_total_us": 5.328000000001339, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.43999999999869, + "self_cpu_time_total_us": 7.43999999999869, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.799999999999272, + "self_cpu_time_total_us": 6.799999999999272, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.824000000000524, + "self_cpu_time_total_us": 5.824000000000524, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.9200000000018917, + "self_cpu_time_total_us": 3.9200000000018917, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.0239999999976135, + "self_cpu_time_total_us": 3.0239999999976135, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -3482273792, + "self_device_memory_usage_bytes": -3482273792, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "attn_out_proj", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 7168, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 3584 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 448 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 7168 + ], + "input_is_contiguous": true, + "input_stride": [ + 7168, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.295989103615284e-06, + "p50_s": 4.7759676817804575e-06, + "p90_s": 7.12319160811603e-06, + "p95_s": 7.425594958476721e-06, + "p99_s": 7.667517638765274e-06, + "min_s": 4.368019290268421e-06, + "max_s": 7.727998308837414e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 9.599997429177166e-06, + "p50_s": 8.68001370690763e-06, + "p90_s": 1.0455993469804522e-05, + "p95_s": 1.3947999104857437e-05, + "p99_s": 1.674160361289978e-05, + "min_s": 7.839989848434925e-06, + "max_s": 1.7440004739910364e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.123203689232469e-06, + "p50_s": 4.064000677317381e-06, + "p90_s": 4.3728097807616e-06, + "p95_s": 4.466398968361318e-06, + "p99_s": 4.5412703184410926e-06, + "min_s": 3.9199949242174625e-06, + "max_s": 4.559988155961037e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0021523084957152604, + "p50_s": 0.0021477669943124056, + "p90_s": 0.0021769711805973203, + "p95_s": 0.0021807655895827336, + "p99_s": 0.0021838011167710646, + "min_s": 0.0021312159951776266, + "max_s": 0.0021845599985681474 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 0.0002433982852380723, + "p50_s": 1.305597834289074e-05, + "p90_s": 0.0010138942569028586, + "p95_s": 0.0011852826195536178, + "p99_s": 0.0013223933096742258, + "min_s": 1.1071970220655203e-05, + "max_s": 0.0013566709822043777 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.2624007649719714e-05, + "p50_s": 1.104001421481371e-05, + "p90_s": 1.4812796143814918e-05, + "p95_s": 2.0126372692175197e-05, + "p99_s": 2.437723393086344e-05, + "min_s": 1.0512012522667646e-05, + "max_s": 2.5439949240535498e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.003013088909210637, + "p50_s": 0.002953759511001408, + "p90_s": 0.003206658235285431, + "p95_s": 0.0032714726112317294, + "p99_s": 0.003323324111988768, + "min_s": 0.00292299099965021, + "max_s": 0.0033362869871780276 + }, + "params_wrap": { + "count": 10, + "mean_s": 8.95520788617432e-06, + "p50_s": 6.960006430745125e-06, + "p90_s": 9.78240277618169e-06, + "p95_s": 1.8379196990281324e-05, + "p99_s": 2.525663236156106e-05, + "min_s": 6.27199187874794e-06, + "max_s": 2.697599120438099e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.0054626068915240465, + "p50_s": 0.005206414993153885, + "p90_s": 0.006140521290944889, + "p95_s": 0.006342604130622931, + "p99_s": 0.0065042704023653645, + "min_s": 0.005111566977575421, + "max_s": 0.0065446869703009725 + }, + "linear": { + "count": 10, + "mean_s": 0.008279907511314378, + "p50_s": 0.008283590519567952, + "p90_s": 0.008627811732003466, + "p95_s": 0.008640072858543135, + "p99_s": 0.008649881759774871, + "min_s": 0.007697679044213146, + "max_s": 0.008652333985082805 + }, + "slice_reshape": { + "count": 10, + "mean_s": 6.809615297242999e-06, + "p50_s": 6.5600324887782335e-06, + "p90_s": 7.382442709058522e-06, + "p95_s": 8.275220170617101e-06, + "p99_s": 8.989442139863968e-06, + "min_s": 6.127986125648022e-06, + "max_s": 9.167997632175684e-06 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 7168 + ], + "input_bytes": 542044160, + "packed_qdata_bytes": 135561216, + "block_scale_bytes": 16973824, + "output_bytes": 406533120, + "separate_materialization_bytes": 1101112320, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "runtime_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1118440448, + "negative_self_device_allocated_bytes": -1118440448, + "profiled_iterations": 2, + "trace_path": null, + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 308.0639999999994, + "self_cpu_time_total_us": 54.4159999999988, + "device_time_total_us": 15811.336000000001, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 246.5920000000001, + "self_cpu_time_total_us": 196.46399999999994, + "device_time_total_us": 15811.336000000001, + "self_device_time_total_us": 15806.536, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 15806.536, + "self_device_time_total_us": 15806.536, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 6025.104000000001, + "self_device_time_total_us": 6025.104000000001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4324.332, + "self_device_time_total_us": 4324.332, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.744000000000142, + "self_cpu_time_total_us": 6.144000000000233, + "device_time_total_us": 82.94400000000041, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 33947648, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.783999999999878, + "self_cpu_time_total_us": 6.479999999999791, + "device_time_total_us": 82.94400000000041, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.304000000000087, + "self_cpu_time_total_us": 5.824000000000069, + "device_time_total_us": 82.94400000000041, + "self_device_time_total_us": 82.94400000000041, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 82.94400000000041, + "self_device_time_total_us": 82.94400000000041, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 79.83999999999992, + "self_cpu_time_total_us": 16.62400000000025, + "device_time_total_us": 5.824000000001888, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 43.24800000000005, + "self_cpu_time_total_us": 21.07199999999989, + "device_time_total_us": 5.824000000001888, + "self_device_time_total_us": 5.824000000001888, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 21.808000000000447, + "self_cpu_time_total_us": 11.903999999999996, + "device_time_total_us": 4.800000000002001, + "self_device_time_total_us": 4.800000000002001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.800000000002001, + "self_device_time_total_us": 4.800000000002001, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.032000000000153, + "self_device_time_total_us": 4.032000000000153, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 27.50400000000036, + "self_cpu_time_total_us": 2.4000000000005457, + "device_time_total_us": 3.2960000000002765, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.2960000000002765, + "self_device_time_total_us": 3.2960000000002765, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 63.4879999999998, + "self_cpu_time_total_us": 8.751999999999697, + "device_time_total_us": 2.5280000000016116, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.5280000000016116, + "self_device_time_total_us": 2.5280000000016116, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24126.972, + "self_cpu_time_total_us": 24126.972, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1180.512, + "self_cpu_time_total_us": 1180.512, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 77.74400000000082, + "self_cpu_time_total_us": 77.74400000000082, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 21.792000000000144, + "self_cpu_time_total_us": 21.792000000000144, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1118437376, + "self_device_memory_usage_bytes": 1118437376, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 19.96799999999962, + "self_cpu_time_total_us": 19.96799999999962, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.47999999999982, + "self_cpu_time_total_us": 6.687999999999647, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.111999999999625, + "self_cpu_time_total_us": 8.111999999999625, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.056000000000495, + "self_cpu_time_total_us": 5.296000000000731, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.096000000000004, + "self_cpu_time_total_us": 2.800000000000182, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.8960000000001855, + "self_cpu_time_total_us": 2.8960000000001855, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.7679999999995744, + "self_cpu_time_total_us": 1.9679999999998472, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.127999999999929, + "self_cpu_time_total_us": 2.127999999999929, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7920000000001721, + "self_cpu_time_total_us": 1.7920000000001721, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7599999999997635, + "self_cpu_time_total_us": 1.7599999999997635, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7280000000000655, + "self_cpu_time_total_us": 1.7280000000000655, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.2959999999998217, + "self_cpu_time_total_us": 1.2959999999998217, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1840000000001965, + "self_cpu_time_total_us": 1.1840000000001965, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7999999999997272, + "self_cpu_time_total_us": 0.7999999999997272, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1118439424, + "self_device_memory_usage_bytes": -1118439424, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc1", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 5376, + "out_features": 28672, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 28672, + 2688 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 28672, + 336 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 5376 + ], + "input_is_contiguous": true, + "input_stride": [ + 5376, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.855993367731571e-06, + "p50_s": 5.0559756346046925e-06, + "p90_s": 7.955211913213134e-06, + "p95_s": 8.18558328319341e-06, + "p99_s": 8.36988037917763e-06, + "min_s": 4.496017936617136e-06, + "max_s": 8.415954653173685e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 7.972479797899724e-05, + "p50_s": 9.687995770946145e-06, + "p90_s": 8.677759324200428e-05, + "p95_s": 0.000390660794801078, + "p99_s": 0.0006337673560483382, + "min_s": 8.128001354634762e-06, + "max_s": 0.000694543996360153 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.559999797493219e-06, + "p50_s": 4.111992893740535e-06, + "p90_s": 4.9424183089286075e-06, + "p95_s": 6.303194095380601e-06, + "p99_s": 7.391814724542201e-06, + "min_s": 3.968016244471073e-06, + "max_s": 7.6639698818326e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.0016238380863796919, + "p50_s": 0.0016251999768428504, + "p90_s": 0.0016435487137641757, + "p95_s": 0.001646701854770072, + "p99_s": 0.0016492243675747887, + "min_s": 0.001595902955159545, + "max_s": 0.0016498549957759678 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 0.0022213820077013223, + "p50_s": 1.616799272596836e-05, + "p90_s": 0.010254466830519959, + "p95_s": 0.01038646443921607, + "p99_s": 0.010492062526172958, + "min_s": 1.166400033980608e-05, + "max_s": 0.01051846204791218 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.7052795737981796e-05, + "p50_s": 1.2168020475655794e-05, + "p90_s": 2.627038047648966e-05, + "p95_s": 3.2023168751038596e-05, + "p99_s": 3.662539937067777e-05, + "min_s": 1.047999830916524e-05, + "max_s": 3.777595702558756e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.002223322878126055, + "p50_s": 0.0022234079951886088, + "p90_s": 0.002246807888150215, + "p95_s": 0.0022486434201709924, + "p99_s": 0.0022501118457876144, + "min_s": 0.002194703964050859, + "max_s": 0.00225047895219177 + }, + "params_wrap": { + "count": 10, + "mean_s": 1.3769592624157666e-05, + "p50_s": 7.775990525260568e-06, + "p90_s": 2.7236773166805506e-05, + "p95_s": 2.7474400121718644e-05, + "p99_s": 2.7664501685649154e-05, + "min_s": 6.383983418345451e-06, + "max_s": 2.7712027076631784e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.006139096606057137, + "p50_s": 0.003941255999961868, + "p90_s": 0.014208465768024326, + "p95_s": 0.014347231370629742, + "p99_s": 0.014458243852714076, + "min_s": 0.0038519350346177816, + "max_s": 0.01448599697323516 + }, + "linear": { + "count": 10, + "mean_s": 0.05182686430052854, + "p50_s": 0.05186239897739142, + "p90_s": 0.06942712950403802, + "p95_s": 0.06958699877432081, + "p99_s": 0.06971489419054706, + "min_s": 0.033370858000125736, + "max_s": 0.06974686804460362 + }, + "slice_reshape": { + "count": 10, + "mean_s": 7.93120707385242e-06, + "p50_s": 6.83200778439641e-06, + "p90_s": 1.0513601591810583e-05, + "p95_s": 1.0592784383334219e-05, + "p99_s": 1.0656130616553127e-05, + "min_s": 6.176007445901632e-06, + "max_s": 1.0671967174857855e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 5376 + ], + "input_bytes": 406533120, + "packed_qdata_bytes": 101670912, + "block_scale_bytes": 12730368, + "output_bytes": 2168176640, + "separate_materialization_bytes": 2689111040, + "epilogue_candidate": "BF16 split and SwiGLU input" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "runtime_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 4566764544, + "negative_self_device_allocated_bytes": -4566764544, + "profiled_iterations": 2, + "trace_path": null, + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 473.4720000000002, + "self_cpu_time_total_us": 84.49600000000032, + "device_time_total_us": 66153.844, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 378.624, + "self_cpu_time_total_us": 306.89600000000087, + "device_time_total_us": 66153.844, + "self_device_time_total_us": 66149.84399999998, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4337958912, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 66149.84399999998, + "self_device_time_total_us": 66149.84399999998, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4390.723000000004, + "self_device_time_total_us": 4390.723000000004, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3205.122000000003, + "self_device_time_total_us": 3205.122000000003, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.680000000000064, + "self_cpu_time_total_us": 7.919999999999845, + "device_time_total_us": 44.16099999999824, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 25460736, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 26.480000000000246, + "self_cpu_time_total_us": 6.048000000000002, + "device_time_total_us": 44.16099999999824, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.432000000000244, + "self_cpu_time_total_us": 8.512000000000398, + "device_time_total_us": 44.16099999999824, + "self_device_time_total_us": 44.16099999999824, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 44.16099999999824, + "self_device_time_total_us": 44.16099999999824, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 173.23199999999997, + "self_cpu_time_total_us": 20.352000000000544, + "device_time_total_us": 5.5679999999993015, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 60.223999999999705, + "self_cpu_time_total_us": 35.82399999999939, + "device_time_total_us": 5.5679999999993015, + "self_device_time_total_us": 5.5679999999993015, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 28.527999999999793, + "self_cpu_time_total_us": 16.751999999999953, + "device_time_total_us": 4.0000000000045475, + "self_device_time_total_us": 4.0000000000045475, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 4.0000000000045475, + "self_device_time_total_us": 4.0000000000045475, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.87199999999757, + "self_device_time_total_us": 3.87199999999757, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 29.488000000000284, + "self_cpu_time_total_us": 2.4160000000001673, + "device_time_total_us": 3.0720000000001164, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.0720000000001164, + "self_device_time_total_us": 3.0720000000001164, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 154.1919999999999, + "self_cpu_time_total_us": 8.032000000000039, + "device_time_total_us": 2.495999999999185, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 2.495999999999185, + "self_device_time_total_us": 2.495999999999185, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 72879.795, + "self_cpu_time_total_us": 72879.795, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1075.264, + "self_cpu_time_total_us": 1075.264, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 105.66399999999953, + "self_cpu_time_total_us": 105.66399999999953, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 92.65599999999972, + "self_cpu_time_total_us": 92.65599999999972, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 30.7199999999998, + "self_cpu_time_total_us": 30.7199999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 4566761472, + "self_device_memory_usage_bytes": 4566761472, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.328000000000202, + "self_cpu_time_total_us": 11.328000000000202, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.32799999999989, + "self_cpu_time_total_us": 7.471999999999582, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.351999999999862, + "self_cpu_time_total_us": 7.568000000000211, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.00799999999981, + "self_cpu_time_total_us": 9.00799999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.344000000000051, + "self_cpu_time_total_us": 3.3920000000000528, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.415999999999713, + "self_cpu_time_total_us": 4.415999999999713, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.856000000000307, + "self_cpu_time_total_us": 3.856000000000307, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.7839999999996508, + "self_cpu_time_total_us": 2.7839999999996508, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.399999999999636, + "self_cpu_time_total_us": 1.6159999999995307, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.9519999999999982, + "self_cpu_time_total_us": 1.9519999999999982, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7279999999996107, + "self_cpu_time_total_us": 1.7279999999996107, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.231999999999971, + "self_cpu_time_total_us": 1.231999999999971, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7840000000001055, + "self_cpu_time_total_us": 0.7840000000001055, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -4566763520, + "self_device_memory_usage_bytes": -4566763520, + "flops": 0, + "input_shapes": "[]" + } + ] + }, + { + "name": "mlp_fc2", + "quantizer": "vortex_native", + "module": { + "class": "Nvfp4Linear", + "in_features": 14336, + "out_features": 5376, + "output_dtype": "torch.bfloat16", + "full_precision_matrix_mult": false, + "weight_dtype": "torch.uint8", + "weight_shape": [ + 5376, + 7168 + ], + "weight_scale_dtype": "torch.float8_e4m3fn", + "weight_scale_shape": [ + 5376, + 896 + ], + "weight_scale_2_dtype": "torch.float32", + "weight_scale_2_shape": [], + "bias_dtype": null, + "bias_shape": null, + "pre_quant_scale": false, + "input_dtype": "torch.bfloat16", + "input_shape": [ + 37810, + 14336 + ], + "input_is_contiguous": true, + "input_stride": [ + 14336, + 1 + ] + }, + "stage_timings": { + "flatten_contiguous": { + "count": 10, + "mean_s": 5.215994315221906e-06, + "p50_s": 4.656001692637801e-06, + "p90_s": 7.521617226302623e-06, + "p95_s": 7.528794230893254e-06, + "p99_s": 7.5345358345657585e-06, + "min_s": 4.511966835707426e-06, + "max_s": 7.535971235483885e-06 + }, + "pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "packed_weight_wrapper": { + "count": 10, + "mean_s": 9.571202099323273e-06, + "p50_s": 8.464005077257752e-06, + "p90_s": 1.0403204942122099e-05, + "p95_s": 1.4737618039362121e-05, + "p99_s": 1.8205148517154158e-05, + "min_s": 7.888011168688536e-06, + "max_s": 1.9072031136602163e-05 + }, + "bias_cast": { + "count": 10, + "mean_s": 4.1168008465319875e-06, + "p50_s": 4.039990017190576e-06, + "p90_s": 4.36801346950233e-06, + "p95_s": 4.4400192564353345e-06, + "p99_s": 4.497623885981738e-06, + "min_s": 3.840017598122358e-06, + "max_s": 4.5120250433683395e-06 + }, + "vortex_absmax_scale": { + "count": 10, + "mean_s": 0.004272058588685468, + "p50_s": 0.004267550975782797, + "p90_s": 0.004282539803534746, + "p95_s": 0.0043181654065847395, + "p99_s": 0.0043466658890247345, + "min_s": 0.00424494402250275, + "max_s": 0.004353791009634733 + }, + "scale_compat_cast": { + "count": 10, + "mean_s": 0.001205106993438676, + "p50_s": 1.369600067846477e-05, + "p90_s": 0.002398188604274761, + "p95_s": 0.006418293298338531, + "p99_s": 0.009634377053589561, + "min_s": 1.1087977327406406e-05, + "max_s": 0.010438397992402315 + }, + "scale_to_device": { + "count": 10, + "mean_s": 1.4732795534655452e-05, + "p50_s": 1.103198155760765e-05, + "p90_s": 2.4928024504333732e-05, + "p95_s": 3.0112022068351492e-05, + "p99_s": 3.425922011956573e-05, + "min_s": 1.0255957022309303e-05, + "max_s": 3.529601963236928e-05 + }, + "vortex_quantize_nvfp4": { + "count": 10, + "mean_s": 0.006082749110646546, + "p50_s": 0.00607146299444139, + "p90_s": 0.006121344619896263, + "p95_s": 0.006154759810306132, + "p99_s": 0.006181491962634027, + "min_s": 0.006053054006770253, + "max_s": 0.006188175000716001 + }, + "params_wrap": { + "count": 10, + "mean_s": 1.0276801185682416e-05, + "p50_s": 6.840011337772012e-06, + "p90_s": 2.2163224639371033e-05, + "p95_s": 2.4265589308924968e-05, + "p99_s": 2.5947481044568122e-05, + "min_s": 6.495974957942963e-06, + "max_s": 2.636795397847891e-05 + }, + "activation_quantize": { + "count": 10, + "mean_s": 0.01161861231084913, + "p50_s": 0.010411958530312404, + "p90_s": 0.012876597791910168, + "p95_s": 0.016871624905616035, + "p99_s": 0.020067646596580745, + "min_s": 0.010355374019127339, + "max_s": 0.02086665201932192 + }, + "linear": { + "count": 10, + "mean_s": 0.053046907490352166, + "p50_s": 0.0532312944997102, + "p90_s": 0.05354207805357874, + "p95_s": 0.05356652251211926, + "p99_s": 0.05358607807895169, + "min_s": 0.052244342979975045, + "max_s": 0.05359096697065979 + }, + "slice_reshape": { + "count": 10, + "mean_s": 7.1952003054320816e-06, + "p50_s": 6.583984941244125e-06, + "p90_s": 9.174406295642255e-06, + "p95_s": 9.707207209430634e-06, + "p99_s": 1.0133447940461337e-05, + "min_s": 6.320013199001551e-06, + "max_s": 1.0240008123219013e-05 + } + }, + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "intermediate_traffic": { + "logical_rows": 37810, + "padded_activation_shape": [ + 37824, + 14336 + ], + "input_bytes": 1084088320, + "packed_qdata_bytes": 271122432, + "block_scale_bytes": 33947648, + "output_bytes": 406533120, + "separate_materialization_bytes": 1795691520, + "epilogue_candidate": "segmented BF16 residual gate/add" + }, + "profiler_summary": { + "device_kernel_launches": 16, + "runtime_kernel_launches": 16, + "unique_device_kernels": 8, + "positive_self_device_allocated_bytes": 1423510528, + "negative_self_device_allocated_bytes": -1423510528, + "profiled_iterations": 2, + "trace_path": null, + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements." + }, + "profiler_top_events": [ + { + "key": "aten::linear", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 629.8879999999997, + "self_cpu_time_total_us": 107.98399999999992, + "device_time_total_us": 104850.064, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 507.5999999999997, + "self_cpu_time_total_us": 417.79199999999855, + "device_time_total_us": 104850.064, + "self_device_time_total_us": 104846.256, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 813367296, + "self_device_memory_usage_bytes": -1024, + "flops": 0, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "cutlass3x_sm120_bstensorop_s16864gemm_block_scaled_ue4m3xe2m1_ue4m3xe2m1_f32_bf16_bf16_128x128x256_1x1x1_0_tnn_align32_o_vs16_bias_bf16_relu_stream_k", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 104846.256, + "self_device_time_total_us": 104846.256, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::quantize_nvfp4_bf16_kernel(unsigned short const*, float const*, unsigned char*, unsigned char*, long, long, long, long, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 11854.592, + "self_device_time_total_us": 11854.592, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::partial_absmax_bf16_vec_kernel(uint4 const*, unsigned short const*, float*, long, long)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 8564.225999999995, + "self_device_time_total_us": 8564.225999999995, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::zeros", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 55.69599999999991, + "self_cpu_time_total_us": 10.943999999999505, + "device_time_total_us": 232.73600000000624, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 67895296, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 39.552000000000135, + "self_cpu_time_total_us": 8.368000000000166, + "device_time_total_us": 232.73600000000624, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 31.18399999999997, + "self_cpu_time_total_us": 9.375999999999749, + "device_time_total_us": 232.73600000000624, + "self_device_time_total_us": 232.73600000000624, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::FillFunctor, std::array >(int, at::native::FillFunctor, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 232.73600000000624, + "self_device_time_total_us": 232.73600000000624, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::_to_copy", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 201.6319999999996, + "self_cpu_time_total_us": 33.75999999999931, + "device_time_total_us": 9.823999999983243, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 81.3760000000002, + "self_cpu_time_total_us": 45.71200000000067, + "device_time_total_us": 9.823999999983243, + "self_device_time_total_us": 9.823999999983243, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::to", + "count": 6, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 172.0959999999998, + "self_cpu_time_total_us": 10.303999999999888, + "device_time_total_us": 6.783999999990556, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array >(int, at::native::bfloat16_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda(float)#1}, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 6.783999999990556, + "self_device_time_total_us": 6.783999999990556, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "(anonymous namespace)::final_scale_warp_kernel(float const*, float*, long, float)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.9040000000049986, + "self_device_time_total_us": 3.9040000000049986, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::mul", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 35.136000000000195, + "self_cpu_time_total_us": 18.80000000000041, + "device_time_total_us": 3.808000000006359, + "self_device_time_total_us": 3.808000000006359, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 1024, + "flops": 2, + "input_shapes": "[[], []]" + }, + { + "key": "void at::native::vectorized_elementwise_kernel<4, at::native::BinaryFunctor >, std::array >(int, at::native::BinaryFunctor >, std::array)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.808000000006359, + "self_device_time_total_us": 3.808000000006359, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 42.57600000000002, + "self_cpu_time_total_us": 2.736000000000331, + "device_time_total_us": 3.0399999999926877, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1024, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)", + "count": 2, + "device_type": "DeviceType.CUDA", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 3.0399999999926877, + "self_device_time_total_us": 3.0399999999926877, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaDeviceSynchronize", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 124293.91399999999, + "self_cpu_time_total_us": 124293.91399999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "Activity Buffer Request", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 924.192, + "self_cpu_time_total_us": 924.192, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "cudaLaunchKernel", + "count": 14, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 139.695999999999, + "self_cpu_time_total_us": 139.695999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::empty_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 86.4960000000001, + "self_cpu_time_total_us": 86.4960000000001, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 2048, + "self_device_memory_usage_bytes": 2048, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 30.992000000000644, + "self_cpu_time_total_us": 30.992000000000644, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 1423507456, + "self_device_memory_usage_bytes": 1423507456, + "flops": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "cuLaunchKernelEx", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 17.29599999999982, + "self_cpu_time_total_us": 17.29599999999982, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 14.304000000000087, + "self_cpu_time_total_us": 11.215999999999894, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "cudaGetSymbolAddress", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 12.75200000000018, + "self_cpu_time_total_us": 12.75200000000018, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 12.255999999999688, + "self_cpu_time_total_us": 7.631999999999607, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 6.400000000000318, + "self_cpu_time_total_us": 3.840000000000373, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.80800000000022, + "self_cpu_time_total_us": 5.80800000000022, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.6240000000000805, + "self_cpu_time_total_us": 4.6240000000000805, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::reshape", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.119999999999891, + "self_cpu_time_total_us": 2.1920000000000073, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.088000000000193, + "self_cpu_time_total_us": 3.088000000000193, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.5599999999999454, + "self_cpu_time_total_us": 2.5599999999999454, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.288000000000011, + "self_cpu_time_total_us": 2.288000000000011, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.3920000000000528, + "self_cpu_time_total_us": 1.3920000000000528, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.9279999999998836, + "self_cpu_time_total_us": 0.9279999999998836, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": 0, + "self_device_memory_usage_bytes": 0, + "flops": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "[memory]", + "count": 10, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "self_cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "cpu_memory_usage_bytes": 0, + "self_cpu_memory_usage_bytes": 0, + "device_memory_usage_bytes": -1423509504, + "self_device_memory_usage_bytes": -1423509504, + "flops": 0, + "input_shapes": "[]" + } + ] + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json b/benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json new file mode 100644 index 0000000..a9bfd35 --- /dev/null +++ b/benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json @@ -0,0 +1,674 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "results": [ + { + "name": "attn_qkv_proj", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 245.29710325616531, + "effective_gbps_p95": 226.7388199302835, + "timing": { + "count": 40, + "mean_s": 0.002157641400117427, + "p50_s": 0.0021236875327304006, + "p90_s": 0.002282506227493286, + "p95_s": 0.002297508649644442, + "p99_s": 0.00230576561531052, + "min_s": 0.002105231978930533, + "max_s": 0.002310976036824286 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_qkv_proj", + "packer": "vortex_native", + "threads": 64, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 224.59107678952034, + "effective_gbps_p95": 207.23756769247012, + "timing": { + "count": 40, + "mean_s": 0.002354832128912676, + "p50_s": 0.002319479506695643, + "p90_s": 0.002483506331918761, + "p95_s": 0.0025137063988950104, + "p99_s": 0.0026119847997324542, + "min_s": 0.002309999952558428, + "max_s": 0.0026610869681462646 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_qkv_proj", + "packer": "vortex_native", + "threads": 128, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 233.2105432536058, + "effective_gbps_p95": 211.95950883121543, + "timing": { + "count": 40, + "mean_s": 0.002263313374714926, + "p50_s": 0.002233751496532932, + "p90_s": 0.002288319927174598, + "p95_s": 0.002457707148278132, + "p99_s": 0.0026292865216964853, + "min_s": 0.0022275829687714577, + "max_s": 0.002655663003679365 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_qkv_proj", + "packer": "vortex_native", + "threads": 256, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 238.55814010715088, + "effective_gbps_p95": 216.5766578813339, + "timing": { + "count": 40, + "mean_s": 0.002226993774820585, + "p50_s": 0.0021836789965163916, + "p90_s": 0.0023945495078805832, + "p95_s": 0.0024053118424490092, + "p99_s": 0.0024354551278520375, + "min_s": 0.0021724309772253036, + "max_s": 0.002446942962706089 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_qkv_proj", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 246.31591915364592, + "effective_gbps_p95": 245.46406404228864, + "timing": { + "count": 40, + "mean_s": 0.002115138196677435, + "p50_s": 0.0021149035019334406, + "p90_s": 0.0021202821750193833, + "p95_s": 0.0021222430339548734, + "p99_s": 0.002123958675074391, + "min_s": 0.0021011190256103873, + "max_s": 0.002125006983987987 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_out_proj", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 244.56265212477584, + "effective_gbps_p95": 231.34306753591417, + "timing": { + "count": 40, + "mean_s": 0.0028569515954586676, + "p50_s": 0.0028400869632605463, + "p90_s": 0.002908607915742323, + "p95_s": 0.00300237741030287, + "p99_s": 0.0030247071670601146, + "min_s": 0.002825614996254444, + "max_s": 0.0030301110236905515 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_out_proj", + "packer": "vortex_native", + "threads": 64, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 231.12093550926818, + "effective_gbps_p95": 211.18017623027535, + "timing": { + "count": 40, + "mean_s": 0.003069607949873898, + "p50_s": 0.0030052630172576755, + "p90_s": 0.0032192086800932884, + "p95_s": 0.0032890359900193287, + "p99_s": 0.0033825889090076085, + "min_s": 0.0029931350145488977, + "max_s": 0.003397358988877386 + }, + "qdata_diff": 0, + "block_scale_diff": 16827811 + }, + { + "name": "attn_out_proj", + "packer": "vortex_native", + "threads": 128, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 236.64753270567476, + "effective_gbps_p95": 219.20626346231876, + "timing": { + "count": 40, + "mean_s": 0.002979868765396532, + "p50_s": 0.00293507898459211, + "p90_s": 0.0031074534170329573, + "p95_s": 0.003168610189459286, + "p99_s": 0.003295062365941703, + "min_s": 0.002914640004746616, + "max_s": 0.003295599017292261 + }, + "qdata_diff": 0, + "block_scale_diff": 16827134 + }, + { + "name": "attn_out_proj", + "packer": "vortex_native", + "threads": 256, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 242.57225236552964, + "effective_gbps_p95": 223.35673896586212, + "timing": { + "count": 40, + "mean_s": 0.0029051131932646967, + "p50_s": 0.002863390982383862, + "p90_s": 0.0029911637655459346, + "p95_s": 0.003109730215510353, + "p99_s": 0.0032386753160972147, + "min_s": 0.002843327005393803, + "max_s": 0.0032468310091644526 + }, + "qdata_diff": 0, + "block_scale_diff": 16827064 + }, + { + "name": "attn_out_proj", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 250.80788235490002, + "effective_gbps_p95": 229.17894761840023, + "timing": { + "count": 40, + "mean_s": 0.002809190830157604, + "p50_s": 0.002769367507426068, + "p90_s": 0.0029243686178233477, + "p95_s": 0.0030307286389870563, + "p99_s": 0.003207745099207386, + "min_s": 0.002747935999650508, + "max_s": 0.0032453910098411143 + }, + "qdata_diff": 0, + "block_scale_diff": 16827062 + }, + { + "name": "mlp_fc1", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 243.50477927357494, + "effective_gbps_p95": 226.67511573686335, + "timing": { + "count": 40, + "mean_s": 0.0021565302042290567, + "p50_s": 0.002139318996341899, + "p90_s": 0.002206075092544779, + "p95_s": 0.002298154335585423, + "p99_s": 0.0023168863559840246, + "min_s": 0.002122094971127808, + "max_s": 0.002320768020581454 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "vortex_native", + "threads": 64, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 223.27653165470315, + "effective_gbps_p95": 203.9842303404134, + "timing": { + "count": 40, + "mean_s": 0.002379627717891708, + "p50_s": 0.00233313548960723, + "p90_s": 0.0025109710812103005, + "p95_s": 0.0025537974142935127, + "p99_s": 0.002575046133133583, + "min_s": 0.0023052149917930365, + "max_s": 0.0025864150375127792 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "vortex_native", + "threads": 128, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 231.32488856945838, + "effective_gbps_p95": 230.5624261809017, + "timing": { + "count": 40, + "mean_s": 0.0022529397654579954, + "p50_s": 0.002251960017019883, + "p90_s": 0.0022583077894523743, + "p95_s": 0.0022594071750063447, + "p99_s": 0.002268195869983174, + "min_s": 0.0022337600239552557, + "max_s": 0.0022697430104017258 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "vortex_native", + "threads": 256, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 237.7793630736635, + "effective_gbps_p95": 220.94531333027638, + "timing": { + "count": 40, + "mean_s": 0.0022103461771621367, + "p50_s": 0.00219083100091666, + "p90_s": 0.0022536566131748257, + "p95_s": 0.0023577526590088382, + "p99_s": 0.0024101987341418863, + "min_s": 0.0021838549873791635, + "max_s": 0.00241096003446728 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 244.1410580831981, + "effective_gbps_p95": 241.96521054631256, + "timing": { + "count": 40, + "mean_s": 0.00214694502792554, + "p50_s": 0.0021337435173336416, + "p90_s": 0.0021429663756862283, + "p95_s": 0.0021529309888137504, + "p99_s": 0.002424711784115061, + "min_s": 0.002115023962687701, + "max_s": 0.0025554709718562663 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 244.68703463081846, + "effective_gbps_p95": 233.4123192225869, + "timing": { + "count": 40, + "mean_s": 0.005744802793196868, + "p50_s": 0.00567728650639765, + "p90_s": 0.005920993315521627, + "p95_s": 0.005951521344832144, + "p99_s": 0.006018744396278634, + "min_s": 0.005660655035171658, + "max_s": 0.006024766014888883 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "vortex_native", + "threads": 64, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 224.47756060169795, + "effective_gbps_p95": 211.92747342574486, + "timing": { + "count": 40, + "mean_s": 0.006284207080898341, + "p50_s": 0.006188406521687284, + "p90_s": 0.00652997883153148, + "p95_s": 0.006554876427981072, + "p99_s": 0.006563341356813908, + "min_s": 0.006155902985483408, + "max_s": 0.006565663032233715 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "vortex_native", + "threads": 128, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 228.51826075047396, + "effective_gbps_p95": 213.8415197976548, + "timing": { + "count": 40, + "mean_s": 0.006181065877899528, + "p50_s": 0.006078982027247548, + "p90_s": 0.006448514922522008, + "p95_s": 0.006496205233270302, + "p99_s": 0.00660812895919662, + "min_s": 0.006062957982067019, + "max_s": 0.0066334379953332245 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "vortex_native", + "threads": 256, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 231.47246817495252, + "effective_gbps_p95": 214.90532206804656, + "timing": { + "count": 40, + "mean_s": 0.00611898949573515, + "p50_s": 0.006001397967338562, + "p90_s": 0.006412335694767534, + "p95_s": 0.00646404838480521, + "p99_s": 0.006495686792186461, + "min_s": 0.005980286980047822, + "max_s": 0.006505389988888055 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 238.58180014194556, + "effective_gbps_p95": 223.09015483782161, + "timing": { + "count": 40, + "mean_s": 0.005879163170175161, + "p50_s": 0.005822566512506455, + "p90_s": 0.006110190035542473, + "p95_s": 0.006226892446284182, + "p99_s": 0.006299175039166584, + "min_s": 0.005806254979688674, + "max_s": 0.006312365992926061 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json b/benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json new file mode 100644 index 0000000..634a563 --- /dev/null +++ b/benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json @@ -0,0 +1,290 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "hidden_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "results": [ + { + "name": "attn_qkv_proj", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 237.3554828038592, + "effective_gbps_p95": 221.54486834351053, + "timing": { + "count": 20, + "mean_s": 0.002216759545262903, + "p50_s": 0.0021947434870526195, + "p90_s": 0.0023419696139171716, + "p95_s": 0.002351371999247931, + "p99_s": 0.0023890071938512847, + "min_s": 0.0021435199887491763, + "max_s": 0.002398415992502123 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_qkv_proj", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 233.3057458114144, + "effective_gbps_p95": 212.10780665923392, + "timing": { + "count": 20, + "mean_s": 0.0022636363486526535, + "p50_s": 0.0022328399936668575, + "p90_s": 0.0024042752280365675, + "p95_s": 0.002455988811561838, + "p99_s": 0.0025236713525373486, + "min_s": 0.0021525119664147496, + "max_s": 0.0025405919877812266 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_out_proj", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 244.5536052248782, + "effective_gbps_p95": 231.08574698145156, + "timing": { + "count": 20, + "mean_s": 0.0028653202491113914, + "p50_s": 0.002840192028088495, + "p90_s": 0.002945154212648049, + "p95_s": 0.003005720642977394, + "p99_s": 0.003050372929428704, + "min_s": 0.0028018239536322653, + "max_s": 0.0030615360010415316 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "attn_out_proj", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 7168 + ], + "qdata_shape": [ + 37824, + 3584 + ], + "block_scale_shape": [ + 37888, + 448 + ], + "bytes_touched": 694579200, + "effective_gbps_p50": 206.63049000273654, + "effective_gbps_p95": 182.7584609572448, + "timing": { + "count": 20, + "mean_s": 0.003328758548013866, + "p50_s": 0.0033614555140957236, + "p90_s": 0.0037920629896689207, + "p95_s": 0.0038005310198059306, + "p99_s": 0.003804483005660586, + "min_s": 0.00287507101893425, + "max_s": 0.00380547100212425 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 188.369281060563, + "effective_gbps_p95": 165.52457219580273, + "timing": { + "count": 20, + "mean_s": 0.0028155810461612417, + "p50_s": 0.002765495504718274, + "p90_s": 0.003082494280533865, + "p95_s": 0.0031471726106246933, + "p99_s": 0.0031740097067086024, + "min_s": 0.0026715680141933262, + "max_s": 0.00318071898072958 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc1", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 5376 + ], + "qdata_shape": [ + 37824, + 2688 + ], + "block_scale_shape": [ + 37888, + 336 + ], + "bytes_touched": 520934400, + "effective_gbps_p50": 240.72231247788002, + "effective_gbps_p95": 218.16303897955598, + "timing": { + "count": 20, + "mean_s": 0.0022075875516748057, + "p50_s": 0.002164047007681802, + "p90_s": 0.002363825583597645, + "p95_s": 0.0023878215230070054, + "p99_s": 0.002408699474763125, + "min_s": 0.002137008006684482, + "max_s": 0.002413918962702155 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "ck", + "threads": null, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 198.68331940880012, + "effective_gbps_p95": 185.7911050344262, + "timing": { + "count": 20, + "mean_s": 0.007080884993774817, + "p50_s": 0.006991821981500834, + "p90_s": 0.007437778793973848, + "p95_s": 0.007476990891154857, + "p99_s": 0.007967294955160468, + "min_s": 0.006620269967243075, + "max_s": 0.008089870971161872 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + }, + { + "name": "mlp_fc2", + "packer": "vortex_native", + "threads": 512, + "input_shape": [ + 37810, + 14336 + ], + "qdata_shape": [ + 37824, + 7168 + ], + "block_scale_shape": [ + 37888, + 896 + ], + "bytes_touched": 1389158400, + "effective_gbps_p50": 183.27811590655392, + "effective_gbps_p95": 174.45709655535202, + "timing": { + "count": 20, + "mean_s": 0.00760674009216018, + "p50_s": 0.007579510478535667, + "p90_s": 0.007870518049458042, + "p95_s": 0.007962750885053538, + "p99_s": 0.008361125348019412, + "min_s": 0.007152206962928176, + "max_s": 0.008460718963760883 + }, + "qdata_diff": 0, + "block_scale_diff": 0 + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-nvfp4-streaming-feasibility.json b/benchmarks/gb10-nvfp4-streaming-feasibility.json new file mode 100644 index 0000000..49ecd44 --- /dev/null +++ b/benchmarks/gb10-nvfp4-streaming-feasibility.json @@ -0,0 +1,40 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "cuda": "13.0.88", + "comfy_kitchen": "0.2.31", + "cutlass_dsl": "4.6.2", + "existing_interface": { + "registered_schema": "scaled_mm_nvfp4(a, b, tensor_scale_a, tensor_scale_b, block_scale_a, block_scale_b, bias, output_dtype_code, alpha) -> Tensor", + "backend": "cublas_gemm_blockwise_fp4", + "allocates_complete_activation_qdata": true, + "allocates_complete_activation_block_scales": true, + "allocates_complete_bf16_output": true, + "accepts_incremental_tiles": false, + "accepts_user_output": false, + "accepts_custom_epilogue": false + }, + "owned_kernel_support": { + "sm121_blockscaled_nvfp4": true, + "persistent_cooperative_schedule": true, + "pipeline_primitives": true, + "custom_software_a_producer": "requires editing the CuTe DSL mainloop", + "cluster_shape": [1, 1, 1], + "activation_tile_multicast": false + }, + "stock_cute_baseline": { + "shape_mnkl": [1024, 1024, 1024, 1], + "tile_shape_mnk": [128, 128, 128], + "a_dtype": "Float4E2M1FN", + "b_dtype": "Float4E2M1FN", + "scale_dtype": "Float8E4M3FN", + "scale_vector_size": 16, + "accumulator_dtype": "Float32", + "output_dtype": "BFloat16", + "execution_microseconds": 16.966399550437927, + "tflops": 126.57273817087317, + "reference_check": "PASS" + }, + "decision": "Use a two-phase exact design: global BF16 scale reduction followed by an owned CuTe DSL GEMM with a software A/SFA producer. Measure N-tile reuse before expanding beyond one fixed tile.", + "design_document": "NVFP4_STREAMING_DESIGN.md" +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json b/benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json new file mode 100644 index 0000000..6160ad4 --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 12, + "feature": "swiglu_fusion", + "seed": 440420, + "baseline_seconds": 289.1363876980031, + "candidate_seconds": 277.362894070975, + "improvement_percent": 4.0719515522637195, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + -91481.7890625, + 678.401611328125 + ], + "candidate_checksums": [ + -91481.7890625, + 678.401611328125 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json b/benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json new file mode 100644 index 0000000..0a2f5ca --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json @@ -0,0 +1,27 @@ +{ + "device": "NVIDIA GB10", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 2, + "feature": "swiglu_fusion", + "seed": 440420, + "baseline_seconds": 47.50661772303283, + "candidate_seconds": 47.295541714003775, + "improvement_percent": 0.4443086440286015, + "video_equal": true, + "audio_equal": true, + "video_max_abs": 0.0, + "audio_max_abs": 0.0, + "reference_checksums": [ + -276716.375, + 727.1075439453125 + ], + "candidate_checksums": [ + -276716.375, + 727.1075439453125 + ], + "equal": true +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json b/benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json new file mode 100644 index 0000000..b65927e --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 0, + "feature": "swiglu_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": 2624555264.0, + "candidate_checksum": 2624555264.0, + "baseline": { + "mean_s": 0.4555916166282259, + "p50_s": 0.45540181902470067, + "min_s": 0.4543589469976723, + "max_s": 0.45712462498340756 + }, + "candidate": { + "mean_s": 0.4453073174881865, + "p50_s": 0.4456424615054857, + "min_s": 0.4429232379770838, + "max_s": 0.44757142796879634 + }, + "p50_improvement_percent": 2.1430211983157754 +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json b/benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json new file mode 100644 index 0000000..10c0bdf --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "feature": "swiglu_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -895169280.0, + "candidate_checksum": -895169280.0, + "baseline": { + "mean_s": 0.4611226721244748, + "p50_s": 0.4611434039834421, + "min_s": 0.4602645250270143, + "max_s": 0.4618963819812052 + }, + "candidate": { + "mean_s": 0.45035380200715736, + "p50_s": 0.4507396965054795, + "min_s": 0.4488199040060863, + "max_s": 0.4511376320151612 + }, + "p50_improvement_percent": 2.256067719519228 +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json b/benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json new file mode 100644 index 0000000..fa6f3da --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json @@ -0,0 +1,28 @@ +{ + "device": "NVIDIA GB10", + "block_index": 49, + "feature": "swiglu_fusion", + "hidden_shape": [ + 37810, + 5376 + ], + "iterations": 8, + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -2184959754240.0, + "candidate_checksum": -2184959754240.0, + "baseline": { + "mean_s": 0.4584760017605731, + "p50_s": 0.45892439200542867, + "min_s": 0.4551540020038374, + "max_s": 0.46055828605312854 + }, + "candidate": { + "mean_s": 0.44894643424777314, + "p50_s": 0.44890999447670765, + "min_s": 0.44649913802277297, + "max_s": 0.45112301100743935 + }, + "p50_improvement_percent": 2.182145404161162 +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json b/benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json new file mode 100644 index 0000000..0b39e06 --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json @@ -0,0 +1,43 @@ +{ + "device": "NVIDIA GB10", + "deployment": { + "compose_project": "h3-blackwell-runtime", + "service": "h3-hot-runtime", + "image": "sha256:b9e2949ad4acbacbda3a16e3e60667f476a6330d5763a2bdfbf9a98150a0babd", + "feature_flags": { + "H3_NVFP4_MODULATE_FUSION": "1", + "H3_NVFP4_SWIGLU_FUSION": "1" + }, + "state": "running", + "health_ready": true + }, + "startup": { + "warmup_request_seconds": 30.482452621974517, + "note": "First startup warmup includes one-time CUDA extension compilation." + }, + "resident_smoke": { + "output": "/output/h3-blackwell-runtime/deployed-nvfp4-swiglu-fusion-smoke.mp4", + "width": 320, + "height": 192, + "frames": 22, + "steps": 1, + "seed": 440420, + "attention": "sage2", + "text_conditioned_seconds": 1.4687969940132461, + "sampled_seconds": 0.2173400520114228, + "vae_decoded_seconds": 0.28646593599114567, + "request_seconds": 2.172695894027129, + "wall_seconds": 2.17405128700193, + "media_validation": { + "codec": "h264", + "width": 320, + "height": 192, + "frames": 22, + "duration_seconds": 0.916667 + } + }, + "tests": { + "nvfp4_cuda_tests": "3 passed" + }, + "result": "pass" +} diff --git a/benchmarks/gb10-nvfp4-swiglu-fusion-summary.json b/benchmarks/gb10-nvfp4-swiglu-fusion-summary.json new file mode 100644 index 0000000..11c645a --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-fusion-summary.json @@ -0,0 +1,46 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "seed": 440420, + "attention": "sage2", + "modulation_fusion_enabled": true + }, + "feature_flag": "H3_NVFP4_SWIGLU_FUSION=1", + "implementation": "Exact BF16 SwiGLU recomputed in Vortex global-scale and native QDATA/SFA pack passes; Comfy FC2 GEMM unchanged", + "producer": { + "input_shape": [37810, 28672], + "output_shape": [37810, 14336], + "tensor_scale_equal": true, + "qdata_differences": 0, + "sfa_differences": 0, + "p50_ms": 21.925064036622643 + }, + "alternating_block_p50_improvement_percent": { + "block_0": 2.1430211983157754, + "block_24": 2.256067719519228, + "block_49": 2.182145404161162 + }, + "warmed_trajectory": { + "two_step": { + "baseline_seconds": 47.50661772303283, + "candidate_seconds": 47.295541714003775, + "improvement_percent": 0.4443086440286015, + "video_equal": true, + "audio_equal": true + }, + "twelve_step": { + "baseline_seconds": 289.1363876980031, + "candidate_seconds": 277.362894070975, + "improvement_percent": 4.0719515522637195, + "video_checksum": -91481.7890625, + "audio_checksum": 678.401611328125, + "video_equal": true, + "audio_equal": true + } + }, + "decision": "Enable for Spark single-GPU deployments with materialized fallbacks for unsupported policies." +} diff --git a/benchmarks/gb10-nvfp4-swiglu-producer-block24.json b/benchmarks/gb10-nvfp4-swiglu-producer-block24.json new file mode 100644 index 0000000..9f8c59f --- /dev/null +++ b/benchmarks/gb10-nvfp4-swiglu-producer-block24.json @@ -0,0 +1,19 @@ +{ + "device": "NVIDIA GB10", + "block_index": 24, + "input_shape": [ + 37810, + 28672 + ], + "output_shape": [ + 37810, + 14336 + ], + "scale_equal": true, + "scale_reference": 2.03125, + "scale_actual": 2.03125, + "qdata_differences": 0, + "sfa_differences": 0, + "producer_p50_ms": 21.925064036622643, + "equal": true +} diff --git a/benchmarks/gb10-post-lora-revert-deployment-smoke.json b/benchmarks/gb10-post-lora-revert-deployment-smoke.json new file mode 100644 index 0000000..5ad2f47 --- /dev/null +++ b/benchmarks/gb10-post-lora-revert-deployment-smoke.json @@ -0,0 +1,43 @@ +{ + "device": "NVIDIA GB10", + "deployment": { + "compose_project": "h3-blackwell-runtime", + "service": "h3-hot-runtime", + "image": "sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83", + "feature_flags": { + "H3_NVFP4_MODULATE_FUSION": "1", + "H3_NVFP4_SWIGLU_FUSION": "1" + }, + "state": "running", + "health_ready": true + }, + "startup": { + "warmup_request_seconds": 30.730418314109556, + "vortex_extension_precompiled": true + }, + "resident_smoke": { + "output": "/output/h3-blackwell-runtime/post-lora-revert-smoke.mp4", + "width": 320, + "height": 192, + "frames": 22, + "steps": 1, + "seed": 440420, + "attention": "sage2", + "text_conditioned_seconds": 1.4340007840073667, + "sampled_seconds": 0.2167754319962114, + "vae_decoded_seconds": 0.2895864300080575, + "request_seconds": 2.3209229299682193, + "wall_seconds": 2.3224709409987554, + "media_validation": { + "codec": "h264", + "width": 320, + "height": 192, + "frames": 22, + "duration_seconds": 0.916667 + } + }, + "tests": { + "nvfp4_cuda_tests": "3 passed" + }, + "result": "pass" +} diff --git a/benchmarks/gb10-post-optimization-block24-profile.json b/benchmarks/gb10-post-optimization-block24-profile.json new file mode 100644 index 0000000..8f24482 --- /dev/null +++ b/benchmarks/gb10-post-optimization-block24-profile.json @@ -0,0 +1,1402 @@ +{ + "prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.", + "model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors", + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "warmup": 3, + "iterations": 10, + "hidden_shape": [ + 37810, + 5376 + ], + "output_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "timings": { + "norm1": { + "count": 10, + "mean_s": 0.003506062604719773, + "p50_s": 0.0034730469924397767, + "p90_s": 0.0035981596854981037, + "p95_s": 0.0036302793218055737, + "p99_s": 0.0036559750308515506, + "min_s": 0.0034213270409964025, + "max_s": 0.0036623989581130445 + }, + "modulate_msa": { + "count": 10, + "mean_s": 0.01051678290241398, + "p50_s": 0.010468507505720481, + "p90_s": 0.01089892717427574, + "p95_s": 0.010943725591641851, + "p99_s": 0.010979564325534739, + "min_s": 0.010146012005861849, + "max_s": 0.01098852400900796 + }, + "linear.attn_qkv_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.558393266052008e-06, + "p50_s": 4.208006430417299e-06, + "p90_s": 4.755187546834348e-06, + "p95_s": 6.281610694713887e-06, + "p99_s": 7.502749213017523e-06, + "min_s": 4.031986463814974e-06, + "max_s": 7.808033842593431e-06 + }, + "linear.attn_qkv_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_qkv_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 9.913597023114561e-06, + "p50_s": 7.968017598614097e-06, + "p90_s": 1.0451191337779158e-05, + "p95_s": 1.924959651660172e-05, + "p99_s": 2.6288320659659805e-05, + "min_s": 7.4239796958863735e-06, + "max_s": 2.804800169542432e-05 + }, + "linear.attn_qkv_proj.bias_cast": { + "count": 10, + "mean_s": 3.6815996281802655e-06, + "p50_s": 3.5040138754993677e-06, + "p90_s": 3.956805448979139e-06, + "p95_s": 4.554382758215069e-06, + "p99_s": 5.032444605603814e-06, + "min_s": 3.375986125320196e-06, + "max_s": 5.151960067451e-06 + }, + "linear.attn_qkv_proj.activation_scale": { + "count": 10, + "mean_s": 0.0017137959017418324, + "p50_s": 0.0016890474944375455, + "p90_s": 0.0017734934226609766, + "p95_s": 0.0017952662019524723, + "p99_s": 0.001812684425385669, + "min_s": 0.0016727829934097826, + "max_s": 0.001817038981243968 + }, + "linear.attn_qkv_proj.scale_to_device": { + "count": 10, + "mean_s": 1.1468783486634494e-05, + "p50_s": 9.303970728069544e-06, + "p90_s": 1.7495953943580386e-05, + "p95_s": 1.7675955314189194e-05, + "p99_s": 1.781995641067624e-05, + "min_s": 8.32002842798829e-06, + "max_s": 1.7855956684798002e-05 + }, + "linear.attn_qkv_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.0022020551143214105, + "p50_s": 0.002172503009205684, + "p90_s": 0.002327527030138299, + "p95_s": 0.002348515027551912, + "p99_s": 0.002365305425482802, + "min_s": 0.0021115349954925478, + "max_s": 0.0023695030249655247 + }, + "linear.attn_qkv_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.527994992211461e-06, + "p50_s": 6.503978511318564e-06, + "p90_s": 6.854429375380277e-06, + "p95_s": 7.243192521855234e-06, + "p99_s": 7.554203039035201e-06, + "min_s": 6.096030119806528e-06, + "max_s": 7.631955668330193e-06 + }, + "linear.attn_qkv_proj.gemm": { + "count": 10, + "mean_s": 0.02555141010088846, + "p50_s": 0.025579269480658695, + "p90_s": 0.026145932916551828, + "p95_s": 0.026412224461091684, + "p99_s": 0.026625257696723564, + "min_s": 0.024755333026405424, + "max_s": 0.026678516005631536 + }, + "linear.attn_qkv_proj.slice_reshape": { + "count": 10, + "mean_s": 5.6432094424963e-06, + "p50_s": 5.471985787153244e-06, + "p90_s": 5.9520069044083355e-06, + "p95_s": 6.744018173776566e-06, + "p99_s": 7.377627189271153e-06, + "min_s": 5.2480027079582214e-06, + "max_s": 7.536029443144798e-06 + }, + "attn_qkv_proj": { + "count": 10, + "mean_s": 0.029553097899770363, + "p50_s": 0.029554627515608445, + "p90_s": 0.03014536619302817, + "p95_s": 0.03036433259549085, + "p99_s": 0.030539505717461, + "min_s": 0.028794804005883634, + "max_s": 0.030583298997953534 + }, + "attn_qkv_split_view": { + "count": 10, + "mean_s": 1.2614397564902902e-05, + "p50_s": 1.1160009307786822e-05, + "p90_s": 1.3886386295780535e-05, + "p95_s": 1.877520990092306e-05, + "p99_s": 2.2686268785037104e-05, + "min_s": 1.0351999662816525e-05, + "max_s": 2.3664033506065607e-05 + }, + "attn_qk_rms_rope": { + "count": 10, + "mean_s": 0.012204991799080744, + "p50_s": 0.012200642988318577, + "p90_s": 0.01241076691658236, + "p95_s": 0.012442684968118557, + "p99_s": 0.012468219409347512, + "min_s": 0.011841004015877843, + "max_s": 0.01247460301965475 + }, + "attention_kernel": { + "count": 10, + "mean_s": 0.26835116609581744, + "p50_s": 0.25847244350006804, + "p90_s": 0.26903588619315993, + "p95_s": 0.31390352409216565, + "p99_s": 0.3497976344113704, + "min_s": 0.25696170097216964, + "max_s": 0.35877116199117154 + }, + "attn_output_reshape": { + "count": 10, + "mean_s": 4.254397936165333e-06, + "p50_s": 4.191999323666096e-06, + "p90_s": 4.472001455724239e-06, + "p95_s": 4.507991252467037e-06, + "p99_s": 4.5367830898612736e-06, + "min_s": 4.016037564724684e-06, + "max_s": 4.543981049209833e-06 + }, + "linear.attn_out_proj.flatten_contiguous": { + "count": 10, + "mean_s": 4.1199964471161366e-06, + "p50_s": 4.063971573486924e-06, + "p90_s": 4.222447751089931e-06, + "p95_s": 4.431235720403492e-06, + "p99_s": 4.598266095854342e-06, + "min_s": 3.983965143561363e-06, + "max_s": 4.640023689717054e-06 + }, + "linear.attn_out_proj.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.attn_out_proj.packed_weight_wrapper": { + "count": 10, + "mean_s": 9.252689778804779e-06, + "p50_s": 8.911971235647798e-06, + "p90_s": 9.937619324773548e-06, + "p95_s": 1.0808813385665413e-05, + "p99_s": 1.150576863437891e-05, + "min_s": 8.671951945871115e-06, + "max_s": 1.1680007446557283e-05 + }, + "linear.attn_out_proj.bias_cast": { + "count": 10, + "mean_s": 3.4816039260476828e-06, + "p50_s": 3.4720287658274174e-06, + "p90_s": 3.5679608117789033e-06, + "p95_s": 3.5679870052263142e-06, + "p99_s": 3.5680079599842427e-06, + "min_s": 3.3919932320713997e-06, + "max_s": 3.568013198673725e-06 + }, + "linear.attn_out_proj.activation_scale": { + "count": 10, + "mean_s": 0.0022514488082379103, + "p50_s": 0.002237695502117276, + "p90_s": 0.0022609557956457136, + "p95_s": 0.0023159494099672883, + "p99_s": 0.0023599443014245478, + "min_s": 0.0022299509728327394, + "max_s": 0.002370943024288863 + }, + "linear.attn_out_proj.scale_to_device": { + "count": 10, + "mean_s": 1.2071995297446846e-05, + "p50_s": 1.1911994079127908e-05, + "p90_s": 1.2502394383773208e-05, + "p95_s": 1.3395198038779196e-05, + "p99_s": 1.4109440962783993e-05, + "min_s": 1.1135998647660017e-05, + "max_s": 1.428800169378519e-05 + }, + "linear.attn_out_proj.activation_quant_pack": { + "count": 10, + "mean_s": 0.002843859599670395, + "p50_s": 0.0028387825295794755, + "p90_s": 0.0028701974020805213, + "p95_s": 0.002917386181070469, + "p99_s": 0.002955137204262428, + "min_s": 0.0028025589999742806, + "max_s": 0.0029645749600604177 + }, + "linear.attn_out_proj.activation_quant_wrap": { + "count": 10, + "mean_s": 6.420805584639311e-06, + "p50_s": 6.4480118453502655e-06, + "p90_s": 6.592011777684093e-06, + "p95_s": 6.664017564617097e-06, + "p99_s": 6.721622194163501e-06, + "min_s": 6.0800230130553246e-06, + "max_s": 6.736023351550102e-06 + }, + "linear.attn_out_proj.gemm": { + "count": 10, + "mean_s": 0.007799276692094281, + "p50_s": 0.007773573015583679, + "p90_s": 0.007835476868785917, + "p95_s": 0.00800838442519307, + "p99_s": 0.008146710470318794, + "min_s": 0.007710108999162912, + "max_s": 0.008181291981600225 + }, + "linear.attn_out_proj.slice_reshape": { + "count": 10, + "mean_s": 5.4480042308568954e-06, + "p50_s": 5.416019121184945e-06, + "p90_s": 5.580816650763155e-06, + "p95_s": 5.710395635105669e-06, + "p99_s": 5.814058822579682e-06, + "min_s": 5.296024028211832e-06, + "max_s": 5.839974619448185e-06 + }, + "attn_out_proj": { + "count": 10, + "mean_s": 0.012975754507351667, + "p50_s": 0.012935250502778217, + "p90_s": 0.013021398818818853, + "p95_s": 0.01331132840423379, + "p99_s": 0.013543272072565742, + "min_s": 0.012829803046770394, + "max_s": 0.01360125798964873 + }, + "gate_msa": { + "count": 10, + "mean_s": 0.008599226904334501, + "p50_s": 0.008538733003661036, + "p90_s": 0.0088796345226001, + "p95_s": 0.008909831769415177, + "p99_s": 0.008933989566867239, + "min_s": 0.00840773998061195, + "max_s": 0.008940029016230255 + }, + "norm2": { + "count": 10, + "mean_s": 0.0036033073149155824, + "p50_s": 0.0035515105118975043, + "p90_s": 0.0037464877241291106, + "p95_s": 0.003796427365159616, + "p99_s": 0.0038363790779840203, + "min_s": 0.0035186070017516613, + "max_s": 0.003846367006190121 + }, + "modulate_mlp": { + "count": 10, + "mean_s": 0.010574622795684264, + "p50_s": 0.010596931009786204, + "p90_s": 0.01087350157322362, + "p95_s": 0.010878188785864041, + "p99_s": 0.010881938555976375, + "min_s": 0.010094380006194115, + "max_s": 0.01088287599850446 + }, + "linear.mlp_fc1.flatten_contiguous": { + "count": 10, + "mean_s": 4.360004095360637e-06, + "p50_s": 4.327972419559956e-06, + "p90_s": 4.619208630174398e-06, + "p95_s": 4.669604822993279e-06, + "p99_s": 4.709921777248382e-06, + "min_s": 4.128029104322195e-06, + "max_s": 4.720001015812159e-06 + }, + "linear.mlp_fc1.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc1.packed_weight_wrapper": { + "count": 10, + "mean_s": 8.18080734461546e-06, + "p50_s": 8.128001354634762e-06, + "p90_s": 8.4448023699224e-06, + "p95_s": 8.718419121578335e-06, + "p99_s": 8.937312522903086e-06, + "min_s": 7.760012522339821e-06, + "max_s": 8.992035873234272e-06 + }, + "linear.mlp_fc1.bias_cast": { + "count": 10, + "mean_s": 3.4687924198806287e-06, + "p50_s": 3.455992555245757e-06, + "p90_s": 3.5727629438042644e-06, + "p95_s": 3.5943987313658e-06, + "p99_s": 3.6117073614150284e-06, + "min_s": 3.311957698315382e-06, + "max_s": 3.6160345189273357e-06 + }, + "linear.mlp_fc1.activation_scale": { + "count": 10, + "mean_s": 0.0017569320043548942, + "p50_s": 0.0017358949990011752, + "p90_s": 0.0018160303065087645, + "p95_s": 0.0018329426558921114, + "p99_s": 0.001846472535398789, + "min_s": 0.0017267030198127031, + "max_s": 0.001849855005275458 + }, + "linear.mlp_fc1.scale_to_device": { + "count": 10, + "mean_s": 1.0102399392053485e-05, + "p50_s": 9.367999155074358e-06, + "p90_s": 1.0531197767704722e-05, + "p95_s": 1.4289590762928117e-05, + "p99_s": 1.729630515910685e-05, + "min_s": 8.5439532995224e-06, + "max_s": 1.804798375815153e-05 + }, + "linear.mlp_fc1.activation_quant_pack": { + "count": 10, + "mean_s": 0.00217321349773556, + "p50_s": 0.0021457590046338737, + "p90_s": 0.0022131046222057193, + "p95_s": 0.00231779980531428, + "p99_s": 0.0024015559518011285, + "min_s": 0.0021100639714859426, + "max_s": 0.002422494988422841 + }, + "linear.mlp_fc1.activation_quant_wrap": { + "count": 10, + "mean_s": 6.355205550789833e-06, + "p50_s": 6.3680054154247046e-06, + "p90_s": 6.510427920147777e-06, + "p95_s": 6.5752043155953285e-06, + "p99_s": 6.62702543195337e-06, + "min_s": 6.128044333308935e-06, + "max_s": 6.639980711042881e-06 + }, + "linear.mlp_fc1.gemm": { + "count": 10, + "mean_s": 0.034308884787606075, + "p50_s": 0.03437204146757722, + "p90_s": 0.034552122588502245, + "p95_s": 0.034628701777546664, + "p99_s": 0.0346899651287822, + "min_s": 0.033615602005738765, + "max_s": 0.03470528096659109 + }, + "linear.mlp_fc1.slice_reshape": { + "count": 10, + "mean_s": 5.462393164634704e-06, + "p50_s": 5.399982910603285e-06, + "p90_s": 5.71361742913723e-06, + "p95_s": 5.86480600759387e-06, + "p99_s": 5.985756870359183e-06, + "min_s": 5.231995601207018e-06, + "max_s": 6.01599458605051e-06 + }, + "mlp_fc1": { + "count": 10, + "mean_s": 0.038320799911161885, + "p50_s": 0.03837749600643292, + "p90_s": 0.03857543897465803, + "p95_s": 0.03861309501517099, + "p99_s": 0.03864321984758135, + "min_s": 0.03753324900753796, + "max_s": 0.03865075105568394 + }, + "mlp_swiglu": { + "count": 10, + "mean_s": 0.023907500400673597, + "p50_s": 0.023899421998066828, + "p90_s": 0.02409050110145472, + "p95_s": 0.024095325541566126, + "p99_s": 0.02409918509365525, + "min_s": 0.023705605999566615, + "max_s": 0.024100149981677532 + }, + "linear.mlp_fc2.flatten_contiguous": { + "count": 10, + "mean_s": 4.129589069634676e-06, + "p50_s": 4.135974450036883e-06, + "p90_s": 4.268810153007507e-06, + "p95_s": 4.470394924283027e-06, + "p99_s": 4.631662741303444e-06, + "min_s": 3.871973603963852e-06, + "max_s": 4.671979695558548e-06 + }, + "linear.mlp_fc2.pre_quant_scale": { + "count": 10, + "mean_s": 0.0, + "p50_s": 0.0, + "p90_s": 0.0, + "p95_s": 0.0, + "p99_s": 0.0, + "min_s": 0.0, + "max_s": 0.0 + }, + "linear.mlp_fc2.packed_weight_wrapper": { + "count": 10, + "mean_s": 7.93120707385242e-06, + "p50_s": 7.888011168688536e-06, + "p90_s": 8.115218952298164e-06, + "p95_s": 8.417596109211444e-06, + "p99_s": 8.65949783474207e-06, + "min_s": 7.424037903547287e-06, + "max_s": 8.719973266124725e-06 + }, + "linear.mlp_fc2.bias_cast": { + "count": 10, + "mean_s": 3.532791743054986e-06, + "p50_s": 3.503984771668911e-06, + "p90_s": 3.6463839933276177e-06, + "p95_s": 3.7111865822225807e-06, + "p99_s": 3.7630286533385513e-06, + "min_s": 3.375986125320196e-06, + "max_s": 3.775989171117544e-06 + }, + "linear.mlp_fc2.activation_scale": { + "count": 10, + "mean_s": 0.004426897293888033, + "p50_s": 0.004421310004545376, + "p90_s": 0.004461953247664497, + "p95_s": 0.004538503623916767, + "p99_s": 0.004599743924918585, + "min_s": 0.004358205944299698, + "max_s": 0.004615054000169039 + }, + "linear.mlp_fc2.scale_to_device": { + "count": 10, + "mean_s": 1.4316797023639084e-05, + "p50_s": 9.480019798502326e-06, + "p90_s": 2.5710387853905554e-05, + "p95_s": 3.182317886967211e-05, + "p99_s": 3.671341168228537e-05, + "min_s": 8.736038580536842e-06, + "max_s": 3.793596988543868e-05 + }, + "linear.mlp_fc2.activation_quant_pack": { + "count": 10, + "mean_s": 0.005852458404842764, + "p50_s": 0.00577670949860476, + "p90_s": 0.006046092335600406, + "p95_s": 0.006052780675236135, + "p99_s": 0.006058131346944719, + "min_s": 0.005697708984371275, + "max_s": 0.0060594690148718655 + }, + "linear.mlp_fc2.activation_quant_wrap": { + "count": 10, + "mean_s": 6.2527891714125875e-06, + "p50_s": 6.199959898367524e-06, + "p90_s": 6.398366531357169e-06, + "p95_s": 6.679186481051146e-06, + "p99_s": 6.903842440806329e-06, + "min_s": 6.064015906304121e-06, + "max_s": 6.960006430745125e-06 + }, + "linear.mlp_fc2.gemm": { + "count": 10, + "mean_s": 0.05624732690048404, + "p50_s": 0.0563016329833772, + "p90_s": 0.05672116960049607, + "p95_s": 0.056825684782234015, + "p99_s": 0.056909296927624385, + "min_s": 0.05543639202369377, + "max_s": 0.05693019996397197 + }, + "linear.mlp_fc2.slice_reshape": { + "count": 10, + "mean_s": 5.435204366222024e-06, + "p50_s": 5.359994247555733e-06, + "p90_s": 5.553592927753925e-06, + "p95_s": 5.8487930800765744e-06, + "p99_s": 6.084953201934695e-06, + "min_s": 5.199981387704611e-06, + "max_s": 6.143993232399225e-06 + }, + "mlp_fc2": { + "count": 10, + "mean_s": 0.06660801698453725, + "p50_s": 0.06663533198297955, + "p90_s": 0.06701652241172269, + "p95_s": 0.06725638319621793, + "p99_s": 0.06744827182381413, + "min_s": 0.0657997639500536, + "max_s": 0.06749624398071319 + }, + "gate_mlp": { + "count": 10, + "mean_s": 0.010363909183070064, + "p50_s": 0.008649876486742869, + "p90_s": 0.010803180868970225, + "p95_s": 0.017989072922500765, + "p99_s": 0.023737786565325224, + "min_s": 0.008425659965723753, + "max_s": 0.025174964976031333 + }, + "block_total": { + "count": 10, + "mean_s": 0.499170052097179, + "p50_s": 0.4875695880036801, + "p90_s": 0.5122190653986763, + "p95_s": 0.5512197931850096, + "p99_s": 0.5824203754140763, + "min_s": 0.4862319889944047, + "max_s": 0.5902205209713429 + } + }, + "module_forward": { + "count": 10, + "mean_s": 0.4679248433036264, + "p50_s": 0.46821730851661414, + "p90_s": 0.46862109302892346, + "p95_s": 0.4686407850269461, + "p99_s": 0.46865653862536416, + "min_s": 0.4657743179704994, + "max_s": 0.4686604770249687 + }, + "module_forward_checksum": -23765568.0, + "fused_elementwise": true, + "profiler_summary": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 9093546496 + }, + "profiler_top_events": [ + { + "key": "aten::rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1284.5120000000002, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -303104, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::_fused_rms_norm", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1275.904, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [5376], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 253.10399999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [21504, 5376], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 204.17599999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 188.06399999999985, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]" + }, + { + "key": "comfy_kitchen::rms_rope_split_half_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 176.30299999999988, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]" + }, + { + "key": "aten::_to_copy", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 134.94399999999905, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 108.04799999999977, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [5376, 14336], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 102.22400000000016, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [5376, 7168], []]" + }, + { + "key": "aten::to", + "count": 12, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 101.503999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::empty", + "count": 34, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 100.47999999999882, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 8009298432, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::linear", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 96.33600000000024, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], [28672, 5376], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 92.97600000000011, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 85.71199999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]" + }, + { + "key": "comfy_kitchen::scaled_mm_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 83.24800000000005, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -512, + "input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]" + }, + { + "key": "aten::copy_", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 76.6719999999998, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], []]" + }, + { + "key": "aten::zeros", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 69.5039999999999, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], []]" + }, + { + "key": "aten::mean", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 64.27199999999993, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 14336, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 57.96799999999985, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], [], [], [], []]" + }, + { + "key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 51.983999999999924, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]" + }, + { + "key": "comfy_kitchen::quantize_nvfp4", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 49.69599999999991, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [], [], [], []]" + }, + { + "key": "aten::mul", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 47.5359999999996, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 2048, + "input_shapes": "[[], []]" + }, + { + "key": "aten::to", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 46.91199999999981, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::silu", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 33.74400000000014, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 1084227584, + "input_shapes": "[[37810, 14336]]" + }, + { + "key": "aten::mul_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 28.192000000000007, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], [37810, 14336]]" + }, + { + "key": "aten::zero_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 27.455999999999904, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336]]" + }, + { + "key": "aten::empty_strided", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 24.495999999999867, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 4096, + "input_shapes": "[[], [], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 20.159999999999854, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], [], []]" + }, + { + "key": "aten::split", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 19.759999999999764, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], []]" + }, + { + "key": "aten::fill_", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 15.248000000000047, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::chunk", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.583999999999833, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 11.376000000000204, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896]]" + }, + { + "key": "aten::narrow", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.959999999999582, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 10.032000000001062, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 9.83999999999969, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::split", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.079999999999927, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 8.016000000000076, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.6159999999999854, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 336], []]" + }, + { + "key": "aten::squeeze", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.472000000000207, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], []]" + }, + { + "key": "aten::zero_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 7.2159999999998945, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448]]" + }, + { + "key": "aten::reshape", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.664000000000442, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::narrow", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.6479999999996835, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::fill_", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.536000000000058, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::slice", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 5.023999999999887, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], [], []]" + }, + { + "key": "aten::to", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 4.480000000000047, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376], [], [], [], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.680000000000291, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4880000000000564, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810], []]" + }, + { + "key": "aten::slice", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 3.4400000000005093, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], [], []]" + }, + { + "key": "aten::view", + "count": 8, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.7360000000005584, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.512000000000171, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::slice", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.2719999999999345, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], [], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 2.2080000000000837, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504]]" + }, + { + "key": "aten::as_strided", + "count": 3, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.9360000000001492, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], [], [], []]" + }, + { + "key": "aten::view", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.7760000000000673, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.6320000000000618, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::as_strided", + "count": 4, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.5199999999999818, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], [], [], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1840000000001965, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 7168], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1520000000000437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1520000000000437, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::reshape", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 1.1359999999999673, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.83199999999988, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 5376], [], [], []]" + }, + { + "key": "aten::as_strided", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7039999999997235, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.7039999999997235, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 896], []]" + }, + { + "key": "aten::alias", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6720000000000255, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 5376]]" + }, + { + "key": "aten::view", + "count": 2, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.6399999999998727, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37888, 448], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.5920000000000982, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128]]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.4800000000000182, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37824, 28672], [], [], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.44800000000032014, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 37810, 56, 128], []]" + }, + { + "key": "aten::alias", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.41600000000016735, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672]]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.38400000000001455, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 21504], []]" + }, + { + "key": "aten::as_strided", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.36799999999993815, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[1, 1, 56, 128], [], [], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.3520000000003165, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 14336], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.35199999999986176, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[37810, 28672], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.3200000000001637, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[21504, 336], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.3040000000000873, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[28672, 336], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.30399999999963256, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 896], []]" + }, + { + "key": "aten::view", + "count": 1, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.2719999999999345, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": 0, + "input_shapes": "[[5376, 448], []]" + }, + { + "key": "[memory]", + "count": 33, + "device_type": "DeviceType.CPU", + "cpu_time_total_us": 0.0, + "device_time_total_us": 0.0, + "self_device_time_total_us": 0.0, + "self_device_memory_usage_bytes": -9093241344, + "input_shapes": "[]" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-post-optimization-profile-summary.json b/benchmarks/gb10-post-optimization-profile-summary.json new file mode 100644 index 0000000..920030b --- /dev/null +++ b/benchmarks/gb10-post-optimization-profile-summary.json @@ -0,0 +1,52 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "steps": 12, + "seed": 440420, + "attention": "sage2", + "production_flags": { + "H3_NVFP4_SCALE_BACKEND": "vortex", + "H3_NVFP4_SCALE_VERSION": "1", + "H3_FUSED_ELEMENTWISE": "1", + "H3_SAGE_QKV_LAYOUT": "strided_nhd", + "H3_CUTE_QKV_RING": "0" + } + }, + "canonical": { + "elapsed_seconds": 288.9257714879932, + "video_checksum": -91481.7890625, + "audio_checksum": 678.401611328125, + "peak_allocated_bytes": 16036566016, + "peak_reserved_bytes": 19484639232 + }, + "block_24_p50_ms": { + "module_forward": 468.21730851661414, + "attention": 258.47244350006804, + "qk_rmsnorm_rope": 12.200642988318577, + "qkv_projection": 29.554627515608445, + "attention_output_projection": 12.935250502778217, + "mlp_fc1": 38.37749600643292, + "mlp_swiglu": 23.899421998066828, + "mlp_fc2": 66.63533198297955, + "layout_materialization": 0.015351508744061 + }, + "sage2_nhd_p50_ms": { + "total": 260.0231445103418, + "qk_quantize": 7.684325042646378, + "v_quantize": 10.653970995917916, + "attention_mainloop": 238.8111045002006, + "framework_residual_estimate": 2.873743971576914 + }, + "active_mainloop": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "hopper_wgmma_retarget": { + "target": "sm_121a", + "result": "compile rejected", + "reason": "SM121 ptxas does not support Hopper wgmma fence, mma_async, commit_group, or wait_group instructions" + }, + "blackwell_umma_constraint": "CUTLASS SM120/121 supports F8/F6/F4 UMMA but not the INT8 QK operation required for exact Sage2 parity", + "decision": "Prototype exact AdaLN modulation directly into NVFP4 scale/pack for QKV and FC1. Retain Comfy GEMMs. Keep native Sage2 attention research paused unless the classic MMA schedule can be retuned without changing arithmetic." +} diff --git a/benchmarks/gb10-post-optimization-sage2-canonical-12step.json b/benchmarks/gb10-post-optimization-sage2-canonical-12step.json new file mode 100644 index 0000000..4b3cd7a --- /dev/null +++ b/benchmarks/gb10-post-optimization-sage2-canonical-12step.json @@ -0,0 +1,22 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 12, + "seed": 440420, + "text_tokens": 100, + "elapsed_seconds": 288.9257714879932, + "stage_trace": [], + "checksums": [ + -91481.7890625, + 678.401611328125 + ], + "peak_allocated_bytes": 16036566016, + "peak_reserved_bytes": 19484639232, + "measurement_policy": "uninstrumented sampling wall time" +} diff --git a/benchmarks/gb10-post-optimization-sage2-nhd-components.json b/benchmarks/gb10-post-optimization-sage2-nhd-components.json new file mode 100644 index 0000000..2708455 --- /dev/null +++ b/benchmarks/gb10-post-optimization-sage2-nhd-components.json @@ -0,0 +1,104 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "sequence": 37810, + "conditioning_stop": 514, + "warmup": 3, + "iterations": 10, + "measurement_policy": { + "component_timings": "CUDA-synchronized nested probes; attribution only", + "residual_timings": "difference of component p50 values; labeled estimates", + "end_to_end": "must be measured separately without component synchronization" + }, + "sage2": { + "version": "2.2.0", + "timings": { + "attention_forward": { + "count": 10, + "mean_s": 0.36555117740063, + "p50_s": 0.2388111045002006, + "p90_s": 0.4044706951128315, + "p95_s": 0.921806076535721, + "p99_s": 1.3356743816740346, + "min_s": 0.2377422249992378, + "max_s": 1.4391414579586126 + }, + "qk_quantize": { + "count": 10, + "mean_s": 0.008795083011500537, + "p50_s": 0.007684325042646378, + "p90_s": 0.010721880599157883, + "p95_s": 0.013224297293345438, + "p99_s": 0.015226230648695492, + "min_s": 0.00766948499949649, + "max_s": 0.015726713987533003 + }, + "total": { + "count": 10, + "mean_s": 0.39030183949507774, + "p50_s": 0.2600231445103418, + "p90_s": 0.4261628873937293, + "p95_s": 0.947522030692197, + "p99_s": 1.3646093453309733, + "min_s": 0.25837415299611166, + "max_s": 1.4688811739906669 + }, + "v_quantize": { + "count": 10, + "mean_s": 0.01344956418615766, + "p50_s": 0.010653970995917916, + "p90_s": 0.018150134984171014, + "p95_s": 0.02297045400191563, + "p99_s": 0.02682670921611134, + "min_s": 0.010557003959547728, + "max_s": 0.027790773019660264 + }, + "framework_and_k_smoothing_estimate": { + "count": 10, + "mean_s": 0.002873743971576914, + "p50_s": 0.002873743971576914, + "p90_s": 0.002873743971576914, + "p95_s": 0.002873743971576914, + "p99_s": 0.002873743971576914, + "min_s": 0.002873743971576914, + "max_s": 0.002873743971576914 + } + }, + "checksum": 9190064.0 + }, + "sol": [] +} diff --git a/benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json b/benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json new file mode 100644 index 0000000..fac9526 --- /dev/null +++ b/benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json @@ -0,0 +1,115 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "packed_tokens": 37810, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "attention": "sage2", + "fused_elementwise": true, + "warmup": 3, + "iterations": 10, + "results": [ + { + "block": 0, + "parity": { + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -8279139.0, + "candidate_checksum": -8279139.0 + }, + "hnd_timing": { + "count": 10, + "mean_s": 0.48708497728803196, + "p50_s": 0.4870640740264207, + "p90_s": 0.48792775589972737, + "p95_s": 0.488274192425888, + "p99_s": 0.4885513416468166, + "min_s": 0.48618641699431464, + "max_s": 0.4886206289520487 + }, + "strided_nhd_timing": { + "count": 10, + "mean_s": 0.471230285911588, + "p50_s": 0.4714779225178063, + "p90_s": 0.4723133562132716, + "p95_s": 0.4731061626109294, + "p99_s": 0.47374040772905573, + "min_s": 0.4685067450045608, + "max_s": 0.4738989690085873 + }, + "p50_speedup": 1.0330580728475696, + "p50_latency_reduction": 0.0320002076518765 + }, + { + "block": 24, + "parity": { + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": 89071408.0, + "candidate_checksum": 89071408.0 + }, + "hnd_timing": { + "count": 10, + "mean_s": 0.4880072570987977, + "p50_s": 0.4879330809926614, + "p90_s": 0.4888307601737324, + "p95_s": 0.48945235056162345, + "p99_s": 0.48994962287193633, + "min_s": 0.4864739279728383, + "max_s": 0.4900739409495145 + }, + "strided_nhd_timing": { + "count": 10, + "mean_s": 0.47193070990615527, + "p50_s": 0.4716692080255598, + "p90_s": 0.47343356219353155, + "p95_s": 0.4738115225860383, + "p99_s": 0.47411389090004374, + "min_s": 0.47014945396222174, + "max_s": 0.47418948297854513 + }, + "p50_speedup": 1.034481523683056, + "p50_latency_reduction": 0.033332179351344804 + }, + { + "block": 49, + "parity": { + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0, + "reference_checksum": -40617144320.0, + "candidate_checksum": -40617144320.0 + }, + "hnd_timing": { + "count": 10, + "mean_s": 0.4874046443030238, + "p50_s": 0.48766444699140266, + "p90_s": 0.48841265672817824, + "p95_s": 0.48861716688843443, + "p99_s": 0.4887807750166394, + "min_s": 0.4857746610068716, + "max_s": 0.4888216770486906 + }, + "strided_nhd_timing": { + "count": 10, + "mean_s": 0.47163516000728123, + "p50_s": 0.47158038747147657, + "p90_s": 0.47319388142204843, + "p95_s": 0.4735081272228854, + "p99_s": 0.47375952386355497, + "min_s": 0.46926438697846606, + "max_s": 0.4738223730237223 + }, + "p50_speedup": 1.0341067184879458, + "p50_latency_reduction": 0.032981816942274755 + } + ] +} diff --git a/benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json b/benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json new file mode 100644 index 0000000..a1c8970 --- /dev/null +++ b/benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json @@ -0,0 +1,254 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ], + "warmup": 3, + "iterations": 5, + "results": [ + { + "segment": "all", + "segment_tuple": null, + "backend": "sage2", + "layout_mode": "hnd", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "output_shape": [ + 37810, + 7168 + ], + "timings": { + "q_transpose_contiguous": { + "count": 5, + "mean_s": 0.005225043802056462, + "p50_s": 0.005116002983413637, + "p90_s": 0.005561528832186013, + "p95_s": 0.0055678074364550415, + "p99_s": 0.005572830319870263, + "min_s": 0.0048384820111095905, + "max_s": 0.005574086040724069 + }, + "k_transpose_contiguous": { + "count": 5, + "mean_s": 0.005020070588216186, + "p50_s": 0.004905874957330525, + "p90_s": 0.005268368811812252, + "p95_s": 0.005289322405587882, + "p99_s": 0.005306085280608386, + "min_s": 0.004802018986083567, + "max_s": 0.005310275999363512 + }, + "v_transpose_contiguous": { + "count": 5, + "mean_s": 0.004939833201933652, + "p50_s": 0.004868466989137232, + "p90_s": 0.005137888411991298, + "p95_s": 0.0051993222092278305, + "p99_s": 0.005248469247017056, + "min_s": 0.00475086597725749, + "max_s": 0.005260756006464362 + }, + "attention_kernel": { + "count": 5, + "mean_s": 0.2837455251952633, + "p50_s": 0.2643922639545053, + "p90_s": 0.32986050880281254, + "p95_s": 0.3455152763985097, + "p99_s": 0.3580390904750675, + "min_s": 0.2538328950176947, + "max_s": 0.36117004399420694 + }, + "output_reshape": { + "count": 5, + "mean_s": 0.0048204828053712845, + "p50_s": 0.004661265993490815, + "p90_s": 0.005151034612208605, + "p95_s": 0.005268343805801124, + "p99_s": 0.005362191160675139, + "min_s": 0.004617601982317865, + "max_s": 0.005385652999393642 + } + }, + "layout_attention_total_mean_s": 0.3037509555928409, + "reference_backend": "sage2:hnd", + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "status": "ok" + }, + { + "segment": "all", + "segment_tuple": null, + "backend": "sage2", + "layout_mode": "sage_nhd", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "output_shape": [ + 37810, + 7168 + ], + "timings": { + "q_nhd_contiguous": { + "count": 5, + "mean_s": 0.005327214219141752, + "p50_s": 0.005574870039708912, + "p90_s": 0.005795779020991177, + "p95_s": 0.005863789014983922, + "p99_s": 0.005918197010178119, + "min_s": 0.004763827018905431, + "max_s": 0.005931799008976668 + }, + "k_nhd_contiguous": { + "count": 5, + "mean_s": 0.004842757794540376, + "p50_s": 0.004699617973528802, + "p90_s": 0.005195734999142587, + "p95_s": 0.005358382000122219, + "p99_s": 0.005488499600905925, + "min_s": 0.004638209997210652, + "max_s": 0.0055210290011018515 + }, + "v_nhd_contiguous": { + "count": 5, + "mean_s": 0.004678740992676467, + "p50_s": 0.004741937969811261, + "p90_s": 0.004805538605432957, + "p95_s": 0.004823794809635728, + "p99_s": 0.004838399772997945, + "min_s": 0.004513168998528272, + "max_s": 0.0048420510138385 + }, + "attention_kernel": { + "count": 5, + "mean_s": 0.3031308094039559, + "p50_s": 0.2796200959710404, + "p90_s": 0.3695915882126428, + "p95_s": 0.3924628216191195, + "p99_s": 0.4107598083443008, + "min_s": 0.25944606703706086, + "max_s": 0.41533405502559617 + }, + "output_reshape": { + "count": 5, + "mean_s": 5.465606227517128e-06, + "p50_s": 4.223955329507589e-06, + "p90_s": 8.115207310765982e-06, + "p95_s": 9.385612793266772e-06, + "p99_s": 1.0401937179267406e-05, + "min_s": 4.00003045797348e-06, + "max_s": 1.0656018275767565e-05 + } + }, + "layout_attention_total_mean_s": 0.317984988016542, + "reference_backend": "sage2:hnd", + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "status": "ok" + }, + { + "segment": "all", + "segment_tuple": null, + "backend": "sage2", + "layout_mode": "sage_strided_nhd", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "output_shape": [ + 37810, + 7168 + ], + "timings": { + "attention_kernel": { + "count": 5, + "mean_s": 0.260253090003971, + "p50_s": 0.2601712809992023, + "p90_s": 0.2612823422066867, + "p95_s": 0.26159851960837843, + "p99_s": 0.26185146152973177, + "min_s": 0.25904494104906917, + "max_s": 0.2619146970100701 + }, + "output_reshape": { + "count": 5, + "mean_s": 4.3424079194664955e-06, + "p50_s": 4.175992216914892e-06, + "p90_s": 5.068804603070021e-06, + "p95_s": 5.350401625037192e-06, + "p99_s": 5.5756792426109315e-06, + "min_s": 3.8080033846199512e-06, + "max_s": 5.631998647004366e-06 + } + }, + "layout_attention_total_mean_s": 0.2602574324118905, + "reference_backend": "sage2:hnd", + "reference_diff": { + "max": 0.0, + "mean": 0.0 + }, + "status": "ok" + } + ] +} \ No newline at end of file diff --git a/benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json b/benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json new file mode 100644 index 0000000..9f49976 --- /dev/null +++ b/benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json @@ -0,0 +1,85 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "torch": "2.9.1+cu130", + "attention": "sage2", + "resolution": [1344, 768], + "frames": 124, + "packed_tokens": 37810, + "seed": 440420, + "feature_flag": "H3_SAGE_QKV_LAYOUT=strided_nhd", + "dataflow": { + "qkv_input": "projection-strided NHD views with no Q/K/V materialization", + "qk_transform": "in-place Comfy Kitchen RMSNorm and RoPE on strided views", + "attention_output": "contiguous Sage2 NHD reshaped as a token-major view for out_proj", + "removed_copies_per_block": 4 + }, + "isolated_attention_layout": { + "hnd_mean_seconds": 0.3037509555928409, + "strided_nhd_mean_seconds": 0.2602574324118905, + "speedup": 1.167117314490817, + "max_abs_difference": 0.0, + "mean_abs_difference": 0.0 + }, + "real_block_results": [ + { + "block": 0, + "hnd_p50_seconds": 0.4870640740264207, + "strided_nhd_p50_seconds": 0.4714779225178063, + "speedup": 1.0330580728475696, + "max_abs_difference": 0.0 + }, + { + "block": 24, + "hnd_p50_seconds": 0.4879330809926614, + "strided_nhd_p50_seconds": 0.4716692080255598, + "speedup": 1.034481523683056, + "max_abs_difference": 0.0 + }, + { + "block": 49, + "hnd_p50_seconds": 0.48766444699140266, + "strided_nhd_p50_seconds": 0.47158038747147657, + "speedup": 1.0341067184879458, + "max_abs_difference": 0.0 + } + ], + "sampling_2_step": { + "hnd_seconds": 52.49690314201871, + "strided_nhd_seconds": 50.95522483601235, + "speedup": 1.0302555490819223, + "checksums": [-276716.375, 727.1075439453125], + "peak_allocated_bytes_saved": 1185838592, + "peak_reserved_bytes_saved": 2036334592 + }, + "sampling_12_step": { + "component1_hnd_seconds": 301.04568115097936, + "strided_nhd_seconds": 290.2269145210157, + "speedup": 1.0372769239814257, + "latency_reduction": 0.03593729226940101, + "checksums": [-91481.7890625, 678.401611328125], + "cumulative_reduction_vs_unfused_310_11_second_baseline": 0.06411616431876277 + }, + "nsight_block_recapture": { + "tool": "Nsight Systems 2025.3.2", + "hnd_kernel_launches": 53, + "strided_nhd_kernel_launches": 49, + "removed_kernel_launches": 4, + "removed_kernels": "three Q/K/V transpose-contiguous copies and one attention-output transpose-contiguous copy", + "large_bf16_layout_copy_instances_after": 0 + }, + "validation": { + "isolated_attention": "bit-exact", + "blocks_0_24_49": "bit-exact", + "full_50_block_two_step_trajectory": "identical video and audio checksums", + "canonical_12_step_trajectory": "identical video and audio checksums", + "deployed_tests": "34 passed" + }, + "source_artifacts": [ + "benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json", + "benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json", + "benchmarks/gb10-component2-layout-hnd-2step.json", + "benchmarks/gb10-component2-layout-strided-nhd-2step.json", + "benchmarks/gb10-component2-layout-strided-nhd-12step.json" + ] +} diff --git a/benchmarks/gb10-sage2-block24-mainloop-analysis.json b/benchmarks/gb10-sage2-block24-mainloop-analysis.json new file mode 100644 index 0000000..05abb8b --- /dev/null +++ b/benchmarks/gb10-sage2-block24-mainloop-analysis.json @@ -0,0 +1,72 @@ +{ + "device": "NVIDIA GB10", + "workload": "real H3 block 24, 37810 tokens, 56 heads, head dimension 128", + "parity": { + "manual_preparation_and_mainloop_vs_public_sage2_equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "uninstrumented_phase_medians_ms": { + "k_mean_and_smoothing": 2.365136, + "q_int8_quantization": 3.799392, + "k_int8_subtract_mean_quantization": 3.825936, + "v_transpose_pad_permute": 5.081152, + "v_scale_fp8_quantization": 5.531616, + "fused_mainloop": 237.089394 + }, + "ncu_mainloop": { + "profiler_perturbed_duration_ms": 258.918464, + "registers_per_thread": 255, + "dynamic_shared_memory_bytes_per_cta": 32768, + "threads_per_cta": 128, + "grid_ctas": 16576, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.83, + "active_warps_per_sm": 8.08, + "compute_throughput_percent": 75.54, + "memory_throughput_percent": 31.61, + "tensor_int_elapsed_cycles_percent": 37.77, + "tensor_fp_elapsed_cycles_percent": 37.77, + "fma_elapsed_cycles_percent": 15.01, + "alu_elapsed_cycles_percent": 8.81, + "scheduler_cycles_no_eligible_warp_percent": 63.53, + "eligible_warps_per_scheduler": 0.46, + "issued_warps_per_scheduler": 0.36, + "warp_cycles_per_issued_instruction": 5.48, + "warp_stall_cycles_per_instruction": { + "wait_fixed_latency_dependency": 2.01, + "math_pipe_throttle": 1.24, + "not_selected": 0.26, + "short_scoreboard": 0.25, + "barrier": 0.19, + "dispatch": 0.16, + "long_scoreboard": 0.13, + "mio_throttle": 0.12 + }, + "l2_request_sectors": 5046921579, + "l2_request_bytes": 161501490528, + "l2_hit_rate_percent": 98.84, + "l2_miss_sectors": 57882308, + "off_chip_request_proxy_bytes": 1852233856, + "shared_wavefronts": 8161078676, + "excess_shared_wavefronts": 626970624, + "excess_shared_wavefront_percent": 7.68, + "local_spilling_requests": 1458688 + }, + "tail_study_ms": { + "q_37760_aligned": 239.229565, + "q_37761_one_row_tail": 239.632672, + "q_37810_real_tail": 239.999844, + "kv_37760_aligned": 239.823065, + "kv_37761_one_row_tail": 240.594522, + "kv_37810_real_tail": 240.623859 + }, + "diagnosis": [ + "The mainloop is compute/scheduler limited, not off-chip-bandwidth limited.", + "INT8 QK and FP8 PV each consume 37.77% of elapsed tensor-pipe capacity; their sum explains the 75.54% aggregate tensor activity.", + "The 255-register footprint limits occupancy to two CTAs and eight warps per SM, leaving no eligible warp in 63.53% of scheduler cycles.", + "Fixed-latency dependencies and tensor math-pipe throttling dominate stalls; global-memory scoreboards are secondary.", + "Tail CTAs add less than 1 ms and are not the primary optimization target.", + "Shared-memory bank conflicts add 7.68% excessive wavefronts and are the safest secondary target after scheduling/register experiments." + ] +} diff --git a/benchmarks/gb10-sage2-block24-mainloop-capture.json b/benchmarks/gb10-sage2-block24-mainloop-capture.json new file mode 100644 index 0000000..60dc252 --- /dev/null +++ b/benchmarks/gb10-sage2-block24-mainloop-capture.json @@ -0,0 +1,72 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "capture": "one unchanged prequantized Sage2 mainloop", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "k_shape": [ + 1, + 37810, + 56, + 128 + ], + "v_shape": [ + 1, + 37810, + 56, + 128 + ], + "checksum": 9190064.0, + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "warps_per_cta": 4, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "q_ctas_per_head": 296, + "heads": 56, + "grid_ctas": 16576, + "k_iterations_per_cta": 591, + "explicit_pipeline_stages": 2 + } +} diff --git a/benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep b/benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep new file mode 100644 index 0000000..6536010 Binary files /dev/null and b/benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep differ diff --git a/benchmarks/gb10-sage2-block24-scheduler-baseline.json b/benchmarks/gb10-sage2-block24-scheduler-baseline.json new file mode 100644 index 0000000..23493ae --- /dev/null +++ b/benchmarks/gb10-sage2-block24-scheduler-baseline.json @@ -0,0 +1,305 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "k_shape": [ + 1, + 37810, + 56, + 128 + ], + "v_shape": [ + 1, + 37810, + 56, + 128 + ], + "warmup": 3, + "iterations": 10, + "phase_timings": { + "k_mean_and_smoothing_preparation": { + "count": 10, + "mean_s": 0.002420537567138672, + "p50_s": 0.0023651360273361207, + "p90_s": 0.0025749952793121334, + "p95_s": 0.0026686816215515134, + "p99_s": 0.0027436306953430175, + "min_s": 0.002331295967102051, + "max_s": 0.0027623679637908934 + }, + "qk_int8_quantization": { + "count": 10, + "mean_s": 0.007748214387893677, + "p50_s": 0.0076711041927337645, + "p90_s": 0.007954323053359986, + "p95_s": 0.007966649413108826, + "p99_s": 0.007976510500907898, + "min_s": 0.007604640007019043, + "max_s": 0.007978975772857666 + }, + "q_int8_quantization": { + "count": 10, + "mean_s": 0.0037990527868270877, + "p50_s": 0.0037993919849395753, + "p90_s": 0.0038029504060745237, + "p95_s": 0.0038038431525230405, + "p99_s": 0.003804557349681854, + "min_s": 0.003791712045669556, + "max_s": 0.0038047358989715574 + }, + "k_int8_subtract_mean_quantization": { + "count": 10, + "mean_s": 0.0038265248060226436, + "p50_s": 0.0038259359598159792, + "p90_s": 0.0038348320722579954, + "p95_s": 0.0038351920247077943, + "p99_s": 0.003835479986667633, + "min_s": 0.003819711923599243, + "max_s": 0.0038355519771575927 + }, + "v_fp8_transpose_scale_quantization": { + "count": 10, + "mean_s": 0.011513513278961182, + "p50_s": 0.010691807746887208, + "p90_s": 0.011705465221405026, + "p95_s": 0.015195276212692253, + "p99_s": 0.017987125005722047, + "min_s": 0.010680031776428223, + "max_s": 0.018685087203979493 + }, + "v_transpose_pad_permute": { + "count": 10, + "mean_s": 0.0051148447513580325, + "p50_s": 0.005081151962280274, + "p90_s": 0.005127699327468873, + "p95_s": 0.005270201659202576, + "p99_s": 0.005384203524589539, + "min_s": 0.005073599815368653, + "max_s": 0.00541270399093628 + }, + "v_scale_fp8_quantization": { + "count": 10, + "mean_s": 0.005551833629608155, + "p50_s": 0.0055316162109375, + "p90_s": 0.0055659454822540285, + "p95_s": 0.005644108700752258, + "p99_s": 0.005706639275550842, + "min_s": 0.00552732801437378, + "max_s": 0.005722271919250488 + }, + "fused_mainloop": { + "count": 10, + "mean_s": 0.23669681091308595, + "p50_s": 0.23708939361572268, + "p90_s": 0.23775543670654295, + "p95_s": 0.2378476737976074, + "p99_s": 0.23792146347045898, + "min_s": 0.23529600524902344, + "max_s": 0.23793991088867186 + } + }, + "fused_mainloop_phases": { + "int8_qk": "fused inside qk_int_sv_f8_attn_kernel", + "scale_application": "fused inside qk_int_sv_f8_attn_kernel", + "online_softmax": "fused inside qk_int_sv_f8_attn_kernel", + "pv_accumulation": "fused inside qk_int_sv_f8_attn_kernel", + "final_normalization_and_output": "fused inside qk_int_sv_f8_attn_kernel", + "timing_policy": "Do not assign independent wall time without changing the exact kernel schedule; use source-correlated hardware counters." + }, + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "warps_per_cta": 4, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "q_ctas_per_head": 296, + "heads": 56, + "grid_ctas": 16576, + "k_iterations_per_cta": 591, + "q_tail_rows": 50, + "k_tail_rows": 50, + "explicit_pipeline_stages": 2 + }, + "manual_decomposition_vs_public_sage2": { + "equal": true, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "manual_checksum": 9190064.0, + "reference_checksum": 9190064.0, + "tail_study": [ + { + "sweep": "q_tail_fixed_kv", + "q_len": 37760, + "kv_len": 37810, + "q_ctas_per_head": 295, + "k_iterations_per_cta": 591, + "q_tail_rows": 0, + "k_tail_rows": 50, + "scheduled_q_rows": 37760, + "q_row_efficiency": 1.0, + "mainloop": { + "count": 5, + "mean_s": 0.2392295654296875, + "p50_s": 0.23901161193847656, + "p90_s": 0.24024520874023436, + "p95_s": 0.24033575439453123, + "p99_s": 0.24040819091796872, + "min_s": 0.23829190063476563, + "max_s": 0.24042630004882812 + }, + "checksum": 9183940.0 + }, + { + "sweep": "q_tail_fixed_kv", + "q_len": 37761, + "kv_len": 37810, + "q_ctas_per_head": 296, + "k_iterations_per_cta": 591, + "q_tail_rows": 1, + "k_tail_rows": 50, + "scheduled_q_rows": 37888, + "q_row_efficiency": 0.9966480152027027, + "mainloop": { + "count": 5, + "mean_s": 0.23963267211914063, + "p50_s": 0.23961180114746095, + "p90_s": 0.23997588500976563, + "p95_s": 0.24002579956054687, + "p99_s": 0.2400657312011719, + "min_s": 0.23915267944335938, + "max_s": 0.24007571411132814 + }, + "checksum": 9183946.0 + }, + { + "sweep": "q_tail_fixed_kv", + "q_len": 37810, + "kv_len": 37810, + "q_ctas_per_head": 296, + "k_iterations_per_cta": 591, + "q_tail_rows": 50, + "k_tail_rows": 50, + "scheduled_q_rows": 37888, + "q_row_efficiency": 0.9979413006756757, + "mainloop": { + "count": 5, + "mean_s": 0.23999984436035157, + "p50_s": 0.23978463745117187, + "p90_s": 0.24042403869628906, + "p95_s": 0.24055045166015623, + "p99_s": 0.24065158203124998, + "min_s": 0.2397151641845703, + "max_s": 0.24067686462402343 + }, + "checksum": 9190064.0 + }, + { + "sweep": "kv_tail_fixed_q", + "q_len": 37810, + "kv_len": 37760, + "q_ctas_per_head": 296, + "k_iterations_per_cta": 590, + "q_tail_rows": 50, + "k_tail_rows": 0, + "scheduled_q_rows": 37888, + "q_row_efficiency": 0.9979413006756757, + "mainloop": { + "count": 5, + "mean_s": 0.23982306518554686, + "p50_s": 0.23983763122558593, + "p90_s": 0.2401393035888672, + "p95_s": 0.24022885437011718, + "p99_s": 0.2403004949951172, + "min_s": 0.2395267791748047, + "max_s": 0.24031840515136718 + }, + "checksum": 9175197.0 + }, + { + "sweep": "kv_tail_fixed_q", + "q_len": 37810, + "kv_len": 37761, + "q_ctas_per_head": 296, + "k_iterations_per_cta": 591, + "q_tail_rows": 50, + "k_tail_rows": 1, + "scheduled_q_rows": 37888, + "q_row_efficiency": 0.9979413006756757, + "mainloop": { + "count": 5, + "mean_s": 0.2405945220947266, + "p50_s": 0.24039689636230469, + "p90_s": 0.24119658203125, + "p95_s": 0.24143297729492186, + "p99_s": 0.24162209350585936, + "min_s": 0.2401432342529297, + "max_s": 0.24166937255859375 + }, + "checksum": 9176321.0 + }, + { + "sweep": "kv_tail_fixed_q", + "q_len": 37810, + "kv_len": 37810, + "q_ctas_per_head": 296, + "k_iterations_per_cta": 591, + "q_tail_rows": 50, + "k_tail_rows": 50, + "scheduled_q_rows": 37888, + "q_row_efficiency": 0.9979413006756757, + "mainloop": { + "count": 5, + "mean_s": 0.24062385864257813, + "p50_s": 0.24038169860839845, + "p90_s": 0.24108528442382812, + "p95_s": 0.2411941619873047, + "p99_s": 0.24128126403808592, + "min_s": 0.2403317108154297, + "max_s": 0.24130303955078125 + }, + "checksum": 9190064.0 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep b/benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep new file mode 100644 index 0000000..30f9095 Binary files /dev/null and b/benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep new file mode 100644 index 0000000..4f868dc Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep new file mode 100644 index 0000000..e34368e Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep new file mode 100644 index 0000000..a3341cf Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep new file mode 100644 index 0000000..e2eaf62 Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep new file mode 100644 index 0000000..5102dec Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep new file mode 100644 index 0000000..d3e7b43 Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep b/benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep new file mode 100644 index 0000000..6c2384b Binary files /dev/null and b/benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json b/benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json new file mode 100644 index 0000000..41ceb28 --- /dev/null +++ b/benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24454946014404297, + "p50_s": 0.24409790802001952, + "p90_s": 0.24773331298828125, + "p95_s": 0.24835807189941406, + "p99_s": 0.24918752502441405, + "min_s": 0.24054057312011717, + "max_s": 0.24935398864746094 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24177088928222656, + "p50_s": 0.24183243560791015, + "p90_s": 0.24249975128173826, + "p95_s": 0.24257464370727538, + "p99_s": 0.24263455764770508, + "min_s": 0.24054057312011717, + "max_s": 0.2426495361328125 + }, + { + "count": 10, + "mean_s": 0.24227575378417968, + "p50_s": 0.2424848861694336, + "p90_s": 0.24278435516357424, + "p95_s": 0.24283492660522463, + "p99_s": 0.24287538375854495, + "min_s": 0.24105349731445314, + "max_s": 0.242885498046875 + }, + { + "count": 10, + "mean_s": 0.24449723815917968, + "p50_s": 0.24409790802001952, + "p90_s": 0.2455601837158203, + "p95_s": 0.24703209075927735, + "p99_s": 0.248209616394043, + "min_s": 0.24326626586914063, + "max_s": 0.24850399780273438 + }, + { + "count": 10, + "mean_s": 0.24652888488769534, + "p50_s": 0.24648104095458984, + "p90_s": 0.24750688781738278, + "p95_s": 0.24784330291748047, + "p99_s": 0.24811243499755858, + "min_s": 0.2454468536376953, + "max_s": 0.24817971801757813 + }, + { + "count": 10, + "mean_s": 0.24767453460693362, + "p50_s": 0.24752355194091796, + "p90_s": 0.24904823913574217, + "p95_s": 0.24920111389160154, + "p99_s": 0.24932341369628908, + "min_s": 0.24665737915039063, + "max_s": 0.24935398864746094 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "early-k", + "path": "/variants/early-k/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.2447421008300781, + "p50_s": 0.2448299789428711, + "p90_s": 0.24727191772460938, + "p95_s": 0.24772567520141603, + "p99_s": 0.24888865036010743, + "min_s": 0.2410546875, + "max_s": 0.24890591430664064 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.2419627883911133, + "p50_s": 0.2418755340576172, + "p90_s": 0.24243917236328125, + "p95_s": 0.2430596664428711, + "p99_s": 0.24355606170654298, + "min_s": 0.2410546875, + "max_s": 0.24368016052246094 + }, + { + "count": 10, + "mean_s": 0.24308984985351562, + "p50_s": 0.24292047882080076, + "p90_s": 0.24403975982666015, + "p95_s": 0.24468525466918944, + "p99_s": 0.24520165054321288, + "min_s": 0.24181376647949218, + "max_s": 0.24533074951171874 + }, + { + "count": 10, + "mean_s": 0.24468006286621097, + "p50_s": 0.24458190155029297, + "p90_s": 0.24544971313476563, + "p95_s": 0.2455271255493164, + "p99_s": 0.24558905548095702, + "min_s": 0.24351507568359376, + "max_s": 0.2456045379638672 + }, + { + "count": 10, + "mean_s": 0.24647027893066403, + "p50_s": 0.24653636932373046, + "p90_s": 0.24727191772460938, + "p95_s": 0.24740000457763672, + "p99_s": 0.2475024740600586, + "min_s": 0.24537295532226563, + "max_s": 0.24752809143066407 + }, + { + "count": 10, + "mean_s": 0.24750752410888674, + "p50_s": 0.24719758605957032, + "p90_s": 0.24887420501708982, + "p95_s": 0.24889005966186523, + "p99_s": 0.24890274337768556, + "min_s": 0.24669772338867188, + "max_s": 0.24890591430664064 + } + ], + "speedup_vs_baseline_percent": -0.29901196169379585 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json b/benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json new file mode 100644 index 0000000..b834d2a --- /dev/null +++ b/benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24575043548583986, + "p50_s": 0.24647820281982422, + "p90_s": 0.24817954711914061, + "p95_s": 0.24843196334838868, + "p99_s": 0.2488530535888672, + "min_s": 0.24133132934570312, + "max_s": 0.24893434143066406 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24229534759521484, + "p50_s": 0.24214422607421876, + "p90_s": 0.24312568054199218, + "p95_s": 0.24336900024414063, + "p99_s": 0.24356365600585939, + "min_s": 0.24133132934570312, + "max_s": 0.24361231994628907 + }, + { + "count": 10, + "mean_s": 0.24461372528076172, + "p50_s": 0.24456707000732422, + "p90_s": 0.24527583465576172, + "p95_s": 0.24591716842651368, + "p99_s": 0.24643023544311524, + "min_s": 0.2432255096435547, + "max_s": 0.24655850219726563 + }, + { + "count": 10, + "mean_s": 0.24647662353515623, + "p50_s": 0.24633929443359376, + "p90_s": 0.24748114624023437, + "p95_s": 0.24783390350341797, + "p99_s": 0.24811610931396483, + "min_s": 0.245321533203125, + "max_s": 0.24818666076660156 + }, + { + "count": 10, + "mean_s": 0.24721214752197268, + "p50_s": 0.2471304473876953, + "p90_s": 0.24781344299316405, + "p95_s": 0.24802697601318358, + "p99_s": 0.24819780242919923, + "min_s": 0.24645692443847655, + "max_s": 0.24824050903320313 + }, + { + "count": 10, + "mean_s": 0.24815433349609375, + "p50_s": 0.2481010208129883, + "p90_s": 0.24878503723144532, + "p95_s": 0.24885968933105468, + "p99_s": 0.2489194110107422, + "min_s": 0.2475707244873047, + "max_s": 0.24893434143066406 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "inplace-scores", + "path": "/variants/inplace-scores/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.2458565710449219, + "p50_s": 0.2464531707763672, + "p90_s": 0.24893431243896486, + "p95_s": 0.24899965438842775, + "p99_s": 0.2501344566345215, + "min_s": 0.24153382873535156, + "max_s": 0.25087110900878906 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.2423179229736328, + "p50_s": 0.2422579345703125, + "p90_s": 0.2428229248046875, + "p95_s": 0.24329040679931638, + "p99_s": 0.24366439239501952, + "min_s": 0.24153382873535156, + "max_s": 0.2437578887939453 + }, + { + "count": 10, + "mean_s": 0.2441113250732422, + "p50_s": 0.24403551483154295, + "p90_s": 0.24508229522705077, + "p95_s": 0.24537674942016602, + "p99_s": 0.2456123127746582, + "min_s": 0.24297340393066405, + "max_s": 0.24567120361328126 + }, + { + "count": 10, + "mean_s": 0.24651520080566405, + "p50_s": 0.24650099182128907, + "p90_s": 0.24793040618896486, + "p95_s": 0.247960213470459, + "p99_s": 0.24798405929565429, + "min_s": 0.24519821166992187, + "max_s": 0.24799002075195312 + }, + { + "count": 10, + "mean_s": 0.24776356201171876, + "p50_s": 0.2478148651123047, + "p90_s": 0.24900929412841796, + "p95_s": 0.2491885154724121, + "p99_s": 0.24933189254760743, + "min_s": 0.24642454528808594, + "max_s": 0.24936773681640625 + }, + { + "count": 10, + "mean_s": 0.24857484436035157, + "p50_s": 0.24835614776611328, + "p90_s": 0.24920902862548827, + "p95_s": 0.2500400688171387, + "p99_s": 0.250704900970459, + "min_s": 0.24763848876953126, + "max_s": 0.25087110900878906 + } + ], + "speedup_vs_baseline_percent": 0.010156916779835434 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-interleave-softmax-latency.json b/benchmarks/gb10-sage2-p0-interleave-softmax-latency.json new file mode 100644 index 0000000..ef5ef29 --- /dev/null +++ b/benchmarks/gb10-sage2-p0-interleave-softmax-latency.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24388563781738282, + "p50_s": 0.24391904449462892, + "p90_s": 0.24706241455078126, + "p95_s": 0.2474617546081543, + "p99_s": 0.24945008117675782, + "min_s": 0.23941552734375, + "max_s": 0.250071044921875 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24110272827148438, + "p50_s": 0.24101924896240234, + "p90_s": 0.24182757263183594, + "p95_s": 0.24287572326660156, + "p99_s": 0.24371424377441406, + "min_s": 0.23941552734375, + "max_s": 0.2439238739013672 + }, + { + "count": 10, + "mean_s": 0.24202915649414064, + "p50_s": 0.2419222717285156, + "p90_s": 0.24283133544921875, + "p95_s": 0.24342016143798828, + "p99_s": 0.2438912222290039, + "min_s": 0.24108163452148437, + "max_s": 0.2440089874267578 + }, + { + "count": 10, + "mean_s": 0.24356186828613283, + "p50_s": 0.2436042709350586, + "p90_s": 0.24417928314208984, + "p95_s": 0.24448768997192383, + "p99_s": 0.244734415435791, + "min_s": 0.2425731201171875, + "max_s": 0.24479609680175782 + }, + { + "count": 10, + "mean_s": 0.24519928436279298, + "p50_s": 0.24513208007812498, + "p90_s": 0.24575088195800782, + "p95_s": 0.2460108322143555, + "p99_s": 0.24621879241943362, + "min_s": 0.24462042236328124, + "max_s": 0.24627078247070314 + }, + { + "count": 10, + "mean_s": 0.24753515167236326, + "p50_s": 0.24709844970703126, + "p90_s": 0.2489304992675781, + "p95_s": 0.24950077209472654, + "p99_s": 0.24995699035644528, + "min_s": 0.24651388549804687, + "max_s": 0.250071044921875 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "interleave-softmax", + "path": "/variants/interleave-softmax/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24397127563476562, + "p50_s": 0.2439577865600586, + "p90_s": 0.24716617736816404, + "p95_s": 0.2477003547668457, + "p99_s": 0.2484949363708496, + "min_s": 0.239262939453125, + "max_s": 0.24906739807128905 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24102854003906252, + "p50_s": 0.24122242736816407, + "p90_s": 0.24205140991210936, + "p95_s": 0.24227293548583984, + "p99_s": 0.24245015594482422, + "min_s": 0.239262939453125, + "max_s": 0.24249446105957032 + }, + { + "count": 10, + "mean_s": 0.24219576721191408, + "p50_s": 0.24199546813964845, + "p90_s": 0.24289817504882813, + "p95_s": 0.24381009521484376, + "p99_s": 0.24453963134765627, + "min_s": 0.24077565002441406, + "max_s": 0.2447220153808594 + }, + { + "count": 10, + "mean_s": 0.2438770950317383, + "p50_s": 0.2437736587524414, + "p90_s": 0.24470800323486327, + "p95_s": 0.24506654891967772, + "p99_s": 0.2453533854675293, + "min_s": 0.24293548583984376, + "max_s": 0.2454250946044922 + }, + { + "count": 10, + "mean_s": 0.24551427154541017, + "p50_s": 0.2453832778930664, + "p90_s": 0.2465328369140625, + "p95_s": 0.24662471008300782, + "p99_s": 0.24669820861816405, + "min_s": 0.24450390625, + "max_s": 0.24671658325195311 + }, + { + "count": 10, + "mean_s": 0.24724070434570314, + "p50_s": 0.24736494445800783, + "p90_s": 0.24801593780517578, + "p95_s": 0.2485416679382324, + "p99_s": 0.24896225204467773, + "min_s": 0.24577845764160156, + "max_s": 0.24906739807128905 + } + ], + "speedup_vs_baseline_percent": -0.015880643112875514 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep b/benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep new file mode 100644 index 0000000..6c21012 Binary files /dev/null and b/benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-pad-v-smem-latency.json b/benchmarks/gb10-sage2-p0-pad-v-smem-latency.json new file mode 100644 index 0000000..a1eb3e9 --- /dev/null +++ b/benchmarks/gb10-sage2-p0-pad-v-smem-latency.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.2461453451538086, + "p50_s": 0.24636819458007814, + "p90_s": 0.24946011962890624, + "p95_s": 0.2502609535217285, + "p99_s": 0.25086575302124026, + "min_s": 0.24162757873535157, + "max_s": 0.250990234375 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.2425358856201172, + "p50_s": 0.24232412719726562, + "p90_s": 0.2432921615600586, + "p95_s": 0.2435843360900879, + "p99_s": 0.24381807571411135, + "min_s": 0.24162757873535157, + "max_s": 0.2438765106201172 + }, + { + "count": 10, + "mean_s": 0.24422555541992189, + "p50_s": 0.2443652801513672, + "p90_s": 0.24503082122802733, + "p95_s": 0.2450529930114746, + "p99_s": 0.2450707304382324, + "min_s": 0.24320451354980469, + "max_s": 0.24507516479492186 + }, + { + "count": 10, + "mean_s": 0.24644686584472658, + "p50_s": 0.24636819458007814, + "p90_s": 0.24773657531738283, + "p95_s": 0.24776796875, + "p99_s": 0.24779308349609377, + "min_s": 0.24484390258789063, + "max_s": 0.2477993621826172 + }, + { + "count": 10, + "mean_s": 0.24803108673095703, + "p50_s": 0.2479405059814453, + "p90_s": 0.24909479675292967, + "p95_s": 0.24924445800781247, + "p99_s": 0.24936418701171875, + "min_s": 0.2466182403564453, + "max_s": 0.2493941192626953 + }, + { + "count": 10, + "mean_s": 0.24948733215332033, + "p50_s": 0.24952471923828123, + "p90_s": 0.2507615951538086, + "p95_s": 0.2508759147644043, + "p99_s": 0.25096737045288087, + "min_s": 0.24757859802246093, + "max_s": 0.250990234375 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "pad-v-smem", + "path": "/variants/pad-v-smem/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24588236846923828, + "p50_s": 0.2464697723388672, + "p90_s": 0.24931472930908202, + "p95_s": 0.24956529541015626, + "p99_s": 0.25046807495117185, + "min_s": 0.24145852661132813, + "max_s": 0.25106903076171877 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24199966735839845, + "p50_s": 0.24201995086669922, + "p90_s": 0.24234957427978515, + "p95_s": 0.24245646438598634, + "p99_s": 0.24254197647094727, + "min_s": 0.24145852661132813, + "max_s": 0.2425633544921875 + }, + { + "count": 10, + "mean_s": 0.24424241943359376, + "p50_s": 0.24423397827148438, + "p90_s": 0.2449089614868164, + "p95_s": 0.24502979049682616, + "p99_s": 0.245126453704834, + "min_s": 0.2430936279296875, + "max_s": 0.24515061950683595 + }, + { + "count": 10, + "mean_s": 0.24634171447753905, + "p50_s": 0.2464697723388672, + "p90_s": 0.2470756805419922, + "p95_s": 0.2471369155883789, + "p99_s": 0.24718590362548828, + "min_s": 0.24513539123535155, + "max_s": 0.24719815063476563 + }, + { + "count": 10, + "mean_s": 0.24810086364746092, + "p50_s": 0.24811367034912107, + "p90_s": 0.2494482360839844, + "p95_s": 0.24964541320800782, + "p99_s": 0.24980315490722654, + "min_s": 0.2465597686767578, + "max_s": 0.24984259033203124 + }, + { + "count": 10, + "mean_s": 0.2487271774291992, + "p50_s": 0.24817182159423828, + "p90_s": 0.2497320083618164, + "p95_s": 0.25040051956176756, + "p99_s": 0.25093532852172856, + "min_s": 0.24757408142089843, + "max_s": 0.25106903076171877 + } + ], + "speedup_vs_baseline_percent": -0.041213069588663664 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep b/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep new file mode 100644 index 0000000..8e920fd Binary files /dev/null and b/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json b/benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json new file mode 100644 index 0000000..1a82c29 --- /dev/null +++ b/benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json @@ -0,0 +1,452 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 3, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24998416646321614, + "p50_s": 0.25057755279541016, + "p90_s": 0.25144031829833985, + "p95_s": 0.2516533340454101, + "p99_s": 0.2520373115539551, + "min_s": 0.24524850463867187, + "max_s": 0.2521801300048828 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24834108581542966, + "p50_s": 0.2484163360595703, + "p90_s": 0.2499428024291992, + "p95_s": 0.25027338790893555, + "p99_s": 0.25053785629272457, + "min_s": 0.24524850463867187, + "max_s": 0.25060397338867185 + }, + { + "count": 10, + "mean_s": 0.2504332321166992, + "p50_s": 0.25060564422607423, + "p90_s": 0.2514479446411133, + "p95_s": 0.25156779861450196, + "p99_s": 0.25166368179321286, + "min_s": 0.24862063598632814, + "max_s": 0.2516876525878906 + }, + { + "count": 10, + "mean_s": 0.2511781814575195, + "p50_s": 0.2510877304077148, + "p90_s": 0.2516682632446289, + "p95_s": 0.25192419662475585, + "p99_s": 0.25212894332885744, + "min_s": 0.25049385070800784, + "max_s": 0.2521801300048828 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "cap240", + "path": "/variants/cap240/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24987627207438154, + "p50_s": 0.25057295989990236, + "p90_s": 0.25185800933837893, + "p95_s": 0.2520280746459961, + "p99_s": 0.2521397077941895, + "min_s": 0.24523724365234376, + "max_s": 0.25215855407714843 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24774005737304688, + "p50_s": 0.24774388885498047, + "p90_s": 0.24998814849853515, + "p95_s": 0.25016738357543944, + "p99_s": 0.2503107716369629, + "min_s": 0.24523724365234376, + "max_s": 0.25034661865234376 + }, + { + "count": 10, + "mean_s": 0.2507166275024414, + "p50_s": 0.25092473602294924, + "p90_s": 0.25196258239746094, + "p95_s": 0.2520280746459961, + "p99_s": 0.25208046844482423, + "min_s": 0.2484784698486328, + "max_s": 0.2520935668945313 + }, + { + "count": 10, + "mean_s": 0.25117213134765626, + "p50_s": 0.25101528167724607, + "p90_s": 0.2518790618896484, + "p95_s": 0.25201880798339843, + "p99_s": 0.25213060485839844, + "min_s": 0.2506709442138672, + "max_s": 0.25215855407714843 + } + ], + "speedup_vs_baseline_percent": 0.0018329573588715675 + }, + { + "name": "cap224", + "path": "/variants/cap224/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.249835031636556, + "p50_s": 0.2506428298950195, + "p90_s": 0.25169461517333985, + "p95_s": 0.2518401992797852, + "p99_s": 0.252011573638916, + "min_s": 0.24491375732421874, + "max_s": 0.25207891845703123 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24761009826660155, + "p50_s": 0.24762243652343752, + "p90_s": 0.24979130096435545, + "p95_s": 0.249968709564209, + "p99_s": 0.2501106364440918, + "min_s": 0.24491375732421874, + "max_s": 0.2501461181640625 + }, + { + "count": 10, + "mean_s": 0.25069587097167967, + "p50_s": 0.250751953125, + "p90_s": 0.25169605865478517, + "p95_s": 0.25177137680053713, + "p99_s": 0.2518316313171387, + "min_s": 0.2485064697265625, + "max_s": 0.2518466949462891 + }, + { + "count": 10, + "mean_s": 0.2511991256713867, + "p50_s": 0.25109932708740235, + "p90_s": 0.2518569259643555, + "p95_s": 0.25196792221069336, + "p99_s": 0.2520567192077637, + "min_s": 0.2506775665283203, + "max_s": 0.25207891845703123 + } + ], + "speedup_vs_baseline_percent": -0.026043872723868766 + }, + { + "name": "cap208", + "path": "/variants/cap208/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24960995839436848, + "p50_s": 0.2505243988037109, + "p90_s": 0.25124317932128903, + "p95_s": 0.2522401481628418, + "p99_s": 0.25248081634521485, + "min_s": 0.24468544006347656, + "max_s": 0.2525556793212891 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.2475009994506836, + "p50_s": 0.24763600158691407, + "p90_s": 0.24983478240966797, + "p95_s": 0.25021980514526365, + "p99_s": 0.25052782333374024, + "min_s": 0.24468544006347656, + "max_s": 0.25060482788085936 + }, + { + "count": 10, + "mean_s": 0.24999963226318359, + "p50_s": 0.25045435333251953, + "p90_s": 0.2507531997680664, + "p95_s": 0.2507778984069824, + "p99_s": 0.2507976573181152, + "min_s": 0.248576416015625, + "max_s": 0.2508025970458984 + }, + { + "count": 10, + "mean_s": 0.2513292434692383, + "p50_s": 0.2511095504760742, + "p90_s": 0.25232334594726563, + "p95_s": 0.25243951263427733, + "p99_s": 0.25253244598388674, + "min_s": 0.2505731201171875, + "max_s": 0.2525556793212891 + } + ], + "speedup_vs_baseline_percent": 0.021217091809444533 + }, + { + "name": "cap192", + "path": "/variants/cap192/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24963107147216798, + "p50_s": 0.2505600128173828, + "p90_s": 0.25116943359375, + "p95_s": 0.2515309310913086, + "p99_s": 0.25208772354125975, + "min_s": 0.245866943359375, + "max_s": 0.25223532104492186 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24760592041015625, + "p50_s": 0.24771544647216798, + "p90_s": 0.24872108612060545, + "p95_s": 0.24993844680786131, + "p99_s": 0.250912335357666, + "min_s": 0.245866943359375, + "max_s": 0.2511558074951172 + }, + { + "count": 10, + "mean_s": 0.25040733795166015, + "p50_s": 0.25052005004882816, + "p90_s": 0.25177725982666016, + "p95_s": 0.252006290435791, + "p99_s": 0.25218951492309566, + "min_s": 0.24859814453125, + "max_s": 0.25223532104492186 + }, + { + "count": 10, + "mean_s": 0.2508799560546875, + "p50_s": 0.2509454574584961, + "p90_s": 0.25108498992919925, + "p95_s": 0.2511885292053223, + "p99_s": 0.2512713606262207, + "min_s": 0.25055087280273436, + "max_s": 0.2512920684814453 + } + ], + "speedup_vs_baseline_percent": 0.007000310157279088 + }, + { + "name": "cap176", + "path": "/variants/cap176/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24970435078938802, + "p50_s": 0.25055494689941404, + "p90_s": 0.25130882110595704, + "p95_s": 0.25177099151611326, + "p99_s": 0.25216066497802736, + "min_s": 0.24536207580566408, + "max_s": 0.25220249938964845 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24781683044433594, + "p50_s": 0.2478003387451172, + "p90_s": 0.24928948822021485, + "p95_s": 0.2501019889831543, + "p99_s": 0.25075198959350586, + "min_s": 0.24536207580566408, + "max_s": 0.25091448974609376 + }, + { + "count": 10, + "mean_s": 0.25014339141845704, + "p50_s": 0.25055494689941404, + "p90_s": 0.25096273803710933, + "p95_s": 0.25119132232666014, + "p99_s": 0.2513741897583008, + "min_s": 0.24859910583496095, + "max_s": 0.25141990661621094 + }, + { + "count": 10, + "mean_s": 0.25115283050537107, + "p50_s": 0.25102681732177734, + "p90_s": 0.25207266845703125, + "p95_s": 0.25213758392333985, + "p99_s": 0.25218951629638675, + "min_s": 0.25053001403808595, + "max_s": 0.25220249938964845 + } + ], + "speedup_vs_baseline_percent": 0.009022330740560136 + }, + { + "name": "cap170", + "path": "/variants/cap170/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.2497205561319987, + "p50_s": 0.25040869140625, + "p90_s": 0.25115494079589845, + "p95_s": 0.25126389846801755, + "p99_s": 0.2520225602722168, + "min_s": 0.2463402862548828, + "max_s": 0.25232691955566405 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24803941955566405, + "p50_s": 0.24783467102050782, + "p90_s": 0.2496176803588867, + "p95_s": 0.25036483993530273, + "p99_s": 0.25096256759643554, + "min_s": 0.2463402862548828, + "max_s": 0.25111199951171875 + }, + { + "count": 10, + "mean_s": 0.2499930618286133, + "p50_s": 0.25026488494873045, + "p90_s": 0.2506913803100586, + "p95_s": 0.25079934158325196, + "p99_s": 0.25088571060180664, + "min_s": 0.2485244140625, + "max_s": 0.2509073028564453 + }, + { + "count": 10, + "mean_s": 0.25112918701171877, + "p50_s": 0.25108009338378906, + "p90_s": 0.251382356262207, + "p95_s": 0.25185463790893553, + "p99_s": 0.25223246322631837, + "min_s": 0.250698974609375, + "max_s": 0.25232691955566405 + } + ], + "speedup_vs_baseline_percent": 0.06743431636173636 + }, + { + "name": "cap168", + "path": "/variants/cap168/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 30, + "mean_s": 0.24968490193684897, + "p50_s": 0.2505195007324219, + "p90_s": 0.25164593200683594, + "p95_s": 0.2521135581970215, + "p99_s": 0.2522719644165039, + "min_s": 0.2455440979003906, + "max_s": 0.25233270263671875 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24764180755615234, + "p50_s": 0.24760572814941406, + "p90_s": 0.24893701629638673, + "p95_s": 0.24976327743530274, + "p99_s": 0.25042428634643554, + "min_s": 0.2455440979003906, + "max_s": 0.25058953857421873 + }, + { + "count": 10, + "mean_s": 0.25005723724365236, + "p50_s": 0.2504941635131836, + "p90_s": 0.2507941650390625, + "p95_s": 0.2508059341430664, + "p99_s": 0.25081534942626954, + "min_s": 0.2486126708984375, + "max_s": 0.2508177032470703 + }, + { + "count": 10, + "mean_s": 0.2513556610107422, + "p50_s": 0.25128993988037107, + "p90_s": 0.25214420471191407, + "p95_s": 0.2522384536743164, + "p99_s": 0.2523138528442383, + "min_s": 0.2504866485595703, + "max_s": 0.25233270263671875 + } + ], + "speedup_vs_baseline_percent": 0.02317267231435327 + } + ] +} diff --git a/benchmarks/gb10-sage2-p0-register-scheduler-analysis.json b/benchmarks/gb10-sage2-p0-register-scheduler-analysis.json new file mode 100644 index 0000000..e5bf90f --- /dev/null +++ b/benchmarks/gb10-sage2-p0-register-scheduler-analysis.json @@ -0,0 +1,146 @@ +{ + "device": "NVIDIA GB10", + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "workload": "real H3 block 24, 37810 tokens, 56 heads, head dimension 128", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "register_cap_ncu": [ + { + "variant": "baseline", + "registers_per_thread": 255, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.66, + "local_spilling_requests": 1458688, + "no_eligible_percent": 63.13, + "profiler_duration_ms": 257.69 + }, + { + "variant": "cap240", + "registers_per_thread": 240, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.65, + "local_spilling_requests": 132608, + "no_eligible_percent": 64.97, + "profiler_duration_ms": 271.79 + }, + { + "variant": "cap224", + "registers_per_thread": 224, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.65, + "local_spilling_requests": 511933184, + "no_eligible_percent": 65.76, + "profiler_duration_ms": 282.63 + }, + { + "variant": "cap208", + "registers_per_thread": 208, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.66, + "local_spilling_requests": 1844378368, + "no_eligible_percent": 67.77, + "profiler_duration_ms": 310.92 + }, + { + "variant": "cap192", + "registers_per_thread": 192, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.66, + "local_spilling_requests": 3176425728, + "no_eligible_percent": 70.86, + "profiler_duration_ms": 356.43 + }, + { + "variant": "cap176", + "registers_per_thread": 176, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.66, + "local_spilling_requests": 4320434944, + "no_eligible_percent": 77.17, + "profiler_duration_ms": 462.61 + }, + { + "variant": "cap170", + "registers_per_thread": 170, + "resident_ctas_per_sm": 2, + "theoretical_occupancy_percent": 16.67, + "achieved_occupancy_percent": 16.65, + "local_spilling_requests": 4790928128, + "no_eligible_percent": 79.43, + "profiler_duration_ms": 522.24 + }, + { + "variant": "cap168", + "registers_per_thread": 168, + "resident_ctas_per_sm": 3, + "theoretical_occupancy_percent": 25.0, + "achieved_occupancy_percent": 24.97, + "local_spilling_requests": 4947538176, + "no_eligible_percent": 78.79, + "profiler_duration_ms": 510.55 + } + ], + "source_variants": [ + { + "variant": "register_scopes", + "byte_exact": true, + "static_spill_stores_bytes": 12, + "static_spill_loads_bytes": 12, + "interleaved_speedup_percent": 0.06234012143488954, + "ncu_local_spilling_requests": 397824, + "ncu_no_eligible_percent": 63.72, + "ncu_profiler_duration_ms": 259.74, + "decision": "reject" + }, + { + "variant": "early_k_prefetch", + "byte_exact": true, + "static_spill_stores_bytes": 44, + "static_spill_loads_bytes": 44, + "interleaved_speedup_percent": -0.29901196169379585, + "decision": "reject" + }, + { + "variant": "inplace_score_fragments", + "byte_exact": true, + "static_spill_stores_bytes": 40, + "static_spill_loads_bytes": 40, + "interleaved_speedup_percent": 0.010156916779835434, + "decision": "reject" + }, + { + "variant": "interleave_softmax_fragments", + "byte_exact": true, + "static_spill_stores_bytes": 44, + "static_spill_loads_bytes": 44, + "interleaved_speedup_percent": -0.015880643112875514, + "decision": "reject" + }, + { + "variant": "pad_v_shared_stride_128", + "byte_exact": true, + "dynamic_shared_memory_bytes": 40960, + "static_spill_stores_bytes": 44, + "static_spill_loads_bytes": 48, + "interleaved_speedup_percent": -0.041213069588663664, + "shared_wavefronts": 8788050708, + "ideal_shared_wavefronts": 8161080084, + "excess_shared_wavefronts": 626970624, + "decision": "reject" + } + ], + "conclusions": [ + "The exact occupancy cliff is 168 registers per thread; 170 registers still permits only two resident CTAs.", + "Forcing the 168-register residency tier causes 4.95 billion local spilling requests and worsens scheduler eligibility.", + "Narrower scopes reduce static and dynamic spills but do not improve latency or scheduler eligibility.", + "The exact output accumulator and instantaneous PV fragments keep the source kernel structurally far above the three-CTA register tier.", + "Early K prefetch and independent softmax-chain interleaving do not improve real-input latency.", + "All excessive shared wavefronts map to repeated V-staging LDGSTS.128 instructions; a 128-byte padded V layout leaves the excess count unchanged.", + "No byte-exact variant exceeds the 3 percent complete-block gate, so no complete-block run or deployment is warranted." + ] +} diff --git a/benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep b/benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep new file mode 100644 index 0000000..2439360 Binary files /dev/null and b/benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json b/benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json new file mode 100644 index 0000000..98006a0 --- /dev/null +++ b/benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24493060607910155, + "p50_s": 0.24506629180908202, + "p90_s": 0.24735857086181642, + "p95_s": 0.24807553634643553, + "p99_s": 0.2484788020324707, + "min_s": 0.24139625549316407, + "max_s": 0.24860610961914062 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24188405456542966, + "p50_s": 0.24180159759521486, + "p90_s": 0.2422922882080078, + "p95_s": 0.24256917114257812, + "p99_s": 0.24279067749023436, + "min_s": 0.24139625549316407, + "max_s": 0.24284605407714843 + }, + { + "count": 10, + "mean_s": 0.24345465087890625, + "p50_s": 0.24355528259277343, + "p90_s": 0.24430398864746095, + "p95_s": 0.24450120697021485, + "p99_s": 0.24465898162841798, + "min_s": 0.24244216918945313, + "max_s": 0.24469842529296876 + }, + { + "count": 10, + "mean_s": 0.24523681640625, + "p50_s": 0.24506629180908202, + "p90_s": 0.24575099334716796, + "p95_s": 0.24596200637817384, + "p99_s": 0.2461308168029785, + "min_s": 0.2445313262939453, + "max_s": 0.24617301940917968 + }, + { + "count": 10, + "mean_s": 0.24642633056640623, + "p50_s": 0.24648467254638673, + "p90_s": 0.24679862213134765, + "p95_s": 0.2469401741027832, + "p99_s": 0.24705341567993164, + "min_s": 0.2457111053466797, + "max_s": 0.24708172607421874 + }, + { + "count": 10, + "mean_s": 0.24765117797851563, + "p50_s": 0.24752259063720702, + "p90_s": 0.24837227935791015, + "p95_s": 0.24848919448852538, + "p99_s": 0.24858272659301756, + "min_s": 0.24690821838378907, + "max_s": 0.24860610961914062 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "scopes", + "path": "/variants/scopes/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.2450569662475586, + "p50_s": 0.24491361236572265, + "p90_s": 0.2475555648803711, + "p95_s": 0.2478000213623047, + "p99_s": 0.24837403427124025, + "min_s": 0.24112467956542968, + "max_s": 0.24871961975097656 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24211443023681642, + "p50_s": 0.242046875, + "p90_s": 0.2430909957885742, + "p95_s": 0.24327196121215822, + "p99_s": 0.2434167335510254, + "min_s": 0.24112467956542968, + "max_s": 0.2434529266357422 + }, + { + "count": 10, + "mean_s": 0.24358483276367188, + "p50_s": 0.24363387298583983, + "p90_s": 0.24407558746337887, + "p95_s": 0.24423013076782224, + "p99_s": 0.24435376541137696, + "min_s": 0.24262857055664064, + "max_s": 0.24438467407226563 + }, + { + "count": 10, + "mean_s": 0.2451986862182617, + "p50_s": 0.24491361236572265, + "p90_s": 0.24644293365478515, + "p95_s": 0.24666307907104493, + "p99_s": 0.24683919540405272, + "min_s": 0.24451295471191406, + "max_s": 0.24688322448730468 + }, + { + "count": 10, + "mean_s": 0.24685755767822268, + "p50_s": 0.24679541015625, + "p90_s": 0.24760929107666016, + "p95_s": 0.24781181716918946, + "p99_s": 0.2479738380432129, + "min_s": 0.24587753295898437, + "max_s": 0.24801434326171876 + }, + { + "count": 10, + "mean_s": 0.2475293243408203, + "p50_s": 0.2474662857055664, + "p90_s": 0.24803896179199217, + "p95_s": 0.24837929077148435, + "p99_s": 0.24865155395507813, + "min_s": 0.24696047973632812, + "max_s": 0.24871961975097656 + } + ], + "speedup_vs_baseline_percent": 0.06234012143488954 + } + ] +} diff --git a/benchmarks/gb10-sage2-p1-block-gate.json b/benchmarks/gb10-sage2-p1-block-gate.json new file mode 100644 index 0000000..82582a3 --- /dev/null +++ b/benchmarks/gb10-sage2-p1-block-gate.json @@ -0,0 +1,259 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "workload": { + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "tokens": 37810, + "hidden_shape": [ + 37810, + 5376 + ], + "rotation_shape": [ + 1, + 37810, + 1, + 48, + 2, + 2 + ] + }, + "rounds": 8, + "blocks": [ + { + "block": 0, + "candidate_vs_baseline": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "candidate_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline": { + "timing": { + "count": 8, + "mean_s": 0.5090636406239355, + "p50_s": 0.5093254634994082, + "p90_s": 0.5097644505149219, + "p95_s": 0.5097685157816159, + "p99_s": 0.509771767994971, + "min_s": 0.50723756296793, + "max_s": 0.5097725810483098 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 0, + "runtime_kernel_launches_per_block": 0.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 0 + } + }, + "candidate": { + "timing": { + "count": 8, + "mean_s": 0.5060154675156809, + "p50_s": 0.5055866409966256, + "p90_s": 0.5077965697273612, + "p95_s": 0.5078650188806932, + "p99_s": 0.5079197782033589, + "min_s": 0.5044256669934839, + "max_s": 0.5079334680340253 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 86, + "runtime_kernel_launches_per_block": 86.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 99 + } + }, + "p50_improvement_percent": 0.7340733520555491 + }, + { + "block": 24, + "candidate_vs_baseline": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "candidate_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline": { + "timing": { + "count": 8, + "mean_s": 0.5123310216222308, + "p50_s": 0.5125731414882466, + "p90_s": 0.5134933520865161, + "p95_s": 0.5140004205662989, + "p99_s": 0.5144060753501253, + "min_s": 0.5090091329766437, + "max_s": 0.5145074890460819 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 87, + "runtime_kernel_launches_per_block": 87.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 100 + } + }, + "candidate": { + "timing": { + "count": 8, + "mean_s": 0.5083635471310117, + "p50_s": 0.5081502740213182, + "p90_s": 0.5093831970880274, + "p95_s": 0.5095810055354377, + "p99_s": 0.509739252293366, + "min_s": 0.5069579069968313, + "max_s": 0.5097788139828481 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 86, + "runtime_kernel_launches_per_block": 86.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 99 + } + }, + "p50_improvement_percent": 0.8628753847863981 + }, + { + "block": 49, + "candidate_vs_baseline": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "candidate_vs_capture": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline": { + "timing": { + "count": 8, + "mean_s": 0.5096271816291846, + "p50_s": 0.5093877254694235, + "p90_s": 0.5109008273051586, + "p95_s": 0.5112217181624146, + "p99_s": 0.5114784308482194, + "min_s": 0.5076214010477997, + "max_s": 0.5115426090196706 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 87, + "runtime_kernel_launches_per_block": 87.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 100 + } + }, + "candidate": { + "timing": { + "count": 8, + "mean_s": 0.506856961757876, + "p50_s": 0.5066641345038079, + "p90_s": 0.5086580028233585, + "p95_s": 0.509256707908935, + "p99_s": 0.5097356719773961, + "min_s": 0.5044357180013321, + "max_s": 0.5098554129945114 + }, + "memory": { + "allocated_bytes_before": 13656331264, + "reserved_bytes_before": 23125295104, + "peak_allocated_bytes": 18806702592, + "peak_reserved_bytes": 23125295104, + "peak_allocated_delta_bytes": 5150371328, + "peak_reserved_delta_bytes": 0 + }, + "launches": { + "profiled_iterations": 1, + "runtime_kernel_launches": 86, + "runtime_kernel_launches_per_block": 86.0, + "positive_self_device_allocated_bytes": 14785398272, + "device_kernel_launches": 99 + } + }, + "p50_improvement_percent": 0.5346793472704265 + } + ] +} diff --git a/benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep b/benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep new file mode 100644 index 0000000..0041aab Binary files /dev/null and b/benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep b/benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep new file mode 100644 index 0000000..8dec0da Binary files /dev/null and b/benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep differ diff --git a/benchmarks/gb10-sage2-p1-entry-fusion-analysis.json b/benchmarks/gb10-sage2-p1-entry-fusion-analysis.json new file mode 100644 index 0000000..46e658c --- /dev/null +++ b/benchmarks/gb10-sage2-p1-entry-fusion-analysis.json @@ -0,0 +1,58 @@ +{ + "status": "rejected", + "reason": "No canonical complete block reached the required 1% median latency improvement.", + "workload": { + "resolution": [1344, 768], + "frames": 124, + "tokens": 37810, + "seed": 440420, + "blocks": [0, 24, 49] + }, + "correctness": { + "randomized_lengths": [1, 31, 32, 33, 127, 128, 129], + "prepared_q": "bit-exact", + "prepared_k": "bit-exact", + "q_int8": "bit-exact", + "q_scale": "bit-exact", + "k_mean": "bit-exact", + "k_int8": "bit-exact", + "k_scale": "bit-exact", + "sage2_output": "bit-exact", + "complete_block_outputs": "bit-exact" + }, + "entry_only_p50_ms": { + "block_0": {"baseline": 16.105536, "candidate": 12.329312}, + "block_24": {"baseline": 15.605952, "candidate": 12.344768}, + "block_49": {"baseline": 15.545312, "candidate": 12.309952} + }, + "complete_block": { + "block_0": {"baseline_p50_ms": 509.325463, "candidate_p50_ms": 505.586641, "improvement_percent": 0.734073}, + "block_24": {"baseline_p50_ms": 512.573141, "candidate_p50_ms": 508.150274, "improvement_percent": 0.862875}, + "block_49": {"baseline_p50_ms": 509.387725, "candidate_p50_ms": 506.664135, "improvement_percent": 0.534679} + }, + "peak_memory": { + "baseline_peak_allocated_delta_bytes": 5150371328, + "candidate_peak_allocated_delta_bytes": 5150371328, + "baseline_peak_reserved_delta_bytes": 0, + "candidate_peak_reserved_delta_bytes": 0 + }, + "launches_per_block": {"baseline": 87, "candidate": 86}, + "l2_traffic_gb": { + "block_0": {"baseline": 326.230466, "candidate": 325.788103, "reduction_percent": 0.135598}, + "block_24": {"baseline": 326.281922, "candidate": 325.775965, "reduction_percent": 0.155067}, + "block_49": {"baseline": 326.242832, "candidate": 325.769295, "reduction_percent": 0.145149} + }, + "decision": { + "runtime_integration": "removed", + "two_step_validation": "skipped", + "canonical_12_step_validation": "skipped", + "next_study": "four-GEMM NVFP4 roofline" + }, + "evidence": [ + "benchmarks/gb10-sage2-p1-randomized.json", + "benchmarks/gb10-sage2-p1-real-parity.json", + "benchmarks/gb10-sage2-p1-block-gate.json", + "benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep", + "benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep" + ] +} diff --git a/benchmarks/gb10-sage2-p1-randomized.json b/benchmarks/gb10-sage2-p1-randomized.json new file mode 100644 index 0000000..5cf7fe4 --- /dev/null +++ b/benchmarks/gb10-sage2-p1-randomized.json @@ -0,0 +1,204 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "metadata": {}, + "randomized": [ + { + "sequence": 1, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 31, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 32, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 33, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 127, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 128, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 129, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + } + ], + "real": [] +} diff --git a/benchmarks/gb10-sage2-p1-real-parity.json b/benchmarks/gb10-sage2-p1-real-parity.json new file mode 100644 index 0000000..3210df7 --- /dev/null +++ b/benchmarks/gb10-sage2-p1-real-parity.json @@ -0,0 +1,515 @@ +{ + "device": "NVIDIA GB10", + "torch": "2.9.1+cu130", + "metadata": { + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "tokens": 37810, + "seed": 440420, + "blocks": [ + 0, + 24, + 49 + ] + }, + "randomized": [ + { + "sequence": 1, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 31, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 32, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 33, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 127, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 128, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 129, + "heads": 2, + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + } + ], + "real": [ + { + "block": 0, + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "q_stride": [ + 813066240, + 21504, + 128, + 1 + ], + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_mean": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "attention_output": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_entry": { + "timing": { + "count": 5, + "mean_s": 0.015943762969970705, + "p50_s": 0.01610553550720215, + "p90_s": 0.016237541961669922, + "p95_s": 0.016242866897583006, + "p99_s": 0.016247126846313478, + "min_s": 0.015570015907287597, + "max_s": 0.016248191833496094 + }, + "allocated_bytes_before": 12271139840, + "reserved_bytes_before": 18274582528, + "peak_allocated_bytes": 12542427136, + "peak_reserved_bytes": 18274582528, + "peak_allocated_delta_bytes": 271287296, + "peak_reserved_delta_bytes": 0 + }, + "candidate_entry": { + "timing": { + "count": 5, + "mean_s": 0.01246666259765625, + "p50_s": 0.012329312324523925, + "p90_s": 0.012692352104187012, + "p95_s": 0.01272153606414795, + "p99_s": 0.0127448832321167, + "min_s": 0.012323200225830078, + "max_s": 0.012750720024108887 + }, + "allocated_bytes_before": 12271139840, + "reserved_bytes_before": 18274582528, + "peak_allocated_bytes": 12542427136, + "peak_reserved_bytes": 18274582528, + "peak_allocated_delta_bytes": 271287296, + "peak_reserved_delta_bytes": 0 + } + }, + { + "block": 24, + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "q_stride": [ + 813066240, + 21504, + 128, + 1 + ], + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_mean": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "attention_output": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_entry": { + "timing": { + "count": 5, + "mean_s": 0.015854873847961425, + "p50_s": 0.015605952262878418, + "p90_s": 0.016245626068115235, + "p95_s": 0.016249741363525392, + "p99_s": 0.016253033599853514, + "min_s": 0.015590432167053223, + "max_s": 0.016253856658935546 + }, + "allocated_bytes_before": 12271885312, + "reserved_bytes_before": 20443037696, + "peak_allocated_bytes": 12544190464, + "peak_reserved_bytes": 20443037696, + "peak_allocated_delta_bytes": 272305152, + "peak_reserved_delta_bytes": 0 + }, + "candidate_entry": { + "timing": { + "count": 5, + "mean_s": 0.012546982383728028, + "p50_s": 0.012344767570495605, + "p90_s": 0.012899776077270508, + "p95_s": 0.012985503959655762, + "p99_s": 0.013054086265563966, + "min_s": 0.012335616111755371, + "max_s": 0.013071231842041016 + }, + "allocated_bytes_before": 12271885312, + "reserved_bytes_before": 20443037696, + "peak_allocated_bytes": 12544190464, + "peak_reserved_bytes": 20443037696, + "peak_allocated_delta_bytes": 272305152, + "peak_reserved_delta_bytes": 0 + } + }, + { + "block": 49, + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "q_stride": [ + 813066240, + 21504, + 128, + 1 + ], + "q_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_prepared": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "q_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_mean": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_int8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "k_scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "attention_output": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_entry": { + "timing": { + "count": 5, + "mean_s": 0.015745088005065917, + "p50_s": 0.01554531192779541, + "p90_s": 0.01607791976928711, + "p95_s": 0.016113983535766603, + "p99_s": 0.016142834548950193, + "min_s": 0.01552620792388916, + "max_s": 0.016150047302246093 + }, + "allocated_bytes_before": 12271139840, + "reserved_bytes_before": 20443037696, + "peak_allocated_bytes": 12542427136, + "peak_reserved_bytes": 20443037696, + "peak_allocated_delta_bytes": 271287296, + "peak_reserved_delta_bytes": 0 + }, + "candidate_entry": { + "timing": { + "count": 5, + "mean_s": 0.012486035346984864, + "p50_s": 0.012309951782226563, + "p90_s": 0.012827526664733887, + "p95_s": 0.012944339561462402, + "p99_s": 0.013037789878845216, + "min_s": 0.01228332805633545, + "max_s": 0.013061152458190919 + }, + "allocated_bytes_before": 12271139840, + "reserved_bytes_before": 20443037696, + "peak_allocated_bytes": 12542427136, + "peak_reserved_bytes": 20443037696, + "peak_allocated_delta_bytes": 271287296, + "peak_reserved_delta_bytes": 0 + } + } + ] +} diff --git a/benchmarks/gb10-sage2-p2-vprep-analysis.json b/benchmarks/gb10-sage2-p2-vprep-analysis.json new file mode 100644 index 0000000..5487e0c --- /dev/null +++ b/benchmarks/gb10-sage2-p2-vprep-analysis.json @@ -0,0 +1,51 @@ +{ + "status": "rejected", + "reason": "The exact candidate missed the 6.0 ms isolated go threshold and cannot reliably clear the 1% complete-block gate.", + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "candidate": "Direct strided-NHD BF16 V to Sage2 padded/permuted FP8 V preparation", + "contract": { + "tensor_layout": "NHD", + "head_dimension": 128, + "scale_max": 2.25, + "mainloop": "unchanged SageAttention 2.2.0 SM89 INT8-QK/FP8-PV mainloop" + }, + "correctness": { + "sequence_lengths": [1, 31, 32, 33, 63, 64, 65, 127, 128, 129, 37760, 37761, 37810], + "heads": 56, + "fp8_v": "byte-exact", + "fp32_v_scale": "byte-exact" + }, + "canonical_isolated_timing": { + "sequence": 37810, + "warmup": 5, + "iterations": 20, + "baseline_p50_ms": 10.56272, + "candidate_p50_ms": 6.392816, + "preparation_improvement_percent": 39.477, + "absolute_saving_ms": 4.169904, + "complete_block_baseline_ms": 458.78, + "projected_complete_block_improvement_percent": 0.909 + }, + "gate": { + "go_p50_ms": 6.0, + "stop_p50_ms": 6.5, + "result": "reject after the required borderline repeat", + "complete_block_benchmark": "skipped", + "trajectory_validation": "skipped", + "runtime_integration": "not attempted" + }, + "implementation": { + "stages": [ + "64-row per-channel BF16 absolute maxima", + "Sage-compatible approximate-reciprocal FP32 scale finalization", + "direct E4M3 quantization into Sage2's padded and sequence-permuted layout" + ], + "intermediate_bf16_transpose": "removed", + "production_dispatch": "unchanged" + }, + "source_artifacts": [ + "benchmarks/gb10-sage2-vprep-randomized.json", + "benchmarks/gb10-sage2-vprep-canonical-repeat.json" + ] +} diff --git a/benchmarks/gb10-sage2-p3-temporal-pair-analysis.json b/benchmarks/gb10-sage2-p3-temporal-pair-analysis.json new file mode 100644 index 0000000..0c6453d --- /dev/null +++ b/benchmarks/gb10-sage2-p3-temporal-pair-analysis.json @@ -0,0 +1,56 @@ +{ + "status": "rejected", + "reason": "The exact temporal two-pair QK/PV schedule improved the isolated mainloop by only 0.10% and missed the absolute 220 ms gate.", + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "candidate": { + "name": "temporal-pair", + "description": "Two warp pairs alternate QK/online-softmax and prior-tile PV while preserving private per-warp scores, softmax state, and output accumulators.", + "production_dispatch": "unchanged" + }, + "correctness": { + "compute_sanitizer_errors": 0, + "short_shape_cases": 13, + "short_shape_result": "byte-exact", + "real_block": 24, + "real_tokens": 37810, + "real_heads": 56, + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "candidate_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574" + }, + "ptxas": { + "baseline": { + "registers_per_thread": 255, + "static_spill_stores_bytes": 24, + "static_spill_loads_bytes": 24 + }, + "candidate": { + "registers_per_thread": 254, + "static_spill_stores_bytes": 32, + "static_spill_loads_bytes": 24 + }, + "dynamic_shared_memory_bytes": 32768 + }, + "timing": { + "rounds": 5, + "iterations_per_round": 10, + "baseline_p50_ms": 245.437279, + "candidate_p50_ms": 245.198769, + "speedup_percent": 0.097272, + "candidate_gate_ms": 220.0, + "gate_result": "fail" + }, + "decision": { + "ncu_followup": "skipped because the latency gate failed", + "block_integration": "skipped", + "trajectory_validation": "skipped", + "runtime_integration": "not attempted", + "conclusion": "Temporal separation alone does not unlock material INT-QK/FP8-PV overlap in the existing SM89 instruction stream on SM121. Further exact work requires a substantially different mainloop or native Blackwell numerical contract." + }, + "source_artifacts": [ + "research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch", + "benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json", + "benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json" + ] +} diff --git a/benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json b/benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json new file mode 100644 index 0000000..54da9f3 --- /dev/null +++ b/benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json @@ -0,0 +1,126 @@ +{ + "device": "NVIDIA GB10", + "baseline": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "candidate": "/variants/temporal-pair/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "heads": 2, + "seed": 73021, + "cases": [ + { + "q_len": 1, + "kv_len": 1, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "801080bc831b88365357f4394c8d2ff233394c0741bfa8143a398478416e4817", + "candidate_sha256": "801080bc831b88365357f4394c8d2ff233394c0741bfa8143a398478416e4817" + }, + { + "q_len": 31, + "kv_len": 63, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "f345fda4a687e8437fd6203287f3bccab2378171d55c11c3f12e593cca4bb858", + "candidate_sha256": "f345fda4a687e8437fd6203287f3bccab2378171d55c11c3f12e593cca4bb858" + }, + { + "q_len": 32, + "kv_len": 64, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "8982ac33531ff95b75fa817f4e349008ad2a448a65fc93424f21256ceafa8bff", + "candidate_sha256": "8982ac33531ff95b75fa817f4e349008ad2a448a65fc93424f21256ceafa8bff" + }, + { + "q_len": 33, + "kv_len": 65, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "c5c0654f71008a76a456f233015f93fcf4ee2f2031addfc6ee80c610a49b0fc2", + "candidate_sha256": "c5c0654f71008a76a456f233015f93fcf4ee2f2031addfc6ee80c610a49b0fc2" + }, + { + "q_len": 63, + "kv_len": 127, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "43d46907c5a999b9451a7f93f766573c705653f3a348275f4810626310204aea", + "candidate_sha256": "43d46907c5a999b9451a7f93f766573c705653f3a348275f4810626310204aea" + }, + { + "q_len": 64, + "kv_len": 128, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "ceca4efa59663e7b124e3cb1b894880e9c7c0c8df47826d203493cfa29d60de2", + "candidate_sha256": "ceca4efa59663e7b124e3cb1b894880e9c7c0c8df47826d203493cfa29d60de2" + }, + { + "q_len": 65, + "kv_len": 129, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "06feda89c48529cec5acc02a69f96a07f6358c5cad09a505d6c449f8e2051738", + "candidate_sha256": "06feda89c48529cec5acc02a69f96a07f6358c5cad09a505d6c449f8e2051738" + }, + { + "q_len": 127, + "kv_len": 63, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "9110dd5480fc2f9513aa563f1bf1bb59d01ebd70a8b12b45748e258c3013514b", + "candidate_sha256": "9110dd5480fc2f9513aa563f1bf1bb59d01ebd70a8b12b45748e258c3013514b" + }, + { + "q_len": 128, + "kv_len": 64, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "9154dd53fecd683f84da37dff679444e2f57feda74078404946129afbc173a1f", + "candidate_sha256": "9154dd53fecd683f84da37dff679444e2f57feda74078404946129afbc173a1f" + }, + { + "q_len": 129, + "kv_len": 65, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "7ef639d3f497cf6a9de9fbd2046d7d2510851616f7567f9cadae59661b947f2b", + "candidate_sha256": "7ef639d3f497cf6a9de9fbd2046d7d2510851616f7567f9cadae59661b947f2b" + }, + { + "q_len": 191, + "kv_len": 191, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "a4d171bfd43ca1b3c8620a9ddd4f4343d2b322dadeb8134c4c0be5ebd8358b28", + "candidate_sha256": "a4d171bfd43ca1b3c8620a9ddd4f4343d2b322dadeb8134c4c0be5ebd8358b28" + }, + { + "q_len": 192, + "kv_len": 192, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "e5b7c5c194dd5d4c78950711445696204638a376d981aa50e5f9f07fb0c9648d", + "candidate_sha256": "e5b7c5c194dd5d4c78950711445696204638a376d981aa50e5f9f07fb0c9648d" + }, + { + "q_len": 193, + "kv_len": 193, + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "baseline_sha256": "47c1c4766fe2d8ee067498244b586dcc5521e467c6c9fd1569688ad0e0501785", + "candidate_sha256": "47c1c4766fe2d8ee067498244b586dcc5521e467c6c9fd1569688ad0e0501785" + } + ] +} diff --git a/benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json b/benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json new file mode 100644 index 0000000..9ceb377 --- /dev/null +++ b/benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json @@ -0,0 +1,198 @@ +{ + "metadata": { + "width": 1344, + "height": 768, + "frames": 124, + "steps": 12, + "sampler_step": 1, + "seed": 440420, + "text_tokens": 100, + "block_index": 24, + "attention": "sage2", + "hidden_shape": [ + 37810, + 5376 + ], + "h_msa_shape": [ + 37810, + 5376 + ], + "segments": [ + [ + 0, + 100, + 1 + ], + [ + 100, + 514, + 2 + ], + [ + 514, + 37810, + 0 + ] + ] + }, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "q_shape": [ + 1, + 37810, + 56, + 128 + ], + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": 16576 + }, + "rounds": 5, + "warmup": 3, + "iterations_per_round": 10, + "variants": [ + { + "name": "baseline", + "path": "/variants/baseline/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.2445254299926758, + "p50_s": 0.2454372787475586, + "p90_s": 0.24733047790527343, + "p95_s": 0.2478844924926758, + "p99_s": 0.24838713729858397, + "min_s": 0.2394311065673828, + "max_s": 0.24867494201660156 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24067847290039063, + "p50_s": 0.24043912506103515, + "p90_s": 0.24227873840332034, + "p95_s": 0.24245605773925782, + "p99_s": 0.2425979132080078, + "min_s": 0.2394311065673828, + "max_s": 0.2426333770751953 + }, + { + "count": 10, + "mean_s": 0.24265675506591794, + "p50_s": 0.2430414047241211, + "p90_s": 0.2439772232055664, + "p95_s": 0.24400416030883787, + "p99_s": 0.24402570999145506, + "min_s": 0.2410487060546875, + "max_s": 0.24403109741210938 + }, + { + "count": 10, + "mean_s": 0.24536304473876952, + "p50_s": 0.2454372787475586, + "p90_s": 0.245896842956543, + "p95_s": 0.2459307289123535, + "p99_s": 0.24595783767700197, + "min_s": 0.24456390380859375, + "max_s": 0.24596461486816407 + }, + { + "count": 10, + "mean_s": 0.24639998474121091, + "p50_s": 0.24632894134521482, + "p90_s": 0.24672851867675782, + "p95_s": 0.2470246139526367, + "p99_s": 0.24726149017333982, + "min_s": 0.24594912719726564, + "max_s": 0.24732070922851562 + }, + { + "count": 10, + "mean_s": 0.24752889251708984, + "p50_s": 0.2473423309326172, + "p90_s": 0.24814632110595702, + "p95_s": 0.2484106315612793, + "p99_s": 0.24862207992553711, + "min_s": 0.2467729034423828, + "max_s": 0.24867494201660156 + } + ], + "speedup_vs_baseline_percent": 0.0 + }, + { + "name": "temporal-pair", + "path": "/variants/temporal-pair/lib/sageattention/_qattn_sm89.cpython-312-aarch64-linux-gnu.so", + "output_sha256": "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574", + "byte_exact": true, + "timing": { + "count": 50, + "mean_s": 0.24434641784667968, + "p50_s": 0.24519876861572265, + "p90_s": 0.2470915344238281, + "p95_s": 0.2473576820373535, + "p99_s": 0.24812875885009766, + "min_s": 0.23884719848632813, + "max_s": 0.2482449035644531 + }, + "rounds": [ + { + "count": 10, + "mean_s": 0.24035887298583986, + "p50_s": 0.2405825424194336, + "p90_s": 0.24129256286621092, + "p95_s": 0.24146988220214843, + "p99_s": 0.24161173767089844, + "min_s": 0.23884719848632813, + "max_s": 0.24164720153808594 + }, + { + "count": 10, + "mean_s": 0.2422781478881836, + "p50_s": 0.2422403030395508, + "p90_s": 0.24364151611328125, + "p95_s": 0.2438028366088867, + "p99_s": 0.2439318930053711, + "min_s": 0.24067245483398436, + "max_s": 0.24396415710449218 + }, + { + "count": 10, + "mean_s": 0.24521179351806638, + "p50_s": 0.24519876861572265, + "p90_s": 0.2460178466796875, + "p95_s": 0.2461315963745117, + "p99_s": 0.2462225961303711, + "min_s": 0.24424879455566406, + "max_s": 0.24624534606933593 + }, + { + "count": 10, + "mean_s": 0.24673807983398435, + "p50_s": 0.2465936508178711, + "p90_s": 0.24718292388916016, + "p95_s": 0.2475953987121582, + "p99_s": 0.24792537857055663, + "min_s": 0.24614927673339843, + "max_s": 0.24800787353515624 + }, + { + "count": 10, + "mean_s": 0.24714519500732424, + "p50_s": 0.24705561828613282, + "p90_s": 0.24748206787109375, + "p95_s": 0.24786348571777345, + "p99_s": 0.24816861999511716, + "min_s": 0.24656553649902344, + "max_s": 0.2482449035644531 + } + ], + "speedup_vs_baseline_percent": 0.09727215727162175 + } + ] +} diff --git a/benchmarks/gb10-sage2-vprep-canonical-repeat.json b/benchmarks/gb10-sage2-vprep-canonical-repeat.json new file mode 100644 index 0000000..a100a03 --- /dev/null +++ b/benchmarks/gb10-sage2-vprep-canonical-repeat.json @@ -0,0 +1,53 @@ +{ + "status": "pass", + "contract": { + "tensor_layout": "NHD", + "head_dim": 128, + "scale_max": 2.25, + "output": "Sage2 padded/permuted E4M3 V and FP32 per-channel scale" + }, + "cases": [ + { + "sequence": 37810, + "heads": 56, + "stride": [ + 813066240, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_timing": { + "count": 20, + "mean_s": 0.010601052856445314, + "p50_s": 0.010562719821929932, + "p90_s": 0.010675382804870605, + "p95_s": 0.010855110502243043, + "p99_s": 0.010869799871444702, + "min_s": 0.010542431831359863, + "max_s": 0.010873472213745117 + }, + "candidate_timing": { + "count": 20, + "mean_s": 0.006498015952110291, + "p50_s": 0.0063928160667419435, + "p90_s": 0.0067695615291595455, + "p95_s": 0.006918193387985229, + "p99_s": 0.006928189001083374, + "min_s": 0.006376319885253906, + "max_s": 0.00693068790435791 + } + } + ] +} diff --git a/benchmarks/gb10-sage2-vprep-randomized.json b/benchmarks/gb10-sage2-vprep-randomized.json new file mode 100644 index 0000000..7df51af --- /dev/null +++ b/benchmarks/gb10-sage2-vprep-randomized.json @@ -0,0 +1,357 @@ +{ + "status": "pass", + "contract": { + "tensor_layout": "NHD", + "head_dim": 128, + "scale_max": 2.25, + "output": "Sage2 padded/permuted E4M3 V and FP32 per-channel scale" + }, + "cases": [ + { + "sequence": 1, + "heads": 56, + "stride": [ + 7168, + 7168, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 31, + "heads": 56, + "stride": [ + 666624, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 32, + "heads": 56, + "stride": [ + 688128, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 33, + "heads": 56, + "stride": [ + 709632, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 63, + "heads": 56, + "stride": [ + 1354752, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 64, + "heads": 56, + "stride": [ + 1376256, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 65, + "heads": 56, + "stride": [ + 1397760, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 127, + "heads": 56, + "stride": [ + 2731008, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 128, + "heads": 56, + "stride": [ + 2752512, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 129, + "heads": 56, + "stride": [ + 2774016, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + } + }, + { + "sequence": 37760, + "heads": 56, + "stride": [ + 811991040, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_timing": { + "count": 8, + "mean_s": 0.010765039920806884, + "p50_s": 0.01076302433013916, + "p90_s": 0.010959852409362793, + "p95_s": 0.010992982006072998, + "p99_s": 0.011019485683441162, + "min_s": 0.010565183639526366, + "max_s": 0.011026111602783203 + }, + "candidate_timing": { + "count": 8, + "mean_s": 0.006614172041416168, + "p50_s": 0.006597951889038086, + "p90_s": 0.006795004796981812, + "p95_s": 0.006805454468727112, + "p99_s": 0.0068138142061233515, + "min_s": 0.006458144187927246, + "max_s": 0.006815904140472412 + } + }, + { + "sequence": 37761, + "heads": 56, + "stride": [ + 812012544, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_timing": { + "count": 8, + "mean_s": 0.0109060640335083, + "p50_s": 0.010748064041137696, + "p90_s": 0.011263570976257326, + "p95_s": 0.011430585479736327, + "p99_s": 0.011564197082519531, + "min_s": 0.010667872428894044, + "max_s": 0.011597599983215332 + }, + "candidate_timing": { + "count": 8, + "mean_s": 0.0065507440567016606, + "p50_s": 0.006485184192657471, + "p90_s": 0.006748016023635865, + "p95_s": 0.006786712002754211, + "p99_s": 0.006817668786048889, + "min_s": 0.006461887836456299, + "max_s": 0.0068254079818725584 + } + }, + { + "sequence": 37810, + "heads": 56, + "stride": [ + 813066240, + 21504, + 128, + 1 + ], + "fp8": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "scale": { + "equal": true, + "different_elements": 0, + "max_abs": 0.0, + "mean_abs": 0.0 + }, + "baseline_timing": { + "count": 8, + "mean_s": 0.01064847207069397, + "p50_s": 0.010628528118133545, + "p90_s": 0.010686016178131105, + "p95_s": 0.010728799867630005, + "p99_s": 0.010763026819229126, + "min_s": 0.010623680114746094, + "max_s": 0.010771583557128907 + }, + "candidate_timing": { + "count": 8, + "mean_s": 0.006505947947502136, + "p50_s": 0.006436048030853271, + "p90_s": 0.00664213752746582, + "p95_s": 0.006790044784545898, + "p99_s": 0.006908370590209961, + "min_s": 0.006423327922821045, + "max_s": 0.006937952041625977 + } + } + ] +} diff --git a/benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json b/benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json new file mode 100644 index 0000000..a1c2750 --- /dev/null +++ b/benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json @@ -0,0 +1,143 @@ +{ + "device": "NVIDIA GB10", + "compute_capability": "SM121", + "torch": "2.9.1+cu130", + "sageattention": "2.2.0", + "resolution": [ + 1344, + 768 + ], + "frames": 124, + "packed_tokens": 37810, + "seed": 440420, + "sampling_uninstrumented_two_step_seconds": { + "sage2": 53.53981032199226, + "sol_bf16_tau_1_3": 46.18868866603589, + "sol_int8_qk_tau_1_3": 42.72281823604135 + }, + "sampling_stage_step_2_seconds": { + "sage2": { + "packing": 0.015245259972289205, + "denoiser": 25.396856592036784, + "output_transform": 0.0006340660038404167, + "sampler_update": 0.000779024965595454, + "step_total": 25.413675360032357 + }, + "sol_bf16_tau_1_3": { + "packing": 0.01417812000727281, + "denoiser": 21.55711886996869, + "output_transform": 0.0006771690095774829, + "sampler_update": 0.0007963530370034277, + "step_total": 21.572905345004983 + }, + "sol_int8_qk_tau_1_3": { + "packing": 0.014827414997853339, + "denoiser": 19.75738707598066, + "output_transform": 0.0006591529818251729, + "sampler_update": 0.0006342889973893762, + "step_total": 19.773632204975 + } + }, + "representative_block_24_p50_milliseconds": { + "sage2": { + "module_forward": 514.917, + "norm": 7.155, + "modulate": 21.734, + "qkv_projection": 30.773, + "rms_rope": 12.535, + "qkv_and_output_layout": 20.346, + "attention": 259.942, + "output_projection": 13.366, + "residual_gates": 17.335, + "mlp_fc1": 39.577, + "swiglu": 25.821, + "mlp_fc2": 67.35 + }, + "sol_bf16_tau_1_3": { + "module_forward": 385.35, + "norm": 7.131, + "modulate": 21.318, + "qkv_projection": 30.116, + "rms_rope": 12.512, + "qkv_and_output_layout": 14.7, + "attention": 141.829, + "output_projection": 12.893, + "residual_gates": 17.469, + "mlp_fc1": 39.171, + "swiglu": 24.015, + "mlp_fc2": 64.833 + }, + "sol_int8_qk_tau_1_3": { + "module_forward": 364.141, + "norm": 7.049, + "modulate": 20.848, + "qkv_projection": 29.719, + "rms_rope": 12.327, + "qkv_and_output_layout": 14.849, + "attention": 120.272, + "output_projection": 13.165, + "residual_gates": 17.433, + "mlp_fc1": 39.411, + "swiglu": 23.971, + "mlp_fc2": 65.111 + } + }, + "attention_internal_p50_milliseconds": { + "sage2": { + "qk_quantize": 6.45, + "v_quantize": 10.185, + "attention_forward": 238.995, + "framework_and_k_smoothing_estimate": 2.982, + "total_excluding_external_layout": 258.613 + }, + "sol_bf16_tau_1_3_exact_conditioning_kv": { + "k_summary": 2.44, + "v_summary": 2.411, + "routing_threshold": 2.423, + "prepare_total": 7.287, + "routed_forward_estimate": 143.455, + "total": 150.741 + }, + "sol_int8_qk_tau_1_3_exact_conditioning_kv": { + "k_summary_and_quantize": 4.449, + "v_summary": 2.454, + "q_quantize_and_threshold": 4.27, + "prepare_total": 11.533, + "routed_forward_estimate": 114.882, + "total": 126.416 + }, + "sol_int8_qk_pv_tau_1_3_exact_conditioning_kv": { + "prepare_total": 11.714, + "v_int8_quantize": 3.875, + "routed_forward_estimate": 126.936, + "total": 138.651 + } + }, + "quality_gates": { + "sol_bf16_tau_1_3_exact_conditioning_kv_relative_l2_vs_sage2": 0.6257168054580688, + "sol_int8_qk_tau_1_3_exact_conditioning_kv_relative_l2_vs_sage2": 0.6258820295333862, + "sol_fully_dense_tau_minus_100_relative_l2_vs_sage2": 0.017087, + "sol_fully_dense_tau_minus_100_p50_milliseconds": 947.901, + "conclusion": "Current Sol routing has no point that is both faster than Sage2 and numerically Sage-like; sparse timings are optimization bounds, not an accepted backend." + }, + "sage2_block_share_estimates": { + "attention": 0.505, + "nvfp4_projections": 0.293, + "modulation_gates_and_swiglu": 0.126, + "layout": 0.04, + "norm_and_rope": 0.038 + }, + "measurement_policy": { + "sampling_totals": "uninstrumented wall time", + "sampling_stages": "CUDA-synchronized stage attribution", + "block_components": "CUDA-synchronized representative block 24 medians", + "attention_components": "nested CUDA-synchronized probes; residual values are p50 difference estimates", + "resident_services": "H3 hot runtime and Qwen vLLM remained loaded and idle; medians are preferred over means" + }, + "next_targets": [ + "Bit-exact fused H3 modulation and residual gates", + "SM121-specific NVFP4 projection and epilogue fusion", + "A dense Sage-compatible kernel that reads strided BSHD QKV without materialization", + "A quality-gated timestep/block hybrid only after full latent and video evaluation" + ] +} diff --git a/compose.spark-stack.yml b/compose.spark-stack.yml index 17049dd..a427862 100644 --- a/compose.spark-stack.yml +++ b/compose.spark-stack.yml @@ -17,6 +17,10 @@ services: H3_DISABLE_MMAP: "1" H3_NVFP4_SCALE_BACKEND: "vortex" H3_NVFP4_SCALE_VERSION: "1" + H3_FUSED_ELEMENTWISE: "1" + H3_NVFP4_MODULATE_FUSION: "1" + H3_NVFP4_SWIGLU_FUSION: "1" + H3_SAGE_QKV_LAYOUT: "strided_nhd" H3_SOL_QKV_LAYOUT: "native" command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--warmup"] diff --git a/compose.spark.yml b/compose.spark.yml index f59e2ea..2f9c0ce 100644 --- a/compose.spark.yml +++ b/compose.spark.yml @@ -34,5 +34,9 @@ services: H3_DISABLE_MMAP: "1" H3_NVFP4_SCALE_BACKEND: "vortex" H3_NVFP4_SCALE_VERSION: "1" + H3_FUSED_ELEMENTWISE: "1" + H3_NVFP4_MODULATE_FUSION: "1" + H3_NVFP4_SWIGLU_FUSION: "1" + H3_SAGE_QKV_LAYOUT: "strided_nhd" H3_SOL_QKV_LAYOUT: "native" command: ["python", "/opt/h3-blackwell-runtime/tools/serve_hot_runtime.py", "--host", "0.0.0.0", "--port", "8000", "--attention", "sage2", "--turbo-lora", "4step=/turbo/minimax_h3_fl2v_turbo_4step_v1.1_768p_comfyui_bf16.safetensors", "--turbo-lora", "8step=/turbo/minimax_h3_fl2v_turbo_8step_v1.0_comfyui_bf16.safetensors", "--latent-upscaler", "/latent-upscaler/minimax_h3_latent_upscaler_3d_fp16.safetensors", "--warmup"] diff --git a/pyproject.toml b/pyproject.toml index d74f797..4ff4f44 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,6 +7,7 @@ dependencies = [ "comfy-kitchen==0.2.31", "fastsafetensors>=0.1.10", "safetensors>=0.5.0", + "scipy>=1.13", "torch==2.9.1+cu130", "transformers>=4.51,<5" ] diff --git a/research/BUYER_QUESTIONS_STATUS.md b/research/BUYER_QUESTIONS_STATUS.md new file mode 100644 index 0000000..72ebd0d --- /dev/null +++ b/research/BUYER_QUESTIONS_STATUS.md @@ -0,0 +1,693 @@ +# Buyer Questions: Current Evidence Status + +**Internal - do not share without review** +**Evidence date:** 2026-08-25 +**Purpose:** Track what Vortex can currently answer, what is only partially evidenced, and what remains open. + +## Status Summary + +| Status | Count | Meaning | +|---|---:|---| +| Answerable | 1 | The current product state supports a direct, evidence-backed answer. The answer may be negative. | +| Partially answerable | 13 | Useful measurements or implementation facts exist, but they do not completely answer the buyer's exact question. | +| Open | 11 | The required comparison, integration, evaluation, or operating evidence does not exist. | +| **Total** | **25** | 14 performance questions and 11 quality questions. | + +These classifications measure whether the buyer's full question can be answered, not whether Vortex has done related technical work. Component-level checksums and kernel timings are useful evidence, but they are not substitutes for end-to-end performance, reliability, or perceptual-quality evidence. + +## Evidence Rules + +- Same-workload cross-hardware measurements must not be presented as Vortex-versus-official-runtime comparisons. +- Scalar checksums do not prove complete tensor equality. +- Kernel and block parity do not prove final video or audio parity. +- Native sampling resolution must be distinguished from latent-upscaled output. +- A demonstrated workload is not a proven maximum-safe workload. +- Projected performance is not measured performance. +- Historical Markdown can describe superseded states. `CURRENT_STATE.md` is the canonical status source where documents conflict. +- Much of the newest evidence is currently preserved in the working tree rather than an earlier Git commit. Release claims should cite the eventual preservation commit. + +# Performance Questions + +## 1. What models are supported? + +**Status: Open** + +### Current answer + +The implemented runtime currently supports the MiniMax H3 stack centered on the pruned FL2VA NVFP4 denoiser and Qwen3-VL 32B NVFP4-AWQ conditioner. Implemented H3 modes include prompt-only T2VA, first-frame I2VA, last-frame L2VA, first/last-frame FL2VA, joint video and stereo audio, base beta/RES sampling, official Turbo 4-step v1.1 and 8-step v1.0 adapters, and an optional learned latent upscaler. + +Full arbitrary Ref2VA video, audio, identity, and voice conditioning is not implemented. + +This does not meet the required buyer-facing model scope. The support matrix must also include: + +- Wan, with exact model families, variants, revisions, and task modes still to be selected. +- LTX 2.3. +- LTX 2.5. +- Any additional strategically selected model family. + +### Existing evidence + +- `README.md` +- `CURRENT_STATE.md` +- `TURBO.md` +- `H3_LATENT_UPSCALER.md` +- `src/h3_blackwell_runtime/runtime.py` + +### Remaining gap + +There is no implemented or validated Vortex runtime for the required Wan and LTX versions. The phrase "supports Wan" is also too broad until exact checkpoints and modes are pinned. + +### Next action + +Define a versioned model-support contract. For every selected model, record checkpoint revision and SHA-256, T2V/I2V capabilities, audio support, precision modes, valid step counts, native resolutions, duration limits, supported GPUs, parity status, and measured performance. Then integrate and validate Wan and LTX 2.3/2.5 against that contract. + +## 2. What is Vortex versus the official H3 runtime on the same RTX PRO 6000? + +**Status: Open** + +### Current answer + +No same-RTX-PRO-6000 comparison against the official H3 runtime exists. The measured `4.44x` result compares the same direct SDPA workload on RTX PRO 6000 and GB10. It is a cross-hardware result, not a Vortex-versus-official-runtime result. + +The closest older same-hardware evidence is a GB10 direct-versus-Comfy warm comparison around `149.304s` versus `150.26s` at 960x544x124. It is not an isolated RTX comparison and does not establish a Vortex advantage. + +### Existing evidence + +- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json` +- `PARITY.md` + +### Remaining gap + +There is no licensed or pinned official runtime running beside Vortex on the same RTX GPU, checkpoint, prompt, seed, precision, attention backend, and output settings. + +### Next action + +Run alternated cold and warm trials of official H3 and Vortex on one RTX PRO 6000. Retain full stage timings, final video/audio latents, decoded media, environment versions, and checkpoint hashes. + +## 3. What is Vortex versus reference on the same B200 or GB300 GPU? + +**Status: Open** + +### Current answer + +B200 has been tested only as a hardware comparison against RTX PRO 6000 using the same direct SDPA runner. At 1344x768x124, B200 sampling was `79.7213s` versus RTX `97.9439s`, a `1.2286x` speedup. At 1344x768x243, B200 was `257.8620s` versus RTX `311.0163s`, a `1.2061x` speedup. Under the recorded provider prices, B200 sampling cost was approximately `2.64-2.69x` the RTX cost. + +These results do not compare Vortex with the official runtime on B200. No GB300 evidence exists. + +### Existing evidence + +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` + +### Remaining gap + +There is no same-B200 official-runtime baseline, no architecture-matched attribution explaining the modest B200 speedup, and no GB300 run. + +### Next action + +Run official H3 and Vortex on the same B200, then repeat the unchanged protocol on GB300. Record clocks, power, kernels, precision, complete timings, latents, and media. + +## 4. How much gain comes from hardware, NVFP4, attention, fusion, runtime residency, and distributed execution? + +**Status: Partially answerable** + +### Current answer + +GB10 block composition and several isolated software gains are measured. In a fully fused block-24 profile, Sage2 represented approximately `57.60%` of block time, NVFP4 GEMMs `26.87%`, NVFP4 scale and packing `8.92%`, norm/RoPE `4.27%`, and gates `2.35%`. + +Measured controlled improvements include: + +- Fused elementwise path: approximately `3.0%` over the 12-step run and about 390 MiB lower peak allocation. +- Strided-NHD QKV copy removal: approximately `3.6%` over its baseline. +- NVFP4 modulation fusion: approximately `0.56%`. +- NVFP4 SwiGLU producer fusion: approximately `4.07%`. +- Two-GPU RTX Ulysses: `1.21x`, `1.55x`, and `1.72x` as workload size increased. + +These numbers do not form a complete additive decomposition. NVFP4's share of runtime is not the gain caused by NVFP4, and independently measured percentages cannot safely be summed. + +### Existing evidence + +- `benchmarks/gb10-fully-fused-fresh-nsight-summary.json` +- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json` +- `benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json` +- `benchmarks/gb10-nvfp4-modulate-fusion-summary.json` +- `benchmarks/gb10-nvfp4-swiglu-fusion-summary.json` +- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json` + +### Remaining gap + +There is no controlled BF16-versus-NVFP4 model comparison, no complete residency ablation, and no factorial matrix across hardware, attention, fusion, precision, and distribution. + +### Next action + +Add explicit component toggles and run a fixed hardware x precision x attention x fusion x residency x GPU-count matrix with parity gates and uninstrumented end-to-end timing. + +## 5. What are cold-start and warm-start times? + +**Status: Partially answerable** + +### Current answer + +A historical GB10 deployment recorded approximately `30.73s` for startup warmup and `2.322s` for a later resident 320x192x22 smoke request. RTX H3 model loading was approximately `4.33-4.82s`; B200 model loading was approximately `10.71-11.83s`. + +These values measure different scopes and workloads. They do not provide controlled process-launch-to-ready, first-request, and warmed-request timings for one configuration. + +### Existing evidence + +- `benchmarks/gb10-post-lora-revert-deployment-smoke.json` +- `benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json` +- `benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json` +- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json` +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` + +### Remaining gap + +Container startup, model loading, CUDA compilation, warmup, conditioning, sampling, decoding, encoding, and muxing have not been separated under one controlled protocol. + +### Next action + +Restart the service and issue one fixed cold request followed by at least ten identical warm requests. Record every stage separately. + +## 6. What are p50, p95, and p99 latencies? + +**Status: Partially answerable** + +### Current answer + +Percentiles exist for selected blocks and kernels. For example, fully fused GB10 block-24 p50 was approximately `458.775ms`. Some three-iteration component artifacts report interpolated p95 and p99 values, and a 50-sample Sage2 study reports stable isolated medians. + +There are no buyer-grade service-request or end-to-end p50, p95, and p99 measurements. Microkernel percentiles must not be presented as request latency. + +### Existing evidence + +- `benchmarks/gb10-fully-fused-fresh-nsight-summary.json` +- `benchmarks/gb10-cute-qkv-block0-baseline.json` +- `benchmarks/gb10-sage2-p3-temporal-pair-analysis.json` +- `PERFORMANCE_ROADMAP.md` + +### Remaining gap + +There is no sufficiently large resident-request distribution, queue timing, or full-media-completion distribution. + +### Next action + +Run at least 100 serialized resident requests per supported operating point and report p50/p95/p99 for queue time, request wall time, sampling, and full media completion. + +## 7. How does performance change across 4, 8, 12, and higher step counts? + +**Status: Partially answerable** + +### Current answer + +Official Turbo 4-step and 8-step modes and base 12-step mode are operational. One 960x544x124 first-frame case measured `60.4s` sampling for Turbo-4, `121.1s` for Turbo-8, and `148.1s` for the stated base 8-step comparison. Other canonical base results use 12 steps. + +Turbo and base runs use different adapters and schedules. They are not a controlled step-count-only sweep, and no buyer-grade result above 12 steps exists. + +### Existing evidence + +- `TURBO.md` +- `tests/test_turbo.py` +- `tools/serve_hot_runtime.py` +- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json` + +### Remaining gap + +No matched 4/8/12/16+ latency and quality sweep exists with adapter and sampler held or explicitly accounted for. + +### Next action + +Run a declared 4/8/12/16-step matrix and score both performance and final quality. Separate the effect of fewer steps from the effect of the Turbo adapter. + +## 8. How does it scale with 5-, 10-, 20-, and 30-second videos? + +**Status: Partially answerable** + +### Current answer + +Approximately five-second and ten-second workloads have been measured. At 24 fps, 124 frames is approximately `5.17s` and 243 frames is approximately `10.13s`. Source guidance describes roughly 124-362 frames, or about 5-15 seconds, as the trained range, with longer output untested. + +No 20- or 30-second performance, memory, completion, audio-continuity, or quality evidence exists. + +### Existing evidence + +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json` +- `src/h3_blackwell_runtime/upstream_nodes.py` + +### Remaining gap + +Longer requests may exceed memory or the model's trained range. That failure boundary is not known. + +### Next action + +Run native 5/10/20/30-second sweeps with peak memory, latency, completion rate, latent health, audio continuity, and temporal-quality review. + +## 9. What happens at native 720p, 1080p, 2K, and 4K? + +**Status: Partially answerable** + +### Current answer + +Demonstrated native sampling resolutions include 864x480, 960x544, and 1344x768. A 1920x1088 result exists after 2x latent upscaling from 960x544. It is not native 1080p generation. No native 2K or 4K evidence exists. + +The term "2K delivery" must therefore be described as an upscaled output workflow, not native 2K sampling. + +### Existing evidence + +- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json` +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `H3_LATENT_UPSCALER.md` +- `TURBO.md` + +### Remaining gap + +Exact native 1280x720 and 1920x1080 have not been validated, and memory-safe native 2K/4K limits are unknown. + +### Next action + +Test exact native 720p and 1080p first. Attempt native 2K and 4K only with explicit memory, duration, quality, and failure criteria. Keep upscaled controls separately labeled. + +## 10. What are maximum safe frame and token counts? + +**Status: Partially answerable** + +### Current answer + +The largest demonstrated complete sampling case is 1344x768x243 frames. The canonical 1344x768x124 workload contains 37,810 packed tokens. On B200, the 243-frame sampling case recorded approximately 22.96 GB peak sampling allocation. + +These are demonstrated points, not maximum-safe limits. The direct API does not currently enforce a validated maximum frame or token count. + +### Existing evidence + +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json` +- `tools/serve_hot_runtime.py` +- `src/h3_blackwell_runtime/upstream_nodes.py` + +### Remaining gap + +No push-to-failure matrix defines safe limits by GPU, model, resolution, duration, conditioning load, or concurrency. + +### Next action + +Increase frames and conditioning tokens until a predeclared memory, latency, quality, or error boundary is reached. Repeatedly validate the preceding safe point and enforce it in the API. + +## 11. Can one rack run many independent jobs efficiently, or is it optimized only for one distributed job? + +**Status: Partially answerable** + +### Current answer + +The resident HTTP service accepts threaded connections but serializes generation through a single runtime lock. It therefore executes one independent generation at a time per process. Distributed Ulysses or tensor execution instead shards one generation across GPUs. + +On two RTX PRO 6000 GPUs, one Ulysses job achieved speedups of `1.21x`, `1.55x`, and `1.72x` as workload size increased. This does not answer whether two independent one-GPU workers would provide better rack throughput or economics. + +### Existing evidence + +- `tools/serve_hot_runtime.py` +- `DISTRIBUTED.md` +- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json` + +### Remaining gap + +There is no rack scheduler, intentional concurrency contract, or equal-work comparison between independent workers and distributed jobs. + +### Next action + +On identical GPUs, compare concurrent one-GPU workers with one multi-GPU Ulysses job. Report aggregate throughput, p95 latency, VRAM, queue behavior, and cost. + +## 12. Does throughput remain stable after hours or days of continuous operation? + +**Status: Open** + +### Current answer + +No claim can currently be made. The repository contains repeatability studies, isolated 50-sample kernel tests, short benchmark matrices, and deployment smoke tests. It does not contain an hours- or days-long resident soak. + +### Existing evidence + +- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json` +- `benchmarks/gb10-sage2-p3-temporal-pair-analysis.json` +- `compose.spark-stack.yml` + +### Remaining gap + +No sustained throughput, memory-leak, thermal, clock, checksum, restart, or failure trend exists. + +### Next action + +Run a 24-hour soak followed by a 72-hour mixed-workload soak. Capture throughput, p50/p95/p99, host/GPU memory, allocator state, temperature, clocks, power, checksums, failures, and restarts. + +## 13. What happens when GPU memory becomes fragmented? + +**Status: Open** + +### Current answer + +Selected experiments report peak allocated and reserved memory, including an elementwise-fusion reduction of approximately 408.8 MB allocated and 130.0 MB reserved. These are single-run peaks and do not measure fragmentation. + +A duplicate-model FC1 experiment caused a pressure-related container stop, but its timing was invalid and it does not establish allocator behavior. + +### Existing evidence + +- `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json` +- `PERFORMANCE_ROADMAP.md` + +### Remaining gap + +There is no mixed-size workload, allocator fragmentation telemetry, largest-free-block tracking, or recovery test. + +### Next action + +Run hundreds of alternating small and large requests while recording allocated, reserved, inactive split blocks, largest free allocation, failures, and post-request recovery. + +## 14. What are failure and retry rates? + +**Status: Partially answerable** + +### Current answer + +The service returns HTTP 400 for selected validation failures and HTTP 500 for other exceptions. There is no automatic generation retry logic. Individual records mention a pre-sampling dependency failure and a container stop under experimental memory pressure, but no rate can be calculated. + +### Existing evidence + +- `tools/serve_hot_runtime.py` +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `CURRENT_STATE.md` +- `PERFORMANCE_ROADMAP.md` + +### Remaining gap + +Failure frequency, timeout behavior, cancellation, idempotency, cleanup, retry success, and durable job recovery are unmeasured. + +### Next action + +Add structured request outcome logging and run a fault-injection matrix covering invalid input, OOM, missing checkpoints, CUDA errors, FFmpeg failure, client disconnect, timeout, and process restart. + +# Quality Questions + +## 15. Are complete per-step latents equal to the official reference? + +**Status: Partially answerable** + +### Current answer + +One small fixed prompt-only FL2VA fixture at 320x192x22, seed 440204, and 12 steps has extensive ComfyUI-oracle replay evidence. The ledger reports exact H3 calls and RES updates at selected boundaries, with some wrapper and state-boundary qualifications. + +This does not prove complete per-step equality for the current production runtime across T2VA, I2VA, L2VA, FL2VA, Turbo, different prompts, or different hardware. Exact QDATA/SFA, projection, block, or encoder hashes prove bounded seams rather than complete end-to-end equality. + +### Existing evidence + +- `PARITY.md` +- `tools/compare_fl2va_steps.py` +- `CURRENT_STATE.md` + +### Remaining gap + +The complete official initial state, conditioning, every scheduler pre/post state, and final video/audio latent set are not preserved in one self-contained assertion-based fixture. + +### Next action + +Capture and assert `torch.equal` at every step for at least base T2VA, FL2VA, and Turbo-4 using pinned official-reference artifacts. + +## 16. Are final video and audio latents equal across hardware? + +**Status: Open** + +### Current answer + +Cross-hardware final-latent equality has not been tested. RTX repeat runs recorded identical scalar checksums, but complete latents were not retained. B200 records timing and scalar values without matching RTX or GB10 tensors. One-rank Ulysses and tensor modes were equal on one GB10, but that tests execution paths on one hardware class rather than cross-hardware equality. + +### Existing evidence + +- `benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json` +- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json` +- `DISTRIBUTED.md` + +### Remaining gap + +Complete matching video/audio tensors are absent across GB10, RTX PRO 6000, B200, and GB300. + +### Next action + +Save full final video and audio latents for identical requests on each available GPU. Compare raw-byte SHA-256, `torch.equal`, maximum and mean error, RMSE, and cosine similarity. + +## 17. Does NVFP4 introduce perceptual degradation relative to BF16? + +**Status: Open** + +### Current answer + +No defensible perceptual answer exists. The extensive NVFP4 parity work compares optimized implementations with the existing NVFP4/Comfy Kitchen path. It does not compare the pruned NVFP4 model's output quality with the official BF16 base. + +Turbo adapters were released for a BF16 base; use over the pruned NVFP4 base is operationally validated but not upstream-certified. + +### Existing evidence + +- `PLAN.md` +- `PERFORMANCE_ROADMAP.md` +- `TURBO.md` +- `research/EXPERIMENT_REGISTRY.md` + +### Remaining gap + +There is no matched BF16/NVFP4 prompt-and-seed corpus, latent comparison, video/audio metric set, or blinded preference study. + +### Next action + +Run official BF16 and pruned NVFP4 models at equal prompts and seeds. Retain all latents and decoded media, score objective quality, and conduct blinded human comparison. + +## 18. Are results stable across hundreds of prompts and seeds? + +**Status: Open** + +### Current answer + +No hundreds-of-prompts-and-seeds campaign exists. The strongest related evidence is a narrow ten-seed audio-format investigation and a ten-prompt text-conditioning repeatability tool. Neither is a broad end-to-end generation study. + +### Existing evidence + +- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json` +- `tools/compare_text_repeatability.py` +- `tests/` + +### Remaining gap + +There is no representative prompt taxonomy, broad seed matrix, retained media set, failure count, collapse rate, or quality distribution. + +### Next action + +Run at least 100 prompts across at least three seeds, spanning dialogue, music, action, static scenes, keyframes, identities, text, faces, hands, and difficult camera motion. + +## 19. What are temporal consistency, motion quality, and identity-preservation results? + +**Status: Open** + +### Current answer + +No systematic results exist. One Turbo comparison contains informal observations about composition, motion, and exposure drift. There are no standard temporal or identity metrics and no multi-seed evaluation. + +Full arbitrary identity/reference conditioning is also outside the currently implemented H3 direct-runtime scope. + +### Existing evidence + +- `TURBO.md` +- `VORTEX_RESEARCH_AGENDA.md` +- `CURRENT_STATE.md` + +### Remaining gap + +There is no subject-tracking suite, motion-adherence test, temporal perceptual score, optical-flow consistency score, or identity-embedding drift measurement. + +### Next action + +Build a fixed multi-seed suite with prescribed subjects and motion. Add temporal perceptual, flow-consistency, subject/face embedding, motion-adherence, and human-review scoring. + +## 20. How well does it handle text, hands, faces, and complex camera motion? + +**Status: Open** + +### Current answer + +The repository contains prompt guidance and isolated diagnostic prompts, not category-level quality measurements. No hand/finger benchmark, OCR suite, face-quality suite, or camera-trajectory evaluation exists. + +The latent upscaler is explicitly described as unsafe for text, QR codes, and identity-critical content, and it can alter faces. + +### Existing evidence + +- `H3_PROMPT_GUIDE.md` +- `H3_LATENT_UPSCALER.md` +- `benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json` + +### Remaining gap + +There is no curated hard-case corpus, automated task scoring, defect taxonomy, or blinded review. + +### Next action + +Create four explicit test buckets: readable text/OCR, hand-object interaction, close and multi-angle faces, and prescribed static/pan/dolly/handheld camera motion. + +## 21. What are speech quality, voice consistency, and lip-sync scores? + +**Status: Partially answerable** + +### Current answer + +Stereo 32 kHz audio generation, decoding, and muxing are operational. A ten-seed quoted-dialogue diagnostic found useful prompt-format behavior and received subjective WAV review. Quoted prompts removed immediate startup activity in all ten recorded cases. + +Speech accuracy, voice consistency, and full-video lip-sync remain unscored. Signal levels and successful muxing are not speech-quality evidence. + +### Existing evidence + +- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json` +- `AUDIO_BOUNDARY_INVESTIGATION.md` +- `CURRENT_STATE.md` +- `PARITY.md` + +### Remaining gap + +There are no ASR/WER results, speaker-embedding measurements, onset/offset accuracy, lip-sync metrics, or structured human scores. + +### Next action + +Generate full dialogue videos with known scripts and visible speakers. Retain lossless PCM and score WER, speaker consistency, speech timing, automated lip-sync, and blinded human lip-sync. + +## 22. Does the latent upscaler introduce ringing, texture, or chromatic artifacts? + +**Status: Partially answerable** + +### Current answer + +Yes, known examples show harsher illustrated texture, ringing or chromatic edges, and facial changes. One principal source-to-downscaled comparison recorded SSIM `0.924986`. The upscaler is not considered fidelity-safe for text, QR codes, or identity-critical content. + +What is unknown is how often these defects occur, their severity distribution, and whether they are preferable to conventional scaling. + +### Existing evidence + +- `H3_LATENT_UPSCALER.md` +- `CURRENT_STATE.md` +- `tests/test_latent_upscaler.py` + +### Remaining gap + +There is no representative multi-clip artifact study, edge-overshoot metric, color-fringe metric, LPIPS distribution, or blind crop review. + +### Next action + +Evaluate at least 20 clips containing text, high-contrast edges, faces, skin, hair, and fine texture against bicubic/Lanczos and native-resolution controls. + +## 23. Has there been a blind human A/B evaluation? + +**Status: Open** + +### Current answer + +No randomized, blinded, multi-rater A/B evaluation is documented. Existing audio, Turbo, and upscaler observations are informal and do not record blindness, randomization, rater count, inter-rater agreement, or confidence intervals. + +### Existing evidence + +- `benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json` +- `TURBO.md` +- `H3_LATENT_UPSCALER.md` + +### Remaining gap + +There is no evaluation protocol, recruited panel, anonymized media set, scoring rubric, or statistical analysis. + +### Next action + +Randomize and anonymize at least 30 matched clip pairs, recruit at least three independent raters, collect preference and defect labels, and report confidence intervals and agreement. + +## 24. Does the four-step Turbo model materially reduce quality? + +**Status: Partially answerable** + +### Current answer + +Turbo-4 is operational and materially faster, but its general quality delta is unknown. On one matched 960x544x124 case, Turbo-4 sampled in `60.4s`, Turbo-8 in `121.1s`, and the stated base comparison in `148.1s`. Informal review found Turbo-4 somewhat more compositionally stable in that example, Turbo-8 showed more motion, and both showed exposure drift. + +One example cannot establish population-level quality. There is no scored comparison with base 12-step output or official LightX2V BF16 Turbo. + +### Existing evidence + +- `TURBO.md` +- `benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json` +- `tests/test_turbo.py` + +### Remaining gap + +No matched prompt suite, latent-quality analysis, video/audio metrics, or blind preference result exists. + +### Next action + +Compare base 12-step, Turbo-8, and Turbo-4 over at least 20 prompts and three seeds, then separately compare NVFP4 Turbo-4 with the official BF16 path. + +## 25. Can customers choose between reference-quality and maximum-throughput modes? + +**Status: Answerable** + +### Current answer + +Not as coherent, named, validated product modes today. + +The runtime exposes low-level controls for attention backend, Turbo adapter, upscaling, cache behavior, VAE dtype, and distributed mode. An expert can approximate a conservative path using base 12-step, Sage2, and no cache/upscaler, or a faster path using Turbo-4 and selected acceleration controls. Production fusions and layouts are also selected through startup environment variables. + +These combinations are not immutable buyer-facing `reference_quality` and `maximum_throughput` contracts. "Reference quality" cannot currently mean proven official-BF16 equivalence because that quality comparison remains open. + +### Existing evidence + +- `src/h3_blackwell_runtime/runtime.py` +- `src/h3_blackwell_runtime/attention.py` +- `tools/serve_hot_runtime.py` +- `compose.spark.yml` +- `CURRENT_STATE.md` + +### Remaining gap + +There are no named profiles, frozen configuration contracts, per-profile regression tests, quality floors, or supported-model matrix. + +### Next action + +Define immutable `reference_quality` and `maximum_throughput` profiles per supported model. Validate the reference profile against the official oracle and the throughput profile against explicit latency, reliability, and perceptual-quality thresholds. + +# Recommended Closure Order + +## Immediate evidence work on available hardware + +1. End-to-end p50/p95/p99 resident-request harness. +2. Controlled cold-start and warm-start instrumentation. +3. Full per-step latent capture and assertion fixture. +4. Five-, ten-, twenty-, and thirty-second duration sweep. +5. Maximum safe frame/token push-to-failure matrix. +6. Structured failure logging and fault injection. +7. Fixed 100-prompt, three-seed stability suite. +8. Upscaler artifact benchmark. + +## Highest-leverage engineering work + +1. Pin and implement the Wan and LTX 2.3/2.5 support matrix. +2. Build a shared video/audio quality-scoring pipeline. +3. Add controlled precision, attention, fusion, residency, and distribution ablations. +4. Define named reference-quality and maximum-throughput profiles. +5. Add allocator telemetry and mixed-workload soak support. + +## External or additional-hardware work + +1. Official H3 versus Vortex on the same RTX PRO 6000. +2. Official H3 versus Vortex on the same B200 and GB300. +3. Independent-job versus distributed-job rack-scale testing. +4. Four-, eight-, and higher-GPU scaling. +5. Blind human A/B evaluation. + +# Primary Evidence Index + +- `CURRENT_STATE.md`: canonical implementation and open-work status. +- `README.md`: runtime scope and operating modes. +- `PARITY.md`: historical correctness ledger; interpret contradictory rows cautiously. +- `PERFORMANCE_ROADMAP.md`: profiles, measured opportunities, and projected targets. +- `DISTRIBUTED.md`: distributed implementation, topology, timing, and parity gaps. +- `TURBO.md`: Turbo revisions, schedules, timing, and limited subjective observations. +- `H3_LATENT_UPSCALER.md`: upscaler implementation, timing, and known quality defects. +- `AUDIO_BOUNDARY_INVESTIGATION.md`: audio diagnostics and unresolved quality work. +- `research/EXPERIMENT_REGISTRY.md`: permanent performance-experiment registry. +- `research/REPRODUCIBILITY.md`: environment, command, and reproducibility gaps. +- `benchmarks/`: raw and summarized timing, parity, memory, profiler, and smoke evidence. diff --git a/research/EXPERIMENT_REGISTRY.md b/research/EXPERIMENT_REGISTRY.md new file mode 100644 index 0000000..d482709 --- /dev/null +++ b/research/EXPERIMENT_REGISTRY.md @@ -0,0 +1,120 @@ +# Experiment Registry + +Generated from the repository working tree, documentation, benchmark artifacts, +and git history available on 2026-08-25. The machine-readable registry is +`research/experiment_registry.json`; it is authoritative for record fields and +artifact existence flags. + +Rejected-from-production does not mean deleted. Rejected and blocked work is +retained when it remains useful as evidence, a reference, a reproducer, or a +warning against repeating an unproductive design. + +## Status Taxonomy + +| Status | Meaning | +| --- | --- | +| `production_accepted` | Currently selected production path for its documented eligible environment. | +| `research_retained` | Useful neutral/reference result, without a production-selection claim. | +| `architecture_rejected` | The design conflicts with the required dataflow or merely optimizes a boundary that should be removed. | +| `performance_rejected` | Correct or sufficiently validated candidate that failed its performance gate. | +| `quality_rejected` | Candidate failed parity or quality requirements, regardless of speed. | +| `temporarily_blocked` | Toolchain, build, compiler-resource, host, OOM, or VRAM issue prevents a defensible conclusion. | +| `incomplete` | Evidence is insufficient for any stronger conclusion. | + +Only these seven exact values are allowed. Unknown versions, measurements, +commands, hashes, and timestamps remain `null`, empty, or explicitly listed as +missing rather than inferred. + +## Complete Registry + +| ID | Experiment | Status | Decision | Primary source/evidence | Evidence gaps | +| --- | --- | --- | --- | --- | --- | +| `elem-fused-residual-rmsnorm` | Fused residual + RMSNorm | `production_accepted` | Exact fused elementwise path passed deployment gates. | `src/h3_blackwell_runtime/h3_fusion.py`; `benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json` | Portable non-GB10 validation | +| `nvfp4-quantize-alone` | NVFP4 quantize alone | `research_retained` | Exact packing reference; no standalone production win. | `src/h3_blackwell_runtime/nvfp4_quant.py`; pack-parity benchmark | Standalone canonical deployment gate | +| `nvfp4-quantize-gemm` | NVFP4 quantize + GEMM | `research_retained` | Exact seam/reference retained. | `src/h3_blackwell_runtime/nvfp4.py`; native-fixed benchmark | Accepted isolated performance gate | +| `nvfp4-prequant-swiglu-gemm` | Fused prequantized SiLU x up + quantize + GEMM | `production_accepted` | Exact producer fusion passed block, trajectory, and smoke gates. | `NVFP4_SWIGLU_FUSION_DESIGN.md`; SwiGLU summary | Other architectures | +| `nvfp4-prequant-modulate-gemm` | Fused modulation + quantize + GEMM | `production_accepted` | Exact producer fusion improved blocks 0/24/49 by 0.28-0.79% and the warmed canonical run by 0.56%. | `nvfp4_quant.py`; `NVFP4_MODULATE_FUSION_DESIGN.md`; modulation summary/deployment smoke | Ineligible cases retain materialized fallback | +| `lora-fused-down-activation` | Fused LoRA down + activation | `incomplete` | No isolated conclusion found. | `src/h3_blackwell_runtime/lora.py`; `TURBO.md` | Source, benchmark, parity | +| `lora-fused-up-residual` | Fused LoRA up + residual | `incomplete` | No isolated conclusion found. | `src/h3_blackwell_runtime/lora.py`; `TURBO.md` | Source, benchmark, residual-order parity | +| `lora-fused-full-down-up` | Fused full LoRA down/up | `incomplete` | No complete candidate found. | `src/h3_blackwell_runtime/lora.py`; `TURBO.md` | Implementation, timing, trajectory parity | +| `lora-direct-materialization-gemm-prepass` | Direct LoRA materialization into GEMM prepass memory | `performance_rejected` | Exact Turbo-4 result was slower. | LoRA producer-fusion benchmark; roadmap | Isolated write-cost profile | +| `fc2-fused-activation-residual` | Fused FC2 activation/residual | `incomplete` | Numerical contract exists; integrated candidate does not. | `PERFORMANCE_ROADMAP.md` | Source, randomized parity, block/trajectory gates | +| `qkv-fused-projection-layout` | Fused QKV projection + layout | `architecture_rejected` | Post-GEMM formatting copy superseded by direct views. | Current state; strided-NHD summary | Historical candidate benchmark | +| `qkv-fused-projection-rmsnorm` | Fused QKV + RMSNorm | `incomplete` | No projection-integrated gate found. | Roadmap; `attention.py` | Source, parity, timing | +| `qkv-direct-sageattention-layout` | Direct QKV output into SageAttention layout | `production_accepted` | Strided NHD views remove copies and pass exact gates. | `attention.py`; strided-NHD summary/contract | Distributed equivalent | +| `qkv-to-sage-fused` | QKV-to-Sage fused | `performance_rejected` | Exact P1 entry candidate missed complete-block gate; callable source remains without production integration. | `sage2_entry.py`; entry validator; P1 analysis and NCU report | Trajectory runs intentionally skipped | +| `qknorm-fused` | qknorm fused | `research_retained` | Current exact preparation reference, not an independent win claim. | `attention.py`; NHD component benchmark | Standalone ablation | +| `qknorm-p0-fused-qk-normalization` | Fused P0 Q/K normalization | `incomplete` | Cannot separate this label from later P1 evidence. | Sage2 design | Candidate, distinct benchmark, decision | +| `qkv-fused-rope-output-format` | Fused RoPE + QKV output formatting | `performance_rejected` | Exact within rejected P1 boundary; callable P1 source remains without production dispatch. | `sage2_entry.py`; entry validator; P1 analysis | Standalone ablation | +| `gemm-persistent` | Persistent GEMM | `incomplete` | Distinct launch-fused scheduler remains untested. | Streaming design; roadmap | Scheduler source, block gate, trajectory | +| `gemm-warp-specialized` | Warp-specialized GEMM | `incomplete` | Fixed-tile contract exists; full runtime conclusion absent. | CuTe P1 stream summary | Canonical implementation and benchmark | +| `runtime-event-overhead` | Event-overhead experiment | `research_retained` | Retained profiling methodology. | Profiling tools; roadmap | Dedicated artifact | +| `attention-full-cuda-graph-capture` | Full-attention CUDA graph/capture | `incomplete` | Explicitly deferred/missing. | Current state; roadmap | Implementation, graph safety, timing | +| `cute-conversion-contract` | CuTe NVFP4 conversion contract | `research_retained` | Exact selected-role interoperability established. | Conversion tool/artifacts | Packaged runtime kernel | +| `cute-tiny-tile` | CuTe tiny-tile attempt | `research_retained` | Successful fixed 128-row checkpoint only. | Tile-producer tool/artifact | Canonical M/padding runtime support | +| `cute-temporary-output` | CuTe temporary-output attempt | `architecture_rejected` | Complete temporary materialization violates boundary-removal goal. | Streaming design; roadmap | Historical candidate artifact | +| `cute-failed-builds` | CuTe failed-build attempts | `temporarily_blocked` | Build failures prevent algorithmic conclusion. | Post-optimization summary | Build logs, versions, source mapping | +| `cute-compile-blowups` | CuTe compile-blowup attempts | `temporarily_blocked` | Compiler resource growth is a toolchain blocker. | Streaming design; roadmap | Compiler log, dimensions, versions | +| `cute-oom-vram` | CuTe OOM/VRAM-failure attempts | `temporarily_blocked` | Memory pressure invalidated reliable timing. | Ring full-projection summary | Peak VRAM and failure log | +| `cute-bounded-ring` | CuTe bounded ring-attention/projection attempts | `performance_rejected` | Exact projection path regressed at complete-block gate; opt-in dispatchable source remains disabled by default and absent from production manifests. | `cute_qkv_ring.py`; `nvfp4.py`; capacity/block-gate summaries | Trajectory intentionally skipped | +| `cute-p1-per-output-cta-streaming` | CuTe P1 per-output-N-CTA streaming | `performance_rejected` | Exact, but repeated A production made the schedule much slower. | P1 timing summary; real-tile validator | Candidate kernel source | +| `cute-p2-full-workspace-ring` | CuTe P2 full-workspace ring | `performance_rejected` | Exact full-capacity variant regressed at block gate. | Full-workspace alternating artifact; block-gate summary | Trajectory intentionally skipped | +| `cute-fc2-streaming-policy` | CuTe FC2 streamed producer attempt | `quality_rejected` | Different FC2 reduction policy prevented exactness. | FC2 P0 artifacts; roadmap | Exact alternative reduction implementation | +| `layout-direct-temporal-output` | Direct-to-temporal output | `incomplete` | No retained source or result. | Roadmap; research agenda | Contract, source, benchmark | +| `layout-direct-sage-output` | Direct-to-Sage output | `incomplete` | Historical producer/output-layout concept has no distinct implementation or evidence; it is not the accepted no-copy view contract. | Roadmap; `attention.py` context | Producer-store source, correctness, benchmark | +| `layout-direct-q-padding-32` | Direct output into `q_padding=32` buffers | `architecture_rejected` | Durable padded buffer conflicts with accepted direct-view path. | Sage2 design; roadmap | Historical source/benchmark | +| `layout-dynamic-to-hot-allocation` | Dynamic `.to(...)` hot-path allocation | `architecture_rejected` | Conflicts with allocation-free and capture goals. | Roadmap; `attention.py` | Historical timing | +| `layout-q-coalesced-16b-store` | Coalesced-16-byte Q layout store | `incomplete` | No named ablation found. | CUDA source context | Mapping, alignment proof, benchmark | +| `layout-qkv-all-16b-stores` | All-16-byte Q/K/V stores | `incomplete` | No defensible conclusion survives. | CUDA source context | Source, correctness, timing | +| `layout-xor-swizzle-q` | XOR-swizzle Q variant | `incomplete` | No candidate-specific evidence. | Sage2 design context | Definition, counters, parity | +| `layout-xor-swizzle-qk` | XOR-swizzle Q/K variant | `incomplete` | Distinct variant, evidence absent. | Sage2 design context | Source, benchmark, parity | +| `layout-swizzle-all` | Swizzle-all variants | `incomplete` | Variant enumeration and results absent. | Sage2 design context | Enumeration, source, counters, timing | +| `layout-vectorized-padded-q-copy` | Vectorized-padded-Q copy | `architecture_rejected` | Superseded by no-copy Sage path. | Current state; roadmap | Historical timing | +| `layout-q-copy-wide-store` | Q-copy wide-store variants | `architecture_rejected` | Faster copy still fails boundary-elimination goal. | Current state; roadmap | Variant timings and source | +| `layout-q-copy-block128x2` | Q-copy wide-store `block128x2` | `architecture_rejected` | Distinct geometry, same superseded copy architecture. | Current state; roadmap | Geometry source, timing, parity | +| `layout-grid-kv-writers` | Grid-style KV writers | `incomplete` | No retained artifact. | Sage2 design; profiler tool | Geometry, source, benchmark | +| `layout-vectorized-kv-staging` | Vectorized K/V staging | `incomplete` | Mainloop context exists, candidate does not. | Mainloop analysis | Source, ablation, counters | +| `sage2-online-pv` | Online-PV Sage2 | `incomplete` | No distinct implementation; P3 is separate. | Sage2 design; mainloop analysis | Source, parity, timing | +| `sage-scheduler-v1` | Sage scheduler/register-pressure v1 | `incomplete` | Aggregate P0 evidence cannot map v1. | P0 analysis | v1 source, timing, parity | +| `sage-scheduler-v2` | Sage scheduler/register-pressure v2 | `incomplete` | Aggregate P0 evidence cannot map v2. | P0 analysis | v2 source, timing, counters | +| `sage-scheduler-v3` | Sage scheduler/register-pressure v3 | `incomplete` | Required archival name lacks exact evidence mapping. | P0 analysis | v3 source, benchmark, decision | +| `sage-p0-register-caps` | Sage P0 register-cap variants | `performance_rejected` | Occupancy gain overwhelmed by spills. | Cap latency and NCU artifacts | Candidate source | +| `sage-p0-narrow-scopes` | Sage P0 narrowed scopes | `performance_rejected` | Spill reduction produced no material gain. | Scope latency and NCU artifacts | Candidate source | +| `sage-p0-inplace-scores` | Sage P0 in-place scores | `performance_rejected` | Exact but neutral/slower. | In-place latency artifact | Candidate source | +| `sage-p0-early-k-prefetch` | Sage P0 early-K prefetch | `performance_rejected` | Exact but no material gain. | Early-K latency artifact | Candidate source | +| `sage-p0-softmax-interleave` | Sage P0 independent softmax-chain interleaving | `performance_rejected` | Exact but neutral/slower. | Interleave latency artifact | Candidate source | +| `sage-p0-padded-v-smem` | Sage P0 padded-V shared-memory layout | `performance_rejected` | Targeted wavefront count unchanged. | Padded-V timing and NCU artifacts | Candidate source | +| `backend-flashinfer` | FlashInfer | `incomplete` | No adapter or artifact found. | `attention.py`; backend comparison tool | Adapter, version, benchmark, quality | +| `backend-flashattention-3` | FlashAttention 3 | `incomplete` | Current repository has FA4, not FA3. | `attention.py`; `FLASH4.md` | FA3 source, version, benchmark, quality | +| `backend-spargeattention` | SpargeAttention | `incomplete` | No adapter, benchmark, or quality run found. | Roadmap; comparison tool | Implementation, version, performance, quality | +| `backend-sol` | Sol | `quality_rejected` | Faster settings were not numerically Sage-like. | `attention.py`; optimization profile; roadmap | Full media-quality sweep | +| `sage2-p1-entry-fusion-final` | Finalized Sage2 P1 entry fusion | `performance_rejected` | Exact, but complete-block gate failed; callable wrapper/export and validator remain without production integration. | `sage2_entry.py`; entry validator; P1 analysis/parity/NCU | Trajectory intentionally skipped | +| `sage2-p2-direct-v-preparation-final` | Finalized Sage2 P2 direct V preparation | `performance_rejected` | Exact isolated gain missed absolute go threshold; callable wrapper/export and validator remain without production integration. | `sage2_entry.py`; V-prep validator; P2 analysis/randomized/repeat artifacts | Block/trajectory intentionally skipped | +| `sage2-p3-temporal-pair-final` | Finalized Sage2 P3 temporal pair | `performance_rejected` | Sanitizer-clean/exact but effectively neutral. | Retained patch/runner and P3 artifacts | NCU/block/trajectory intentionally skipped | + +## Interpretation Notes + +- `production_accepted` is scoped to the documented eligible deployment; it is + not a portability claim. +- `research_retained` does not imply runtime dispatch. +- `architecture_rejected` records a design-level reason not to invest further in + the same boundary, even when historical per-variant timing is missing. +- `temporarily_blocked` deliberately avoids converting build or memory failures + into performance conclusions. +- `incomplete` is used for named historical experiments that could not be mapped + to recoverable source and evidence without inventing facts. +- Exact measurements remain in linked artifacts instead of being recopied into + records where doing so could detach them from workload context. + +## Totals + +| Status | Count | +| --- | ---: | +| `production_accepted` | 4 | +| `research_retained` | 6 | +| `architecture_rejected` | 7 | +| `performance_rejected` | 15 | +| `quality_rejected` | 2 | +| `temporarily_blocked` | 3 | +| `incomplete` | 25 | +| **Total** | **62** | diff --git a/research/PRESERVATION_AUDIT.md b/research/PRESERVATION_AUDIT.md new file mode 100644 index 0000000..fa61517 --- /dev/null +++ b/research/PRESERVATION_AUDIT.md @@ -0,0 +1,212 @@ +# Preservation Audit + +Audit date: 2026-08-25. + +## Preservation Scope + +This audit covers the local repository metadata and working tree, the configured +Git remote, the Spark checkout inspected through `ssh spark`, local top-level +benchmark/profiler artifacts plus the packaged SageAttention wheel, Spark's +selected physical research artifacts, and SHA-256 reconciliation of top-level +benchmark JSON filenames. It does not claim to preserve model checkpoints, +mounted parity tensors, generated media, container layers, external package +repositories, or every file under the broader Spark output tree. + +No commit, tag, branch, archive, upload, or copy was created by this audit. The +audit records are this document, `research/REPRODUCIBILITY.md`, and the physical +file inventory in `research/artifact_manifest.json`. + +## Git Topology And Commit Context + +- Current local branch: `fl2va-direct-runtime`. +- Local `HEAD`: `bee998d8403aea18f9636b3773a8e7719ff3817c` (`Record RTX Pro 6000 multi-GPU scaling`, + committed 2026-08-22 16:42:09 +0700). +- `origin/fl2va-direct-runtime` and `origin/master` both resolve to + `bee998d8403aea18f9636b3773a8e7719ff3817c`; `git ls-remote` confirms the same + two server heads. +- Local `master` remains at `ea4f4a2dadbd9754bcadfcfbe9a789f773cf838a`, + which is the merge base with `fl2va-direct-runtime`; the feature branch is 76 + commits ahead and zero behind that local branch. +- `fl2va-direct-runtime` was created from + `8730920634c3f4572353222e4ad307f01862fa16` on 2026-08-19 and then advanced + linearly to `bee998d`. +- The other advertised head is `vae-decode-optimization` at + `8730920634c3f4572353222e4ad307f01862fa16`. +- There are no tags locally or on the configured remote. +- Local reflogs contain the visible branch history but no deleted branch tip. + Local `git fsck --full --no-reflogs --unreachable` finds only the empty blob + `e69de29bb2d1d6434b8b29ae775ad8c2e48c5391`; it finds no useful dangling or + unreachable commit. There is therefore no recoverable deleted branch history + in the audited local object database. +- The Spark checkout is older: branch `fl2va-direct-runtime` and its tracking ref + are at `9f62f6ea838150967422fe648ee25b8b2491356d`. Spark also has a local preservation + branch `preserve/spark-parity-diagnostics-2026-08-13` at + `f1ddc416f732c3de3a9f9de92eec3724d7f65ebc`, based on + `a8df9ef1730b9cf30f9341fd1298214a3b406dce`. It is historical parity context, + not a preservation point for the current optimization tree. + +The exact former P1 dispatch source and some other removed experimental source +cannot be recovered from Git. They were never present in the reachable commit +history, reflogs do not expose a deleted tip containing them, and `fsck` exposes +no useful dangling commit. Current summaries can preserve decisions and measured +results, but they cannot reconstruct those exact source states. + +## Dirty-Worktree Risk + +The local tree has extensive tracked modifications and untracked source, +documentation, tests, tools, benchmark JSONs, and profiler reports relative to +`bee998d`. The post-commit research narrative and implementation are therefore +not protected by the remote heads. A clone of the remote reproduces only the +committed baseline. + +The Spark checkout is independently dirty and differs from both local `HEAD` and +the local working tree. It contains modified runtime source and many untracked +research artifacts while remaining at `9f62f6e`. Neither dirty tree is a backup +of the other. Editing, cleaning, container rebuilds, output rotation, checkout +replacement, disk loss, or an accidental `git clean` can permanently remove +unique evidence. Git cannot restore untracked profiler binaries or source that +was never committed. + +The risk is immediate rather than theoretical: the expected Spark production +container exists, but container state and Spark's mutable output directories +must not be treated as durable preservation. Final validation temporarily +started that container, confirmed `/ready` with Sage2 as both the initial and +current backend, and then stopped it again to leave the research environment +quiescent. + +## Local Inventory + +The local inventory is byte-exact as observed: + +| Class | Location | Count | Bytes | +| --- | --- | ---: | ---: | +| Benchmark JSON | top-level `benchmarks/*.json` | 148 | 1,175,484 | +| Nsight Compute reports | top-level `benchmarks/*.ncu-rep` | 14 | 199,677,223 | +| Nsight Systems reports | top-level `benchmarks/*.nsys-rep` | 2 | 7,686,256 | +| Nsight CSV exports | top-level `benchmarks/*.csv` | 2 | 169,373 | +| SageAttention wheel | `wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl` | 1 | 1,680,352 | +| Selected local physical artifacts | all rows above | 167 | 210,388,688 | + +All 167 selected files have one path, byte size, and SHA-256 record in +`research/artifact_manifest.json`; the manifest totals reproduce this table. + +The two CSV exports and wheel explain why the selected total is larger than the +JSON plus `.ncu-rep` plus `.nsys-rep` subtotal. The unrelated +`benchmarks/rtxpro6000-server-2gpu-topology.txt` and research source files are +outside this physical-artifact total, although they remain within the broader +preservation scope. + +## Spark Inventory + +The audit originally retained a selected Spark aggregate of 279 physical +artifacts totaling 665,950,155 bytes. Within it, the top-level output JSON set +contains exactly 215 files totaling 2,259,709 bytes at +`/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks`. + +The selected 279-file figure is a scoped preservation aggregate, not an +unqualified recursive count of the output directory. Re-enumeration found 280 +current top-level files totaling 547,708,583 bytes, including the exact 215-file +JSON set, while the recursive directory has 291 JSONs because older/nested result +collections are also present. The original 279 paths were not retained, so the +665,950,155-byte aggregate cannot be reproduced or honestly assigned to physical +files. `research/artifact_manifest.json` therefore inventories the complete +reproducible 280-file top-level set and records the discrepancy explicitly: one +additional observed file and 118,241,572 fewer bytes than the retained aggregate. +It must not be presented as a reconstruction of the lost 279-path selection. + +Spark also has the same 1,680,352-byte SageAttention ARM64 wheel in the checkout. +The current top-level output includes large `.ncu-rep`, `.nsys-rep`, `.sqlite`, +`.qdstrm`, CSV, tensor, media, log, and report files. Presence on Spark is not a +second durable copy unless the artifact is also independently stored and +checksum-verified. + +## JSON Reconciliation + +Reconciliation uses basename identity and SHA-256 over local top-level +`benchmarks/*.json` and Spark top-level output `*.json`: + +| Result | Count | +| --- | ---: | +| Identical filename and SHA-256 | 106 | +| Same filename, different SHA-256 | 2 | +| Local only | 40 | +| Spark only | 107 | +| Local total | 148 | +| Spark total | 215 | + +The two hash mismatches are: + +- `gb10-cute-p0-attn-qkv-128rows.json` +- `gb10-fully-fused-one-step-fresh.json` + +The arithmetic reconciles exactly: 106 identical plus 2 mismatches plus 40 +local-only equals 148 local files; 106 plus 2 plus 107 Spark-only equals 215 +Spark files. Same-name mismatches must be preserved as two versions, not resolved +by timestamp or overwritten. Local-only and Spark-only files are unique evidence +until copied into an immutable, checksummed collection. + +## Large Artifact Policy + +Git is appropriate for source, patches, small JSON summaries, exact commands, +manifests, and checksums. Raw `.ncu-rep`, `.nsys-rep`, `.sqlite`, `.qdstrm`, +tensor captures, media, and wheels should not be added casually to ordinary Git +history. They are large, tool/version-sensitive, and can make every clone carry +research payload indefinitely. + +For each large-artifact preservation set: + +1. Freeze an explicit relative-path manifest with byte size and SHA-256 for every + file, including both versions of hash-mismatched JSONs. +2. Record producer revision or working-tree archive hash, exact command and + environment where known, host/GPU facts, profiler version, and linked summary + JSON. +3. Store the payload in immutable or versioned object storage, an archival share, + or a dedicated artifact release. Use Git LFS only if its remote retention and + clone policy are intentionally managed. +4. Keep at least two independently administered copies and verify restoration by + checksum. A mutable Spark output directory and its checkout count as one + failure domain, not two. +5. Do not delete raw profiler databases after exporting CSV until the export and + summary are verified and the raw file is archived. +6. Never replace same-name mismatches in place. Preserve origin-qualified paths + such as `local/` and `spark/` in the archive. + +## Recoverability Conclusions + +- `bee998d` and all reachable committed ancestors are recoverable from the + configured remote. +- The present local optimization source, tests, documents, and many artifacts + are not recoverable from that commit or remote because they are dirty or + untracked. +- Spark provides additional unique evidence but is older, dirty, mutable, and + not an exact working-tree replica. +- The 106 identical JSONs have two observed copies; the 40 local-only and 107 + Spark-only JSONs do not. The two mismatches represent four distinct payloads. +- Raw local profiler evidence is concentrated in 16 reports totaling + 207,363,479 bytes; Git cannot recreate it. +- No tag identifies the research state, no deleted branch tip or useful dangling + commit was found, and the exact former P1 dispatch source plus other removed + source cannot be recovered from Git. +- The preserved P3 patch and launcher are a positive exception: they retain an + exact external source commit, patch, and orchestration path. +- The production image still exists locally on Spark, but the container is + stopped and an image present in one Docker store is not archival recovery. + +## Missing Evidence + +- The original path list and class-by-class composition behind the retained + 279-file, 665,950,155-byte Spark aggregate; the complete current 280-file + top-level inventory does not resolve that historical discrepancy. +- Exact commands and complete environments for most historical benchmarks and + profiler captures. +- A content-addressed archive of each dirty working tree. +- Base-image digest, Python patch version, full installed-package lock, firmware, + clock/power state, and model/checkpoint hashes. +- Checksums and retention status for mounted parity captures, model files, + generated media, build logs, and P3 run worktrees. +- A Git-recoverable copy of the former P1 dispatch and other removed experimental + source. +- A currently passing `/ready` response and a fresh complete test-suite result. + +Until those gaps are closed, the evidence supports the recorded conclusions but +does not provide complete independent replay of every historical experiment. diff --git a/research/REPRODUCIBILITY.md b/research/REPRODUCIBILITY.md new file mode 100644 index 0000000..52527d7 --- /dev/null +++ b/research/REPRODUCIBILITY.md @@ -0,0 +1,191 @@ +# Reproducibility Record + +Audit date: 2026-08-25. + +## Scope And Revision + +The recoverable Git baseline is branch `fl2va-direct-runtime` at +`bee998d8403aea18f9636b3773a8e7719ff3817c`. At audit time both +`origin/fl2va-direct-runtime` and `origin/master` resolve to that commit. The +research results added after that commit depend on a dirty working tree and are +not reproducible from the commit alone. Do not describe the current tree as a +committed revision. + +The local checkout is +`S:\PycharmProjects\AuthorCompanion\tmp\h3-blackwell-runtime`. The Spark +checkout inspected over `ssh spark` is +`/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime`. It is also dirty, is at +the older commit `9f62f6ea838150967422fe648ee25b8b2491356d`, and is not a mirror of +the local working tree. + +## Evidenced Environment + +- Spark hostname: `Inceptal`. +- Spark host architecture: `aarch64`; observed kernel: + `Linux 6.17.0-1026-nvidia #26-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 25 00:57:17 UTC 2026`. +- GPU: `NVIDIA GB10`, compute capability `12.1` (`SM121`). +- NVIDIA driver: `580.173.02`. +- Docker server: `29.2.1`. +- Active research image recorded in the deployment evidence and still present + locally on Spark: + `sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`, + created `2026-08-23T21:05:05.748554319+07:00` for `linux/arm64`. +- The inspected image environment reports CUDA `13.0.2`; benchmark JSONs and + `pyproject.toml` identify Torch `2.9.1+cu130`. +- `Dockerfile.spark` pins Comfy Kitchen `0.2.31`, CUTLASS DSL `4.6.2`, + FlashAttention 4 `4.0.0b27`, Quack Kernels `0.6.4`, and Sol-Attn commit + `930a4d6e432ff8b8ed5e30ff2f72519b92d69bdf`. +- The SageAttention3 ARM64 wheel is + `wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl` (1,680,352 bytes). +- The exact digest of the base image + `ghcr.io/aeon-7/comfyui-aeon-spark:slim`, Python patch version, OS package + set, firmware, clocks, power state, and model/checkpoint hashes were not + recorded. They are unavailable historical inputs unless recovered outside + this repository. + +Do not generalize GB10 facts to the RTX PRO 6000 or B200 runs. Their result JSONs +record device-specific measurements, but this audit did not recover complete +host, image, driver, and command manifests for those historical runs. + +## Artifact Locations + +- Local benchmark JSON and profiler evidence: `benchmarks/`. +- Local isolated P3 source, patch, validator, and exact launcher: + `research/sage2_temporal_pair/`. +- Spark top-level benchmark output: + `/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks`. +- Spark checkout artifacts mounted read-only by Compose: + `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts`. +- Immutable parity locations and their status are catalogued in `PARITY.md`; + notably the FL2VA sampler reference is under + `/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime/artifacts/fl2va-sampler-reference`. +- P3 run work roots and build logs default to + `/home/daniel/aeon-spark-test/h3/sage2-temporal-pair/`. +- Runtime media output is mounted from + `/home/daniel/StoryStudioAssets/H3-output` to `/output`. + +Summary JSONs cite their source artifacts. Examples include +`benchmarks/gb10-fully-fused-fresh-nsight-summary.json`, +`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`, +`benchmarks/gb10-sage2-p1-entry-fusion-analysis.json`, +`benchmarks/gb10-sage2-p2-vprep-analysis.json`, +`benchmarks/gb10-sage2-p3-temporal-pair-analysis.json`, and +`benchmarks/gb10-four-gemm-nvfp4-roofline.json`. + +## Recovered Commands + +The documented Spark service build, startup, and readiness sequence is: + +```bash +docker compose -f compose.spark.yml build h3-hot-runtime +docker compose -f compose.spark.yml up -d h3-hot-runtime +curl http://127.0.0.1:8001/ready +``` + +The documented contract-suite command is: + +```bash +python -m unittest discover -s tests -v +``` + +The exact P3 temporal-pair experiment launcher is preserved: + +```bash +bash research/sage2_temporal_pair/run_spark_experiment.sh +``` + +Its preserved controls are: + +```bash +RUN_SANITIZER=0 bash research/sage2_temporal_pair/run_spark_experiment.sh +RUN_NCU=1 bash research/sage2_temporal_pair/run_spark_experiment.sh +SAGE2_TEMPORAL_RUN_ID=manual-01 bash research/sage2_temporal_pair/run_spark_experiment.sh +MAX_JOBS=4 bash research/sage2_temporal_pair/run_spark_experiment.sh +``` + +`research/sage2_temporal_pair/run_spark_experiment.sh` pins SageAttention commit +`d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5`, applies the preserved patch, builds +isolated baseline and candidate extensions, runs Compute Sanitizer, checks 13 +short cases and the real block-24 SHA, and runs the rotating timing gate. This +is the only audited recent experiment for which the complete orchestration +command and source patch were recovered. + +Exact historical invocation commands for most JSON, Nsight Compute, and Nsight +Systems artifacts were not embedded in the artifacts and could not be recovered +from Git or current documentation. Tool filenames and summary metadata are not +substitutes for the original argv, environment, warmup order, or profiler +options. In particular, do not invent commands for the P0/P1/P2 studies, fused +producer trajectories, fully fused profiles, or four-GEMM roofline capture. + +## Production-Default Checks + +`compose.spark.yml` defines the validated Spark service with Sage2 and these +environment values: + +```text +H3_DISABLE_MMAP=1 +H3_NVFP4_SCALE_BACKEND=vortex +H3_NVFP4_SCALE_VERSION=1 +H3_FUSED_ELEMENTWISE=1 +H3_NVFP4_MODULATE_FUSION=1 +H3_NVFP4_SWIGLU_FUSION=1 +H3_SAGE_QKV_LAYOUT=strided_nhd +H3_SOL_QKV_LAYOUT=native +``` + +The current container metadata independently contains all eight values and the +expected image ID. Check a deployment without changing it with: + +```bash +docker inspect h3-blackwell-runtime-h3-hot-runtime-1 --format '{{.Image}}|{{.State.Status}}|{{json .Config.Env}}' +curl -fsS http://127.0.0.1:8001/ready +``` + +At initial audit time the container was `Exited (137)`. Final validation +temporarily started the same image, confirmed `/ready` with Sage2 as both the +initial and current backend, and produced the configured 320x192, 22-frame +warmup output in 6.293 seconds. The container was stopped again after the check. +`benchmarks/gb10-post-lora-revert-deployment-smoke.json` independently records +the same image, both NVFP4 producer fusions enabled, readiness true, three CUDA +tests passing, and a valid 320x192, 22-frame H.264 smoke output. + +Production must retain the rejected paths as rejected: the bounded-ring QKV +runtime gate regressed, Sage2 P1 entry fusion missed its complete-block gate, +P2 V preparation missed its absolute saving gate, and P3 temporal pairing +missed 220 ms. Their summary artifacts state that production dispatch remained +unchanged or integration was removed/skipped. + +## Tests And Readiness Procedure + +1. Verify the intended revision and record the dirty state with `git rev-parse HEAD`, + `git branch --show-current`, and `git status --short --branch`. +2. Verify the image ID and all production-default environment values with the + non-mutating `docker inspect` command above. +3. Run `python -m unittest discover -s tests -v` in the built environment. Final + validation ran the Spark checkout through image `1d340e14cb6f`; all 41 tests + exposed by that checkout passed in 53.079 seconds. The local Windows checkout + was not run because its Python environment does not contain Torch, and the + independently dirty Spark checkout is not byte-identical to the local tree. +4. Start or recreate the service only when a deployment action is intended, + using the recovered Compose commands above. Wait for startup warmup. +5. Require `curl -fsS http://127.0.0.1:8001/ready` to succeed and inspect the + returned backend, loaded options, and warmup state. `/health` alone is not a + substitute for readiness. +6. Run a real resident generation smoke and validate media dimensions, codec, + frame count, and duration. The exact historical request command for the final + deployment smoke is unavailable; its request fields are preserved in + `benchmarks/gb10-post-lora-revert-deployment-smoke.json`. +7. For a candidate numerical path, follow the applicable validation ladder in + `NVFP4_STREAMING_DESIGN.md`: adversarial/randomized checks, real projections, + complete blocks, all 50 blocks, two-step trajectories, canonical 12-step + checksums, then profiler and peak-memory recapture. Skipped gates must remain + explicitly marked skipped. + +The dependency-free registry validator also passed all 62 records and reported +the expected seven-status distribution. `git diff --check` passed before the +preservation commit. + +Passing historical summaries establish prior evidence only. A fresh run must +record revision or tree hash, full argv, environment, dependency and image +digests, input/checkpoint hashes, warmup order, raw outputs, and checksums to be +independently reproducible. diff --git a/research/artifact_manifest.json b/research/artifact_manifest.json new file mode 100644 index 0000000..b916e25 --- /dev/null +++ b/research/artifact_manifest.json @@ -0,0 +1,3615 @@ +{ + "metadata": { + "algorithm": "SHA-256", + "generated_date": "2026-08-25", + "scope_note": "Local selected scope is the established 167-file set. Spark records are the complete reproducible current top-level benchmark output set; the audit-retained 279-file/665950155-byte aggregate cannot be reconstructed because its path list was not retained.", + "summary": { + "local": { + "record_count": 167, + "size_bytes": 210388688, + "expected_record_count": 167, + "expected_size_bytes": 210388688, + "reconciled": true + }, + "spark": { + "record_count": 280, + "size_bytes": 547708583, + "expected_record_count": 279, + "expected_size_bytes": 665950155, + "reconciled": false, + "record_count_delta": 1, + "size_bytes_delta": -118241572 + }, + "total": { + "record_count": 447, + "size_bytes": 758097271 + } + }, + "json_reconciliation": { + "identical": 106, + "mismatches": 2, + "local_only": 40, + "spark_only": 107, + "local_total": 148, + "spark_total": 215 + } + }, + "artifacts": [ + { + "scope": "local", + "path": "benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json", + "size_bytes": 1770, + "sha256": "bbe9c4393002b6683ba6f02adb72941df18cad04f16ed33c80a869005fd01dbc", + "artifact_class": "benchmark_json", + "location": "benchmarks/audio-dialogue-format-sweep-sage2-seeds440420-440429.json" + }, + { + "scope": "local", + "path": "benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json", + "size_bytes": 2962, + "sha256": "629921184959e911e385b6edef0d21aa6ff4c789f1573c66363295c7cc063604", + "artifact_class": "benchmark_json", + "location": "benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-37810-token-optimization-profile-seed440420.json", + "size_bytes": 2803, + "sha256": "3894f2c2d862155e6b96c103e4c2f3c1d27cb2da26979b25e71fb223e9e97762", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-37810-token-optimization-profile-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-component2-layout-hnd-2step.json", + "size_bytes": 452, + "sha256": "caa39d2e7eb6f016f6a2e27c07612745680a63de26b20fabcc9773c0cb0c8683", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-component2-layout-hnd-2step.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-component2-layout-strided-nhd-12step.json", + "size_bytes": 455, + "sha256": "6c44a7fcabcd83e9cc2f396b6c3e8def7e896ba808b9be70b73948dbed58a10a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-component2-layout-strided-nhd-12step.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-component2-layout-strided-nhd-2step.json", + "size_bytes": 452, + "sha256": "5b508a9c93c5921c331c2a127e29db07acfbe24296102ffde6d7e8a79ee2ff82", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-component2-layout-strided-nhd-2step.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json", + "size_bytes": 3039, + "sha256": "c0d74d5f0d69e77b1afe166409b4e6d47cc5614ef597d01bb9533dfa2f8c713a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-nvfp4-conversion-contract.json", + "size_bytes": 839, + "sha256": "1eb0ccd438862a035ccc6e1adc4a6b63ad99aef485f22a1bb6439017588a16bd", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-nvfp4-conversion-contract.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-nvfp4-tile-producer.json", + "size_bytes": 1682, + "sha256": "d5d3d0b87fbe6443b22d72b30f5ed5c5b86990ef0aaea419562c796cd3ce18e5", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-nvfp4-tile-producer.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json", + "size_bytes": 1697, + "sha256": "fc2f017eefc54c94582b707b170304f65729dc6353b70fb25a3f5a35e397c045", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-attn-out-128rows.json", + "size_bytes": 1686, + "sha256": "874f3eb972e11cd4a447fadddfcfb85cfe033eb1be2341af572d283f3df71df0", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-attn-out-128rows.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json", + "size_bytes": 1727, + "sha256": "757d5558d6fc25ff866eec0e4970af808e1a03d853dddce67513938542f432db", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-attn-qkv-128rows.json", + "size_bytes": 1700, + "sha256": "e56685384bdf814ad0579f1bed38f677fb7b4f33145c149c6ea1a1fd52142f84", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-attn-qkv-128rows.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-h3-summary.json", + "size_bytes": 2350, + "sha256": "8203065f4717de7028ad1db7183c14eb32354247f516436e442fac5e9b6e12b3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-h3-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json", + "size_bytes": 1708, + "sha256": "5f6d8758907ad6e09766d176c172a5556411c1957404170ee5b2c06a1a636181", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-mlp-fc1-128rows.json", + "size_bytes": 1697, + "sha256": "65ba2736b06697983c294bd14effb7406a0d9a68032dfba976ee9ca982acaafc", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-mlp-fc1-128rows.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json", + "size_bytes": 1738, + "sha256": "4949f86c93c85dedf513409c8c23006017eaccfb6ddd21fe67b1bacc80d2cc16", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p0-mlp-fc2-128rows.json", + "size_bytes": 1703, + "sha256": "af9c5b2104cdcad74c43846c2971f122e80617ea0a95ddc01b8587a8a1b2b3b2", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p0-mlp-fc2-128rows.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-real-tiles-attn-out.json", + "size_bytes": 718, + "sha256": "3a43fde087d952a31f608f1636fb182ae62424b0d491d6c3211d66094587d9cc", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-real-tiles-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json", + "size_bytes": 718, + "sha256": "b99b2982b7654fd60d66c6a729ea405bcb5f88201187aa89391179b89ab697a3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json", + "size_bytes": 716, + "sha256": "e1476a8a9185b67951ecc9053e4fb19bad1c26a600b5a2ddce9ee60a58032506", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json", + "size_bytes": 1608, + "sha256": "7b036a92280938251e58dd5e61823481addf5cf903f57f76d7ddfb26c1ff389c", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json", + "size_bytes": 1638, + "sha256": "ea7103f5bb09e9185e7e7768dd0883853eae172e0e555211bdff72d869073c5c", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json", + "size_bytes": 1619, + "sha256": "1af5fda39dad2e4f9e9107e4f092e2b5857669bd8d4648cc5c274eeaf9af8c5b", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-stream-a-summary.json", + "size_bytes": 1637, + "sha256": "4b7de5f2b5c455d1b4ae275ab13559024d64b3d9ab84f6d74fb7e10ecfc0142e", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-stream-a-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-stream-a-timing-summary.json", + "size_bytes": 2152, + "sha256": "c977fdc49a0ff832a7824e67a8275c6b6a591d088ca12514f93f152632adfeef", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-stream-a-timing-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-p0-attn-out.json", + "size_bytes": 1986, + "sha256": "a559fc4f423cb0d9d9b4c2b75ba046c0aa71c232597c9494cea1a857dfd7691f", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-p0-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json", + "size_bytes": 2014, + "sha256": "3cc0a8ae1546c1555f9b5dbba054f30a1f9a29d84b6c116cd1108aa2df34dd04", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json", + "size_bytes": 1997, + "sha256": "d6d0b0572119efd07035f66cbfdcca34b99b3859d56f13b3e9f2e859d035b420", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-stream-attn-out.json", + "size_bytes": 1834, + "sha256": "47537cc58014da98d8e226780fe1ced1592a900893a5c19cddab39d64d824271", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-stream-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json", + "size_bytes": 1863, + "sha256": "f7eb6d53526cabaf05301ac0c88b3432fbec6551c2aeb3763300a4d8c3f924d1", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json", + "size_bytes": 1844, + "sha256": "8cdd4d186a2fa76704850ddd438d1086a896d3d8bb1c612b6133965a0866a59e", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-attn-out.json", + "size_bytes": 1348, + "sha256": "2bcfdfc07d5363e71e54d0bc1e06267b50ba9bb2beab745d70126446ed10d1fe", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-attn-qkv.json", + "size_bytes": 1347, + "sha256": "c2adef4c4ce78da83eb0a363cc127aeaaa7e2dc20168f1896bf19b326f4ca652", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-attn-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json", + "size_bytes": 1908, + "sha256": "388a3b46e7ef959db342e10125a234b382cdb0efb1578cb852f9ef7eae4871c7", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json", + "size_bytes": 1562, + "sha256": "4a28e37c644421f69c26ccb98661019e37416596ac340413af1c421437d2d692", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json", + "size_bytes": 1905, + "sha256": "6b044e03eb39f7d0a2deba9b870a553d64638e1883fe2ac281d3e68690ff64a3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json", + "size_bytes": 1904, + "sha256": "8dd3820ea442f9b78958553e2caea65497462d45678bcb565b428ae833e0fdeb", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json", + "size_bytes": 1910, + "sha256": "5f7ee47eb011cb0e7b3e13b901211732dbdd76c224eff0eded4f540584f091d2", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json", + "size_bytes": 1902, + "sha256": "2c825fa42fe15edf9716d40e3f0a1b616d746172679c0c0110142f5cb6b47e8c", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json", + "size_bytes": 1897, + "sha256": "c4ab658003e53edb4201f1e386ffa1420b6e8c0c8d8b48dfbd374369874d8e9d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-capacity-summary.json", + "size_bytes": 1911, + "sha256": "29f213f8bb3c0dd4d169a6970bc580042935477bd5a265f212e69413267cf807", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-capacity-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json", + "size_bytes": 4250, + "sha256": "a08649ac565a1f0089e9985c947c738f06c6f33809fcb69dea3b44d321928317", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block0-fc1.json", + "size_bytes": 4244, + "sha256": "b969a7a861ff55cfc0b9d002341f88773b76e037acf33cc62d8ffbce4b5859d4", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block0-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block0-qkv.json", + "size_bytes": 4249, + "sha256": "e5da4779b93de4dcc40b4b061d7b116b4183815185615438afad97595d49de76", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block0-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json", + "size_bytes": 6881, + "sha256": "96e478b2ae4955fd5398715727847bb9f95aecceb524b147eebc7d3b69129cfb", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json", + "size_bytes": 2891, + "sha256": "87d3e8edb856d9bb311184b435e30d4d45187ceb1031c13b0484442caf46ab06", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json", + "size_bytes": 4251, + "sha256": "6ad7e66b2e1bdb3e38a42eb6f69b3ed932290f2389f3832680a6de5f757d132b", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block24-fc1.json", + "size_bytes": 4245, + "sha256": "d2234ebfe7e47e64356c7e1db675009e39cd656c978cb299ee046e5d44045d85", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block24-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block24-qkv.json", + "size_bytes": 4247, + "sha256": "901393c679892cb9dce477e3a123e1a19138cde58c3eaa4a9760e1be090b42af", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block24-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json", + "size_bytes": 4253, + "sha256": "cdb28db72235443704a425378935c07b803e92671dedb965219c7f796c1e106d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block49-fc1.json", + "size_bytes": 4249, + "sha256": "80e2f820032a945c51dae253a92498294a614a82d64eaddf2ca088c7094589e6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block49-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-block49-qkv.json", + "size_bytes": 4248, + "sha256": "0f0ee9ac23ab1d667fc19adec74aa44b9b0a6d1e1ad6acb177bf1d1b273b7157", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-block49-qkv.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-full-projection-summary.json", + "size_bytes": 1848, + "sha256": "7cc893597925bf2a3bc76a2ba0b640d10749d8f6e7d6fe1ef0c9e5046b333c1d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-full-projection-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-mlp-fc1.json", + "size_bytes": 1340, + "sha256": "afa27e173688702dfb8c88030c1f51e8c8a7969575c53f24c347d7aa091e149b", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-mlp-fc1.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-p2-ring-summary.json", + "size_bytes": 1633, + "sha256": "71190c8f76267fa7c673d913572294c1ddd37225e9ebae8525706a58fbd61553", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-p2-ring-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block0-alternating.json", + "size_bytes": 600, + "sha256": "23b8e714e820650e5e93071e129dd2710992c505e28108ea27cf13aee0ceebfe", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block0-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block0-baseline.json", + "size_bytes": 27133, + "sha256": "48693ae255fc2593ed7de5ab65e96dfb313016e4d538e2887c483884294f10e9", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block0-baseline.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block0-ring.json", + "size_bytes": 27073, + "sha256": "c5e7304b02d9400f8e923de05e404c3b9bb95972717747ca415a0812b76127d3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block0-ring.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-alternating.json", + "size_bytes": 602, + "sha256": "90533c42630ad8d3d4fa63efa08c03ad753ea7df8b2f04c22b2d8c3bb7b2a82b", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-baseline.json", + "size_bytes": 26908, + "sha256": "9e3d9625b13bf2d6e7dcb5f9ae3939081bb2c4ba26404e0428bdbe0e07c66d7c", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-baseline.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json", + "size_bytes": 601, + "sha256": "6c184b64aaed5b75fa547a1d10b4e65e884e05aef9849c389629c93d811a12c0", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json", + "size_bytes": 601, + "sha256": "5a5c568043c501aca73a89599b61839e21a4d766d0c8e6ce67bdeb2380d06636", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json", + "size_bytes": 597, + "sha256": "ac91711659470d974c4d4135122983b9c4b5d40d22400f3eb315d2bbc67564d6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json", + "size_bytes": 598, + "sha256": "0e2659ee1999027ffb53fa0d7e69ccce05bbb38c4a80df0ba853fd8d7af42be9", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block24-ring.json", + "size_bytes": 27001, + "sha256": "08de80d226099fb74f148daea16e7a4e06a3cc7a3027f790d2666f1760f51f8d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block24-ring.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block49-alternating.json", + "size_bytes": 610, + "sha256": "a54564508cfd85d28404ee10312c14e5a02fda48f290c053710989f273b1857a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block49-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block49-baseline.json", + "size_bytes": 27000, + "sha256": "a536ef25fde0d128d080c94b648773b789ed887229f58873dc77941e0dd0e801", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block49-baseline.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-block49-ring.json", + "size_bytes": 27059, + "sha256": "f6acf4c5e20c1bdcad9ace73c6b61da3ec99c7dd58c7095b84738076118ac77a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-block49-ring.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json", + "size_bytes": 1143, + "sha256": "9b94e4a7438d2c443a84d56f4e70d27ad6fa7ff70203a7574ce9c846dac5dc1a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-runtime-block24-2048.json", + "size_bytes": 658, + "sha256": "3af972d51bd756613756e791e0975d6a7aa62d2c2a358aaf3e9ba48c96be54fb", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-runtime-block24-2048.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-cute-qkv-runtime-block24-full.json", + "size_bytes": 657, + "sha256": "ae1b98ac8d73ba802bf29a8a9d0216ac03e67f3a14a700de5d98042906238f66", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-cute-qkv-runtime-block24-full.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-deployed-modulate-fusion-block24-profile.json", + "size_bytes": 43903, + "sha256": "e0eeb8d6f69b4955d04e56c9a4f0916bb4c719d9be5568985ed42457ef8ec6a2", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-deployed-modulate-fusion-block24-profile.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-four-gemm-nvfp4-roofline.json", + "size_bytes": 3820, + "sha256": "d46c5ba25eb48710da46517329230919fe139463de459124eba9f87b3db13133", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-four-gemm-nvfp4-roofline.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-block24-fresh-capture.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv", + "size_bytes": 85417, + "sha256": "2a87e48b4f750201840f839ac0819e555b310a678f60b54854355856aa67b437", + "artifact_class": "nsight_csv_export", + "location": "benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-l2.csv", + "size_bytes": 83956, + "sha256": "287033296d0a30d6e782e8b99bad7c7e660c6cbde372cefa48190ea0db6845ef", + "artifact_class": "nsight_csv_export", + "location": "benchmarks/gb10-fully-fused-block24-fresh-l2.csv" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh-torch.json", + "size_bytes": 40746, + "sha256": "7c9279e4e58fb4819cbfdb2508efc2cafced56fb92b3ffb2e53824bdd79c28d0", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-block24-fresh-torch.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-fresh.nsys-rep", + "size_bytes": 980672, + "sha256": "e3a6a2520b137a9cd9f201cf462f06285fcb4cd22d781e6f26f4191036a79031", + "artifact_class": "nsight_systems_report", + "location": "benchmarks/gb10-fully-fused-block24-fresh.nsys-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-block24-profile.json", + "size_bytes": 40738, + "sha256": "3a7683f319034dc0505fc59ea8d7ce4045a67a412615658595a0486bbc10af03", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-block24-profile.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-fresh-nsight-summary.json", + "size_bytes": 5220, + "sha256": "d375b88781af035788e10ccafc09b7d1fdd173d3ac47154c5aa6c10df2a90035", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-fresh-nsight-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-one-step-fresh.json", + "size_bytes": 548, + "sha256": "9d8f394a75366fd92fc9dcbe4ef50ffc32fdb726713436a2e03a198d6a56af0a", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fully-fused-one-step-fresh.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep", + "size_bytes": 6705584, + "sha256": "1331704eb45809fd4342d83f6622f670401fb735d6e4afa7ffcd88edda1f27a7", + "artifact_class": "nsight_systems_report", + "location": "benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json", + "size_bytes": 48394, + "sha256": "679664d9b73af80f5e483c1354b967362a7ccb100e02a8f6c4332f9654be07e4", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json", + "size_bytes": 2205, + "sha256": "05120d988113946e0231fc3135dd143af5815137581229ae1a008571dd26905c", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json", + "size_bytes": 348280, + "sha256": "ce7f4f702aae8372ad337439339485b8c21bb3b93799ba4cdc6b2161d15dd4a9", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json", + "size_bytes": 548, + "sha256": "9500408e3d129bd0b31cbd4f5138dcca3dc69778d353cc1c4a892e0fa72fbb83", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json", + "size_bytes": 539, + "sha256": "65e09873ff2d5e1ae7bbcc28c6c33e34f9cd7b0559a0f5bb8b083c48a321b5d1", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json", + "size_bytes": 507, + "sha256": "1bd170b81d0945acaf6e0fd351781e0a6313a59761f0dc4c75c29290280ceb15", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json", + "size_bytes": 501, + "sha256": "aa788e8eb7a3b0a98518bde6f0dacf601f52639dd6d0e9f9a27abd8894199678", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json", + "size_bytes": 630, + "sha256": "04af5cd47f54b0f129eea1edec172f5c0768ca8e93584c0227ed01d2ea3792e6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json", + "size_bytes": 629, + "sha256": "ebfa0bb2624dd2d663754572557722340ff3342eb38ea823ed2e895dac3b6074", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json", + "size_bytes": 637, + "sha256": "28a4c8777df328762c7a3d722e5d6ae795eaa25001b2550903e04601c5ffebdd", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json", + "size_bytes": 1201, + "sha256": "617fb4633db6bb028af77d43d98f47fb7c381b3b9eec31e76a1b99c391cf0af0", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-fusion-summary.json", + "size_bytes": 1500, + "sha256": "b4d5b17de2d23d550baaa2efa5a4864d1e85555f68594964235e6535c8e9762b", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-fusion-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-producer-block0-full.json", + "size_bytes": 280, + "sha256": "16250c2f34405598f3f759316cebcc2e6984c4ef0b21090cb0b18da4f8a5c066", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-producer-block0-full.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json", + "size_bytes": 284, + "sha256": "c2b41ca10de722ea1f3a771c15feeb02cbb35073f394ae443f2d08f4aa160026", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json", + "size_bytes": 331, + "sha256": "d6e332704a6311685a0837ded0b91922ca758ddf67ad828b5d968a8959493f57", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-producer-block24-full.json", + "size_bytes": 285, + "sha256": "ed86d7de50276c4e63a5d6f36ed4f1a06b4f4c7398c601c22020e35b07e2e5cf", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-producer-block24-full.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-modulate-producer-block49-full.json", + "size_bytes": 273, + "sha256": "104ad106180a55ff3784ac64561672efb878a8b6f06379e69502877290f57de4", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-modulate-producer-block49-full.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json", + "size_bytes": 104950, + "sha256": "9d70e6440d8bbde4ede03bacc372e5626692bc3ebd431a630a05136ccfb58d77", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json", + "size_bytes": 16104, + "sha256": "6c69fa62d97827ef95e056b49311b0f6d0f28af7731578e7a03385b84ef257b3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json", + "size_bytes": 6657, + "sha256": "4cabc8d55e3e9fd90b3017d44f7b29c32781ada8444b94241aaeff29708139d6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-streaming-feasibility.json", + "size_bytes": 1596, + "sha256": "6e0ec7eabf79956720ab1056d786456ad85be0bdacba5e4b2da3fc91a01e6ab6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-streaming-feasibility.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json", + "size_bytes": 535, + "sha256": "97d08cef0c2b8a070514bf960d0e824e89c24d59fb7bdaad3d10c3383df5c136", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json", + "size_bytes": 532, + "sha256": "be0efe0f4a6ce2a5ff7add57b7b5b70a94ec17fdb8fe6211b79e15aa75f75407", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json", + "size_bytes": 633, + "sha256": "435c162eedc6774f4c665e4d9d7b83e22294b974af2ed9d70414d7244151393f", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json", + "size_bytes": 631, + "sha256": "da222d124dd1f6f6216ef085e31bf6e9c0e54cb4ff37a168ab0ddc93906b487e", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json", + "size_bytes": 644, + "sha256": "e885e57acd7f2e4b275f3e2a9ac3cd68ea54389dacaae3dfb13dcba54062a411", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json", + "size_bytes": 1202, + "sha256": "e7bcb8d1b1bf8804e94540ea240eda132a1eaae45899ede7403ad6f8a04d5bf3", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-deployment-smoke.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-fusion-summary.json", + "size_bytes": 1456, + "sha256": "6eba47d71fb7a7d62b11c0c2e4b25ad7e9c91a1c3f502953f56f778dced4d630", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-fusion-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-nvfp4-swiglu-producer-block24.json", + "size_bytes": 330, + "sha256": "459c868f3005197d2e900325184372b492c0d2131f981ff2c59352f38807618d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-nvfp4-swiglu-producer-block24.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-post-lora-revert-deployment-smoke.json", + "size_bytes": 1152, + "sha256": "c6e2edcc0d64e71f53687d25f8f3410a8f56e66bce29abb67f4bb09e2e339079", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-post-lora-revert-deployment-smoke.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-post-optimization-block24-profile.json", + "size_bytes": 44244, + "sha256": "07b11875620e09064afcec3487ace4da95f2ca62666dc3c2ea98926a55f3f9c1", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-post-optimization-block24-profile.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-post-optimization-profile-summary.json", + "size_bytes": 1982, + "sha256": "f36005d21a4ca8819e6bc6135c14074eb07334ebf18ed5b057cf3a1d0cdb0103", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-post-optimization-profile-summary.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-post-optimization-sage2-canonical-12step.json", + "size_bytes": 455, + "sha256": "c74934cf0b20135847771b0149a4d8168220faea12730b05d9407cc6a765adae", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-post-optimization-sage2-canonical-12step.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-post-optimization-sage2-nhd-components.json", + "size_bytes": 2579, + "sha256": "5dc70206df7ca045edfdfa1c18ee1ccd5dc2d80724f40e55be97aade6d60fdfe", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-post-optimization-sage2-nhd-components.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json", + "size_bytes": 3133, + "sha256": "2781c3110051f78ebd9c5deae6ca882532f38f30e1776e64f82c5ad1ef49e3b2", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json", + "size_bytes": 6358, + "sha256": "de934e74527121f808bbc3d8baf510c47dea09d8116616aabd2bd70d9bbf5e9f", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json", + "size_bytes": 3045, + "sha256": "6e4553bc01649ce196eb0c197b8b4baebed8a87a28e2d224c213684631849612", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-block24-mainloop-analysis.json", + "size_bytes": 2921, + "sha256": "66a121fa3a75356b002caf371074a4485dd56dae5eaa1cc1dc18e5231a2de523", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-block24-mainloop-analysis.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-block24-mainloop-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-block24-mainloop-capture.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep", + "size_bytes": 6830626, + "sha256": "5c34ba2b1a33f667fb519da0f54ac1dc68a6e2da894c67d1ce9dd02bae09b32a", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-block24-scheduler-baseline.json", + "size_bytes": 8102, + "sha256": "f7ef716f6541fe93b3db296ad9bd5b5aeae74ada457689ceaee6434a705424b6", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-block24-scheduler-baseline.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep", + "size_bytes": 2744326, + "sha256": "562580a9b8bcc4093b2c972ce5474edccd80f5792a3ac2ac5cf28def64a2e2af", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep", + "size_bytes": 3436545, + "sha256": "a2d7c8a25f526a47818fec5cff93dd235b890c217ece5998a25349743247af10", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep", + "size_bytes": 3402676, + "sha256": "fdf770a8a48c363e7cf980a70f768692811e12eeaddbac2c5f5c8ca06296f735", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep", + "size_bytes": 3354037, + "sha256": "54b63418a17450227f99a1881a1ddb44e2d429491a25fbcd122d06e3b5e6ce53", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep", + "size_bytes": 3095411, + "sha256": "8812d7aea56834e2ae40425d55c573166e224fd40a64a85c186e38a34afcf312", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep", + "size_bytes": 2978742, + "sha256": "4b26f8dd3318ec05449b06d2d6be7cf0be78e602ecacfc92998bc4ba0496d884", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep", + "size_bytes": 2860598, + "sha256": "c61863931bb82ea1796c2a240b11c790286f5ee1a2f8109004cf3e633aac07d6", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep", + "size_bytes": 2772103, + "sha256": "4200f4661bcfa722a03f8d28437c5d78e130fae721105518b376aeaca8b0998c", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json", + "size_bytes": 5478, + "sha256": "4765d80b7a49003680f64e3661982cf5fa64a7a386e78f79cabc40530bd09f82", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json", + "size_bytes": 5498, + "sha256": "e23f210303d7c73a13ed185a58914cb924da1c8944ac328f47899912a8ac62e0", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-interleave-softmax-latency.json", + "size_bytes": 5489, + "sha256": "a4613040c132d87447065483e63f2e66cf467a2faeb632f72ba4f010eee10ab2", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-interleave-softmax-latency.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep", + "size_bytes": 2768968, + "sha256": "4a0fd35e60166a4d74906bd68406c848a07051482076cc04bb13752bde28080e", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-pad-v-smem-latency.json", + "size_bytes": 5471, + "sha256": "0ba04433c7985f8fe566f7d787e1cd1fda9bf4159f0be1062d57a21c548929dc", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-pad-v-smem-latency.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep", + "size_bytes": 2646478, + "sha256": "f5c81c03dafef6f55705c7cf8fc366fbcdb79f21676fcc9ac808fbecbfa23174", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json", + "size_bytes": 13821, + "sha256": "e0d8b3b40978dc4de39b7bd882bf5213d4ea4f27ca7f025e3374a07d4c6bb086", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json", + "size_bytes": 5250, + "sha256": "d4584b6d3d8a1034ea353e939d63bdb65ae62367fb1c2efc57798779de718834", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep", + "size_bytes": 2753860, + "sha256": "b39df62abd4cc6c6ca3be0791447cb239af1e66a1c40de06a6a67dcd99598915", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json", + "size_bytes": 5475, + "sha256": "3be011df053f8e22f247b9a33fc9d9e30747aa276edeef5f1587ee5807cfc2a7", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-block-gate.json", + "size_bytes": 7574, + "sha256": "ec3f522ec117fdb86431a1be7f7efc7ea45e0f9a3adc9d16caf866419a682e2e", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p1-block-gate.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep", + "size_bytes": 85127877, + "sha256": "961cba7d92925c4ed740327f17d43bd160be72266ba398307dbd0a463fbecb89", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep", + "size_bytes": 74904976, + "sha256": "12ce74472ba25c4dfda1907bd054b7bb1637ce6f8701826013b6aa9e108b3d8a", + "artifact_class": "nsight_compute_report", + "location": "benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json", + "size_bytes": 2281, + "sha256": "b2e5045f7c39ea2be1f1d04c4044334a57fa77623c1f62ffa9b80fe1aee8d706", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-randomized.json", + "size_bytes": 4207, + "sha256": "99eb067955796bd768da9b0659483d129b860d8f69f03a9f237f3b3b94ca885d", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p1-randomized.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p1-real-parity.json", + "size_bytes": 12023, + "sha256": "378aac3a8754f2f6318c526798239ce58204caa5c186a416d2b71af7ddd6db19", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p1-real-parity.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p2-vprep-analysis.json", + "size_bytes": 1743, + "sha256": "4aba487f54ae72b8468889c3551f825f4f3b8301c97a4061d21158b69f507320", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p2-vprep-analysis.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-p3-temporal-pair-analysis.json", + "size_bytes": 2197, + "sha256": "6cb2cd0606f07ab27dbaa55260c8f37d2d4e01cd526bdacc97a72ccbdeb863a9", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-p3-temporal-pair-analysis.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json", + "size_bytes": 4348, + "sha256": "e3c7b0f56125a358025184ca5c2b188d61999f8c3fdbd82fea41ee987b931f00", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json", + "size_bytes": 5487, + "sha256": "74b69b08873f31e98a0d40c501186e1f5853176c653381a1d4e65ea5d9a28552", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-vprep-canonical-repeat.json", + "size_bytes": 1265, + "sha256": "d88e0b897d233989b7ae644587da924ed91cd47803f201f0768464d520afde26", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-vprep-canonical-repeat.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sage2-vprep-randomized.json", + "size_bytes": 7299, + "sha256": "0c05e04a46c2f36c98bd5d38fad1a7ed15227e5c441a3521a815b73e261e19e4", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sage2-vprep-randomized.json" + }, + { + "scope": "local", + "path": "benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json", + "size_bytes": 4728, + "sha256": "bb9f5f8fd5b67731bc55e563d25bdb1ec45ada591ba2b7f12c4402eb292674dc", + "artifact_class": "benchmark_json", + "location": "benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/ref2va-960x544-124f.json", + "size_bytes": 2957, + "sha256": "5c69426a2adee438244ec0d80c5b39d84f6a8aac5cc154ab124d2801d288831a", + "artifact_class": "benchmark_json", + "location": "benchmarks/ref2va-960x544-124f.json" + }, + { + "scope": "local", + "path": "benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json", + "size_bytes": 3519, + "sha256": "858982b0d602fb6f24ee9c9b0a824c71d9dc40db58444acaf6c7aeef6ed9e232", + "artifact_class": "benchmark_json", + "location": "benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json", + "size_bytes": 1765, + "sha256": "00bb92ed21f7ca347cfa6f7ffed76681b5788bd31a3574fb1bff1654405b1aab", + "artifact_class": "benchmark_json", + "location": "benchmarks/rtxpro6000-server-vs-gb10-sdpa-864x480-141f-base12-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-dialogue-864x480-141f-base12-sdpa-seed440420.json", + "size_bytes": 16063, + "sha256": "69583c2f0f443c6daace2b832ab4dd84f267909d252022fae15441ccfd9c991a", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-dialogue-864x480-141f-base12-sdpa-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-dialogue-quoted-1344x768-124f-base12-sdpa-seed440420.json", + "size_bytes": 691, + "sha256": "ab282762ee11d54e221e4afd9655298189b5e4e8f16b683919469ce2bc744038", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-dialogue-quoted-1344x768-124f-base12-sdpa-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-dialogue-quoted-1344x768-243f-base12-sdpa-seed440420.json", + "size_bytes": 691, + "sha256": "7f9c4555a5b91ee36dd74c258a8ef193a26b64d9640bfa8dd2fbac242389fb76", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-dialogue-quoted-1344x768-243f-base12-sdpa-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json", + "size_bytes": 2071, + "sha256": "045df8b874bf1296d0e4654f3c5d2944f29619dc44bf23a29cbac5d25df0b8fc", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-dialogue-quoted-864x480-141f-base12-sage2-seed440420.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-distributed-smoke-256x256-9f-1step.json", + "size_bytes": 243, + "sha256": "938d0193f2ac6eb119d3d25777715ef1d8da658291f5f07fb2112b590ff9d119", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-distributed-smoke-256x256-9f-1step.json" + }, + { + "scope": "local", + "path": "benchmarks/t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421.json", + "size_bytes": 2012, + "sha256": "cc52e29ad0529a8137fc814833c2cc661c26ecdc92041a8230e2a25bf619e961", + "artifact_class": "benchmark_json", + "location": "benchmarks/t2va-nightclub-music-onset-864x480-141f-base12-sage2-seed440421.json" + }, + { + "scope": "local", + "path": "wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl", + "size_bytes": 1680352, + "sha256": "d145ece529c7201c7ba2ecceb48a79dadf9d87afcfdada370b2a1c6c7285371a", + "artifact_class": "python_wheel", + "location": "wheels/sageattn3-1.0.0-cp312-cp312-linux_aarch64.whl" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-report-100.md", + "size_bytes": 3429, + "sha256": "0adc1c1504ba04aa75546831c4e4602c6ef83fdd7de2d2e2280463fb62678fba", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-report-100.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-segments-100.json", + "size_bytes": 45083, + "sha256": "956abc6fd7f3daf379c14d56ba6928da15fe74f34d7b4ecce8e2d120f345334c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-segments-100.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-smoke.json", + "size_bytes": 5253, + "sha256": "8fa3136e020f6844e6a6ead2c133431182ca7f65264121fcce011908493dede7", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-100.json", + "size_bytes": 2253, + "sha256": "367130b4b09a2685b48cbae9d9ea530f99ba83a05bc3eb0ea39c46a8696f368b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-100.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-smoke.json", + "size_bytes": 2240, + "sha256": "cbed4951c4b75a9ed411ac10b89eaadcfc64aa242713beb38620f3ab6e503be3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-vec-smoke.json", + "size_bytes": 6755, + "sha256": "34ef768c970aeb154102f54b80c7cb2ed6581ff01c9b38b5945409fb7ea50801", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-fused-vec-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-layout-100.json", + "size_bytes": 5246, + "sha256": "0996c5398f67b40cb85720a219d840dd6d14720638eabe829b4cefe13101d763", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-layout-100.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-native-smoke.json", + "size_bytes": 5239, + "sha256": "fbdd69ad4db5c09351d8da11c8400759ff2bc3c6d613413c7f2cc2bf435aff79", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-native-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-smoke.json", + "size_bytes": 5257, + "sha256": "9ba442147680bfba0882f717b4b387be1bfda6bed9d3e99c96a4dc310d9d3283", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-sol-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-whole-100.json", + "size_bytes": 16491, + "sha256": "17f4ea34a3ea6a461d29119e34ecd80510d5fd983a108b69f2281c587a280bc0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-whole-100.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-whole-.json", + "size_bytes": 16491, + "sha256": "17f4ea34a3ea6a461d29119e34ecd80510d5fd983a108b69f2281c587a280bc0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-path-whole-.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-sage2-cudnn-ck-960x544x124.json", + "size_bytes": 28596, + "sha256": "1d600fbbfd3db4e026d07ce06c23be663ad126ca945a486d19eb1d796d6f4631", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-sage2-cudnn-ck-960x544x124.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-sol-layout-report-100.md", + "size_bytes": 3612, + "sha256": "2a3f11fd09ee32539a6a14b92d56238940169fe38a3cc4924d177624f1473df7", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/attention-sol-layout-report-100.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/benchmark-report-3d0c093.json", + "size_bytes": 86211, + "sha256": "a5c384051f27586034535e46747d8f1c4b3a5f5c432644924768353bebaafb18", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/benchmark-report-3d0c093.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/benchmark-report-3d0c093.md", + "size_bytes": 9011, + "sha256": "1ccab7c895af0c6a505ad3e7b422f5fe90473f4e747144ca0a6bf6945a806860", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/benchmark-report-3d0c093.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407-docker.log", + "size_bytes": 27580, + "sha256": "8e1cb313e78e94787d5a614bfa0c2319b7538d09f53220506068f393841cd4dd", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407-docker.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407.json", + "size_bytes": 1243, + "sha256": "746fb7f62326d3e428e8b9648a666837845adc464328ec2321c1be962fd357bd", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407.log", + "size_bytes": 1244, + "sha256": "d9933ec81ef478600670c1980addababfa51a3cdd9e9aa53785e5ca67c04b58d", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/comfy-cat-benchmark-960x544-124f-seed440407.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-53cd8bd.log", + "size_bytes": 4070, + "sha256": "63c9fbf17c0b1593ab2dc1e5913587428eb8a14da4c9802c679329df448e0fb4", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-53cd8bd.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-fp16-vae.log", + "size_bytes": 3356, + "sha256": "da9bf229a9f4fa1265b27b3ae458efdbfce821ac274ea4e12a208b3370ff6004", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-fp16-vae.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-sage3-fp16-vae.log", + "size_bytes": 3363, + "sha256": "3c395e7e113bf00eb3a98e17e46c304eeaf3e1e258874acb9d937c5ab35494e4", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-sage3-fp16-vae.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-sol_attn-fp16-vae.json", + "size_bytes": 1410, + "sha256": "8a596c1605d67a879d5f3aaf34676d27808d9b3403a5217e8e2818544d5f596f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/direct-cat-benchmark-960x544-124f-seed440407-sol_attn-fp16-vae.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-attention-components-block24-37810.json", + "size_bytes": 11135, + "sha256": "74adfe3d42676acdaf9f7fd8bc072f08efd89ee546fd45733400033c2f8cf041", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-attention-components-block24-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-attention-path-block24-1344x768-124f.json", + "size_bytes": 16495, + "sha256": "f0baf7f649027c6ba6455af22c61fa67f686baca75b31a98f3d6838bae6444ff", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-attention-path-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block0-37810-production-comfy.json", + "size_bytes": 19935, + "sha256": "7de25ca6e5895e9fb5b699b5c460af4e8aed21c9a282ae7cc1c0d7455ef38380", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block0-37810-production-comfy.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block0-37810-production-vortex.json", + "size_bytes": 19967, + "sha256": "a51853a1a50a199fb2d95c36b7d0406fea2a03d49843939ebb0e0ae8b58d32e3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block0-37810-production-vortex.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-37810-production-comfy.json", + "size_bytes": 19985, + "sha256": "8d8afe2f9acc076f0403dda95301a662c7d89d86a22728147b5c5c99087a4c4d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-37810-production-comfy.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-37810-production-vortex.json", + "size_bytes": 20019, + "sha256": "065f87e9896429023d2624b81b8e42318a9b7c62a491a9cfc43fcdfe6a12b28e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-37810-production-vortex.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-components-sol-bf16-37810.json", + "size_bytes": 20034, + "sha256": "c8fdd14843e8e030f7cfe511a87c0794a9bcee18140a51e81c977688f9207e9a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-components-sol-bf16-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-components-sol-int8qk-37810.json", + "size_bytes": 20035, + "sha256": "080e7d33fef220de5e15a30611447ff08f53056032b00c3852e7e4f17b81bb2a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block24-components-sol-int8qk-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block49-37810-production-comfy.json", + "size_bytes": 19957, + "sha256": "87582c22ef53fff629cb983bf8b8c8fb7522bed8c9b23d81a22328ff597288cd", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block49-37810-production-comfy.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block49-37810-production-vortex.json", + "size_bytes": 19953, + "sha256": "009bc8bc8b35be42dad707ab5f89a4fb2f268f584219f742b8ad1758f05e8675", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-block49-37810-production-vortex.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-hnd-2step.json", + "size_bytes": 452, + "sha256": "caa39d2e7eb6f016f6a2e27c07612745680a63de26b20fabcc9773c0cb0c8683", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-hnd-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-strided-nhd-12step.json", + "size_bytes": 455, + "sha256": "6c44a7fcabcd83e9cc2f396b6c3e8def7e896ba808b9be70b73948dbed58a10a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-strided-nhd-12step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-strided-nhd-2step.json", + "size_bytes": 452, + "sha256": "5b508a9c93c5921c331c2a127e29db07acfbe24296102ffde6d7e8a79ee2ff82", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-component2-layout-strided-nhd-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-conversion-contract.json", + "size_bytes": 839, + "sha256": "1eb0ccd438862a035ccc6e1adc4a6b63ad99aef485f22a1bb6439017588a16bd", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-conversion-contract.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-tile-producer.json", + "size_bytes": 1682, + "sha256": "d5d3d0b87fbe6443b22d72b30f5ed5c5b86990ef0aaea419562c796cd3ce18e5", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-tile-producer.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json", + "size_bytes": 1697, + "sha256": "fc2f017eefc54c94582b707b170304f65729dc6353b70fb25a3f5a35e397c045", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-out-128rows-reencode.json", + "size_bytes": 1686, + "sha256": "874f3eb972e11cd4a447fadddfcfb85cfe033eb1be2341af572d283f3df71df0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-out-128rows-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json", + "size_bytes": 1727, + "sha256": "757d5558d6fc25ff866eec0e4970af808e1a03d853dddce67513938542f432db", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-direct.json", + "size_bytes": 1523, + "sha256": "7b43b5b310a3fc8804b65ad8881718ba843fb0682d637066251038d7335d0e8c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-direct.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows.json", + "size_bytes": 1410, + "sha256": "f211f62d21c5b61853523ae33deaeadf773dd42c652d4b968caa300aa0f2da9e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-reencode.json", + "size_bytes": 1700, + "sha256": "e56685384bdf814ad0579f1bed38f677fb7b4f33145c149c6ea1a1fd52142f84", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-swap.json", + "size_bytes": 985, + "sha256": "f70c7c67ada3c42dda103f269baa4e9a5b8eb7d01ee761329eba2b4935dc1e9d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-swap.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-zero-a.json", + "size_bytes": 1453, + "sha256": "8177ce215f2bc751e8a683aeb93fc87477639fe4feae1f1629ce0f5ac3a76347", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-zero-a.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-zero-a-reencode.json", + "size_bytes": 1475, + "sha256": "0e4474740629ca474f35200576091b0c86786cdade102d92008be3e627b214ac", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-attn-qkv-128rows-zero-a-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json", + "size_bytes": 1708, + "sha256": "5f6d8758907ad6e09766d176c172a5556411c1957404170ee5b2c06a1a636181", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc1-128rows-reencode.json", + "size_bytes": 1697, + "sha256": "65ba2736b06697983c294bd14effb7406a0d9a68032dfba976ee9ca982acaafc", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc1-128rows-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json", + "size_bytes": 1738, + "sha256": "4949f86c93c85dedf513409c8c23006017eaccfb6ddd21fe67b1bacc80d2cc16", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-k256-reencode.json", + "size_bytes": 1703, + "sha256": "af9c5b2104cdcad74c43846c2971f122e80617ea0a95ddc01b8587a8a1b2b3b2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-k256-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-reencode.json", + "size_bytes": 1703, + "sha256": "342b4cc46c2afb8eb37c83f446d3420196ebc5212dbc4b0cb74f6b12f6245dc6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p0-mlp-fc2-128rows-reencode.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-attn-out.json", + "size_bytes": 718, + "sha256": "3a43fde087d952a31f608f1636fb182ae62424b0d491d6c3211d66094587d9cc", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json", + "size_bytes": 718, + "sha256": "b99b2982b7654fd60d66c6a729ea405bcb5f88201187aa89391179b89ab697a3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json", + "size_bytes": 716, + "sha256": "e1476a8a9185b67951ecc9053e4fb19bad1c26a600b5a2ddce9ee60a58032506", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json", + "size_bytes": 1608, + "sha256": "7b036a92280938251e58dd5e61823481addf5cf903f57f76d7ddfb26c1ff389c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json", + "size_bytes": 1638, + "sha256": "ea7103f5bb09e9185e7e7768dd0883853eae172e0e555211bdff72d869073c5c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json", + "size_bytes": 1619, + "sha256": "1af5fda39dad2e4f9e9107e4f092e2b5857669bd8d4648cc5c274eeaf9af8c5b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-attn-out.json", + "size_bytes": 1986, + "sha256": "a559fc4f423cb0d9d9b4c2b75ba046c0aa71c232597c9494cea1a857dfd7691f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json", + "size_bytes": 2014, + "sha256": "3cc0a8ae1546c1555f9b5dbba054f30a1f9a29d84b6c116cd1108aa2df34dd04", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json", + "size_bytes": 1997, + "sha256": "d6d0b0572119efd07035f66cbfdcca34b99b3859d56f13b3e9f2e859d035b420", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-attn-out.json", + "size_bytes": 1834, + "sha256": "47537cc58014da98d8e226780fe1ced1592a900893a5c19cddab39d64d824271", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json", + "size_bytes": 1863, + "sha256": "f7eb6d53526cabaf05301ac0c88b3432fbec6551c2aeb3763300a4d8c3f924d1", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json", + "size_bytes": 1844, + "sha256": "8cdd4d186a2fa76704850ddd438d1086a896d3d8bb1c612b6133965a0866a59e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-attn-out.json", + "size_bytes": 1348, + "sha256": "2bcfdfc07d5363e71e54d0bc1e06267b50ba9bb2beab745d70126446ed10d1fe", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-attn-qkv.json", + "size_bytes": 1347, + "sha256": "c2adef4c4ce78da83eb0a363cc127aeaaa7e2dc20168f1896bf19b326f4ca652", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-attn-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json", + "size_bytes": 1908, + "sha256": "388a3b46e7ef959db342e10125a234b382cdb0efb1578cb852f9ef7eae4871c7", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json", + "size_bytes": 1562, + "sha256": "4a28e37c644421f69c26ccb98661019e37416596ac340413af1c421437d2d692", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json", + "size_bytes": 1905, + "sha256": "6b044e03eb39f7d0a2deba9b870a553d64638e1883fe2ac281d3e68690ff64a3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json", + "size_bytes": 1904, + "sha256": "8dd3820ea442f9b78958553e2caea65497462d45678bcb565b428ae833e0fdeb", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json", + "size_bytes": 1910, + "sha256": "5f7ee47eb011cb0e7b3e13b901211732dbdd76c224eff0eded4f540584f091d2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json", + "size_bytes": 1902, + "sha256": "2c825fa42fe15edf9716d40e3f0a1b616d746172679c0c0110142f5cb6b47e8c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json", + "size_bytes": 1897, + "sha256": "c4ab658003e53edb4201f1e386ffa1420b6e8c0c8d8b48dfbd374369874d8e9d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json", + "size_bytes": 4250, + "sha256": "a08649ac565a1f0089e9985c947c738f06c6f33809fcb69dea3b44d321928317", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-fc1.json", + "size_bytes": 4244, + "sha256": "b969a7a861ff55cfc0b9d002341f88773b76e037acf33cc62d8ffbce4b5859d4", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-qkv.json", + "size_bytes": 4249, + "sha256": "e5da4779b93de4dcc40b4b061d7b116b4183815185615438afad97595d49de76", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block0-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json", + "size_bytes": 6881, + "sha256": "96e478b2ae4955fd5398715727847bb9f95aecceb524b147eebc7d3b69129cfb", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json", + "size_bytes": 2891, + "sha256": "87d3e8edb856d9bb311184b435e30d4d45187ceb1031c13b0484442caf46ab06", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json", + "size_bytes": 4251, + "sha256": "6ad7e66b2e1bdb3e38a42eb6f69b3ed932290f2389f3832680a6de5f757d132b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-fc1.json", + "size_bytes": 4245, + "sha256": "d2234ebfe7e47e64356c7e1db675009e39cd656c978cb299ee046e5d44045d85", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-qkv.json", + "size_bytes": 4247, + "sha256": "901393c679892cb9dce477e3a123e1a19138cde58c3eaa4a9760e1be090b42af", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block24-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json", + "size_bytes": 4253, + "sha256": "cdb28db72235443704a425378935c07b803e92671dedb965219c7f796c1e106d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-fc1.json", + "size_bytes": 4249, + "sha256": "80e2f820032a945c51dae253a92498294a614a82d64eaddf2ca088c7094589e6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-qkv.json", + "size_bytes": 4248, + "sha256": "0f0ee9ac23ab1d667fc19adec74aa44b9b0a6d1e1ad6acb177bf1d1b273b7157", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-full-block49-qkv.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-mlp-fc1.json", + "size_bytes": 1340, + "sha256": "afa27e173688702dfb8c88030c1f51e8c8a7969575c53f24c347d7aa091e149b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-p2-ring-mlp-fc1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-alternating.json", + "size_bytes": 600, + "sha256": "23b8e714e820650e5e93071e129dd2710992c505e28108ea27cf13aee0ceebfe", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-baseline.json", + "size_bytes": 27133, + "sha256": "48693ae255fc2593ed7de5ab65e96dfb313016e4d538e2887c483884294f10e9", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-baseline.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-ring.json", + "size_bytes": 27073, + "sha256": "c5e7304b02d9400f8e923de05e404c3b9bb95972717747ca415a0812b76127d3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block0-ring.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-alternating.json", + "size_bytes": 602, + "sha256": "90533c42630ad8d3d4fa63efa08c03ad753ea7df8b2f04c22b2d8c3bb7b2a82b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-baseline.json", + "size_bytes": 26908, + "sha256": "9e3d9625b13bf2d6e7dcb5f9ae3939081bb2c4ba26404e0428bdbe0e07c66d7c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-baseline.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json", + "size_bytes": 601, + "sha256": "6c184b64aaed5b75fa547a1d10b4e65e884e05aef9849c389629c93d811a12c0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json", + "size_bytes": 601, + "sha256": "5a5c568043c501aca73a89599b61839e21a4d766d0c8e6ce67bdeb2380d06636", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json", + "size_bytes": 597, + "sha256": "ac91711659470d974c4d4135122983b9c4b5d40d22400f3eb315d2bbc67564d6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json", + "size_bytes": 598, + "sha256": "0e2659ee1999027ffb53fa0d7e69ccce05bbb38c4a80df0ba853fd8d7af42be9", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-ring.json", + "size_bytes": 27001, + "sha256": "08de80d226099fb74f148daea16e7a4e06a3cc7a3027f790d2666f1760f51f8d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block24-ring.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-alternating.json", + "size_bytes": 610, + "sha256": "a54564508cfd85d28404ee10312c14e5a02fda48f290c053710989f273b1857a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-baseline.json", + "size_bytes": 27000, + "sha256": "a536ef25fde0d128d080c94b648773b789ed887229f58873dc77941e0dd0e801", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-baseline.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-ring.json", + "size_bytes": 27059, + "sha256": "f6acf4c5e20c1bdcad9ace73c6b61da3ec99c7dd58c7095b84738076118ac77a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-block49-ring.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-runtime-block24-2048.json", + "size_bytes": 658, + "sha256": "3af972d51bd756613756e791e0975d6a7aa62d2c2a358aaf3e9ba48c96be54fb", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-runtime-block24-2048.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-runtime-block24-full.json", + "size_bytes": 657, + "sha256": "ae1b98ac8d73ba802bf29a8a9d0216ac03e67f3a14a700de5d98042906238f66", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-cute-qkv-runtime-block24-full.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-deployed-modulate-fusion-block24-profile.json", + "size_bytes": 43903, + "sha256": "e0eeb8d6f69b4955d04e56c9a4f0916bb4c719d9be5568985ed42457ef8ec6a2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-deployed-modulate-fusion-block24-profile.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2.csv", + "size_bytes": 83956, + "sha256": "287033296d0a30d6e782e8b99bad7c7e660c6cbde372cefa48190ea0db6845ef", + "artifact_class": "nsight_csv_export", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2.csv" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses.csv", + "size_bytes": 85338, + "sha256": "15a78d07d1f295d794ddf3121e05f98371a15493e4fde87940fa3ec78eb25131", + "artifact_class": "nsight_csv_export", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses.csv" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses.ncu-rep", + "size_bytes": 43250988, + "sha256": "a9bbf1a316d0ddc7e940580ff309601f1671c71b1921592d09c9820cd0984e90", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv", + "size_bytes": 85417, + "sha256": "2a87e48b4f750201840f839ac0819e555b310a678f60b54854355856aa67b437", + "artifact_class": "nsight_csv_export", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.ncu-rep", + "size_bytes": 43249389, + "sha256": "13cad831c2e5869e82d6cebf32be55e150d9e4bf87aaa3945d775b6768f2042b", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2.ncu-rep", + "size_bytes": 43241840, + "sha256": "151a3e1d7a05dfbffcd11727b8f960c26ba69caf79ff2e80eb69856211e44a07", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-l2.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json", + "size_bytes": 487, + "sha256": "49f907042c9d13103aa63c3be649c6d8fa9984d0e999eae8e4d7b61997f12036", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh.nsys-rep", + "size_bytes": 980672, + "sha256": "e3a6a2520b137a9cd9f201cf462f06285fcb4cd22d781e6f26f4191036a79031", + "artifact_class": "nsight_systems_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh.nsys-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh.sqlite", + "size_bytes": 6324224, + "sha256": "a8560df03998fa684b493857e9a06d49b1d158dd7af196a7cbda3a64d78a6bf0", + "artifact_class": "nsight_sqlite_database", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh.sqlite" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-torch.json", + "size_bytes": 40746, + "sha256": "7c9279e4e58fb4819cbfdb2508efc2cafced56fb92b3ffb2e53824bdd79c28d0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-fresh-torch.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-profile.json", + "size_bytes": 40738, + "sha256": "3a7683f319034dc0505fc59ea8d7ce4045a67a412615658595a0486bbc10af03", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-block24-profile.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.json", + "size_bytes": 553, + "sha256": "99faf56e72130d9d818dbdba96e1fa34bc2ed0a6ffac1cb00a3e8bd93c6989c1", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep", + "size_bytes": 6705584, + "sha256": "1331704eb45809fd4342d83f6622f670401fb735d6e4afa7ffcd88edda1f27a7", + "artifact_class": "nsight_systems_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.sqlite", + "size_bytes": 163209216, + "sha256": "3e90a273485f0f10d236444fb570682d8a6e609a318322864a1f56140e812463", + "artifact_class": "nsight_sqlite_database", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fully-fused-one-step-fresh.sqlite" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-component2-capture.json", + "size_bytes": 352, + "sha256": "39acf0c5f77c88f1d71e16e682011af401ee73bea289481dd1041c2938609574", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-component2-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-component2.qdstrm", + "size_bytes": 2203448, + "sha256": "e549d8c993be3d7725dc5c586bfc014127d149c78260f1eee5b3167991fbc696", + "artifact_class": "nsight_qdstrm_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-component2.qdstrm" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json", + "size_bytes": 48394, + "sha256": "679664d9b73af80f5e483c1354b967362a7ccb100e02a8f6c4332f9654be07e4", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-h3-block24-1344x768-124f-sage2.json", + "size_bytes": 19696, + "sha256": "933a02981f11dadc7c4aa1f2d248d1e2179bbcc07021dc1cb88e16e9432dd732", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-h3-block24-1344x768-124f-sage2.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-h3-fusion-blocks-0-24-49-37810.json", + "size_bytes": 3069, + "sha256": "0c0f0f7d0a8d4f69ce4fa6a05b726ca2ae51b19df7eeb6ee4c0f000f025d5400", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-h3-fusion-blocks-0-24-49-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-block24-37810.json", + "size_bytes": 16006, + "sha256": "f2e61f7afc529305ef2d49d5492dec4854f261813e97c669ffb41f091bf82864", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-block24-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-dense-boundary-block24-37810.json", + "size_bytes": 16000, + "sha256": "21c8ab23bef176e98547bb42a1ba5c3b60ffc384feb8aa67601c497755d73813", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-dense-boundary-block24-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-int8qk-block24-37810.json", + "size_bytes": 10006, + "sha256": "3bea107fd7b2e52ede696d1646c798770edb9eb98a59b6ce6072c83f455ad624", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-hybrid-sage-sol-int8qk-block24-37810.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json", + "size_bytes": 348280, + "sha256": "ce7f4f702aae8372ad337439339485b8c21bb3b93799ba4cdc6b2161d15dd4a9", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-linears-block24-1344x768-124f.json", + "size_bytes": 106956, + "sha256": "3f019ea913295a587256913f5d2dcbab78a601edafc881901c7520bcf2283244", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-linears-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json", + "size_bytes": 548, + "sha256": "9500408e3d129bd0b31cbd4f5138dcca3dc69778d353cc1c4a892e0fa72fbb83", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json", + "size_bytes": 539, + "sha256": "65e09873ff2d5e1ae7bbcc28c6c33e34f9cd7b0559a0f5bb8b083c48a321b5d1", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json", + "size_bytes": 507, + "sha256": "1bd170b81d0945acaf6e0fd351781e0a6313a59761f0dc4c75c29290280ceb15", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-12step-warm.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-2step.json", + "size_bytes": 501, + "sha256": "cebcd549e9255c239971baaa5b37858560185f43fd1af8c083c8aa6d0b9d68d5", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json", + "size_bytes": 501, + "sha256": "aa788e8eb7a3b0a98518bde6f0dacf601f52639dd6d0e9f9a27abd8894199678", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-2step-warm.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json", + "size_bytes": 630, + "sha256": "04af5cd47f54b0f129eea1edec172f5c0768ca8e93584c0227ed01d2ea3792e6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block0-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json", + "size_bytes": 629, + "sha256": "ebfa0bb2624dd2d663754572557722340ff3342eb38ea823ed2e895dac3b6074", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block24-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json", + "size_bytes": 637, + "sha256": "28a4c8777df328762c7a3d722e5d6ae795eaa25001b2550903e04601c5ffebdd", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-fusion-block49-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block0-full.json", + "size_bytes": 280, + "sha256": "16250c2f34405598f3f759316cebcc2e6984c4ef0b21090cb0b18da4f8a5c066", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block0-full.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json", + "size_bytes": 284, + "sha256": "c2b41ca10de722ea1f3a771c15feeb02cbb35073f394ae443f2d08f4aa160026", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-2048.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-full.json", + "size_bytes": 285, + "sha256": "ed86d7de50276c4e63a5d6f36ed4f1a06b4f4c7398c601c22020e35b07e2e5cf", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-full.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json", + "size_bytes": 331, + "sha256": "d6e332704a6311685a0837ded0b91922ca758ddf67ad828b5d968a8959493f57", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block24-full-timing.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block49-full.json", + "size_bytes": 273, + "sha256": "104ad106180a55ff3784ac64561672efb878a8b6f06379e69502877290f57de4", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-modulate-producer-block49-full.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json", + "size_bytes": 104950, + "sha256": "9d70e6440d8bbde4ede03bacc372e5626692bc3ebd431a630a05136ccfb58d77", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json", + "size_bytes": 16104, + "sha256": "6c69fa62d97827ef95e056b49311b0f6d0f28af7731578e7a03385b84ef257b3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-pack-parity-component2-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json", + "size_bytes": 6657, + "sha256": "4cabc8d55e3e9fd90b3017d44f7b29c32781ada8444b94241aaeff29708139d6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json", + "size_bytes": 535, + "sha256": "97d08cef0c2b8a070514bf960d0e824e89c24d59fb7bdaad3d10c3383df5c136", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-12step-warm.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json", + "size_bytes": 532, + "sha256": "be0efe0f4a6ce2a5ff7add57b7b5b70a94ec17fdb8fe6211b79e15aa75f75407", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-2step-warm.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json", + "size_bytes": 633, + "sha256": "435c162eedc6774f4c665e4d9d7b83e22294b974af2ed9d70414d7244151393f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block0-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json", + "size_bytes": 631, + "sha256": "da222d124dd1f6f6216ef085e31bf6e9c0e54cb4ff37a168ab0ddc93906b487e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block24-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json", + "size_bytes": 644, + "sha256": "e885e57acd7f2e4b275f3e2a9ac3cd68ea54389dacaae3dfb13dcba54062a411", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-fusion-block49-alternating.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-producer-block24.json", + "size_bytes": 330, + "sha256": "459c868f3005197d2e900325184372b492c0d2131f981ff2c59352f38807618d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-nvfp4-swiglu-producer-block24.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-block24-profile.json", + "size_bytes": 44244, + "sha256": "07b11875620e09064afcec3487ace4da95f2ca62666dc3c2ea98926a55f3f9c1", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-block24-profile.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-sage2-canonical-12step.json", + "size_bytes": 455, + "sha256": "c74934cf0b20135847771b0149a4d8168220faea12730b05d9407cc6a765adae", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-sage2-canonical-12step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-sage2-nhd-components.json", + "size_bytes": 2579, + "sha256": "5dc70206df7ca045edfdfa1c18ee1ccd5dc2d80724f40e55be97aade6d60fdfe", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-post-optimization-sage2-nhd-components.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-mainloop-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-mainloop-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep", + "size_bytes": 6830626, + "sha256": "5c34ba2b1a33f667fb519da0f54ac1dc68a6e2da894c67d1ce9dd02bae09b32a", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-mainloop-full.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-scheduler-baseline.json", + "size_bytes": 8102, + "sha256": "f7ef716f6541fe93b3db296ad9bd5b5aeae74ada457689ceaee6434a705424b6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-block24-scheduler-baseline.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-hnd-vs-nhd-block24-1344x768-124f.json", + "size_bytes": 5183, + "sha256": "e124294a7ba16b4e8a7f8b2c09e10beb738aba983366debde60189ce336e2912", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-hnd-vs-nhd-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-kernel-variants-block24-1344x768-124f.json", + "size_bytes": 11997, + "sha256": "85784569d79d6bcd00323146039137a2af6fcf7ceae0c1d50b6805c34a9481df", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-kernel-variants-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-baseline-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-baseline-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep", + "size_bytes": 2744326, + "sha256": "562580a9b8bcc4093b2c972ce5474edccd80f5792a3ac2ac5cf28def64a2e2af", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-baseline-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap168-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap168-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep", + "size_bytes": 3436545, + "sha256": "a2d7c8a25f526a47818fec5cff93dd235b890c217ece5998a25349743247af10", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap170-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap170-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep", + "size_bytes": 3402676, + "sha256": "fdf770a8a48c363e7cf980a70f768692811e12eeaddbac2c5f5c8ca06296f735", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap170-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap176-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap176-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep", + "size_bytes": 3354037, + "sha256": "54b63418a17450227f99a1881a1ddb44e2d429491a25fbcd122d06e3b5e6ce53", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap176-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap192-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap192-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep", + "size_bytes": 3095411, + "sha256": "8812d7aea56834e2ae40425d55c573166e224fd40a64a85c186e38a34afcf312", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap192-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap208-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap208-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep", + "size_bytes": 2978742, + "sha256": "4b26f8dd3318ec05449b06d2d6be7cf0be78e602ecacfc92998bc4ba0496d884", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap208-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap224-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap224-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep", + "size_bytes": 2860598, + "sha256": "c61863931bb82ea1796c2a240b11c790286f5ee1a2f8109004cf3e633aac07d6", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap224-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap240-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap240-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep", + "size_bytes": 2772103, + "sha256": "4200f4661bcfa722a03f8d28437c5d78e130fae721105518b376aeaca8b0998c", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-cap240-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-deployed-reference.json", + "size_bytes": 4513, + "sha256": "1ad7736973cf4a8975e8e22f99d130918a81f50ddd8b9677f561953aaa7c4922", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-deployed-reference.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json", + "size_bytes": 5478, + "sha256": "4765d80b7a49003680f64e3661982cf5fa64a7a386e78f79cabc40530bd09f82", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json", + "size_bytes": 5498, + "sha256": "e23f210303d7c73a13ed185a58914cb924da1c8944ac328f47899912a8ac62e0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-interleave-softmax-latency.json", + "size_bytes": 5489, + "sha256": "a4613040c132d87447065483e63f2e66cf467a2faeb632f72ba4f010eee10ab2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-interleave-softmax-latency.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep", + "size_bytes": 2768968, + "sha256": "4a0fd35e60166a4d74906bd68406c848a07051482076cc04bb13752bde28080e", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-latency.json", + "size_bytes": 5471, + "sha256": "0ba04433c7985f8fe566f7d787e1cd1fda9bf4159f0be1062d57a21c548929dc", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-latency.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep", + "size_bytes": 2646478, + "sha256": "f5c81c03dafef6f55705c7cf8fc366fbcdb79f21676fcc9ac808fbecbfa23174", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json", + "size_bytes": 13821, + "sha256": "e0d8b3b40978dc4de39b7bd882bf5213d4ea4f27ca7f025e3374a07d4c6bb086", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-register-cap-latency.json", + "size_bytes": 13840, + "sha256": "49adb8eae5268406c0593491ce0bb561c02a5331fb330f2946013de7fe2c3d03", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-register-cap-latency.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-capture.json", + "size_bytes": 1069, + "sha256": "a1b6d2599e3d82dcc13be81bd40f765be73f9d4137718273e8c6de1bd87b4408", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep", + "size_bytes": 2753860, + "sha256": "b39df62abd4cc6c6ca3be0791447cb239af1e66a1c40de06a6a67dcd99598915", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json", + "size_bytes": 5475, + "sha256": "3be011df053f8e22f247b9a33fc9d9e30747aa276edeef5f1587ee5807cfc2a7", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-source-baseline.json", + "size_bytes": 4566, + "sha256": "d6968abc601610ca3c440125f912bccdd943b6249b40b2876ff601195a58ce7a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-source-baseline.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block0-post-removal-smoke.json", + "size_bytes": 2723, + "sha256": "ed52de337353a53a28615794188fe68f2a94cb0fc88c1274213e1c08656c570e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block0-post-removal-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block0-smoke.json", + "size_bytes": 2461, + "sha256": "ab8f8153d59176b79b8be3f8d15c79298466596f543e705fe36584384e5df652", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block0-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block-gate.json", + "size_bytes": 7574, + "sha256": "ec3f522ec117fdb86431a1be7f7efc7ea45e0f9a3adc9d16caf866419a682e2e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block-gate.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-baseline-l2.json", + "size_bytes": 552, + "sha256": "c7fffb6b5ebeb748700b9dcfab34f9dd9cb82c9bfbf3dadaa7d1ae5f69a7e0f4", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-baseline-l2.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep", + "size_bytes": 85127877, + "sha256": "961cba7d92925c4ed740327f17d43bd160be72266ba398307dbd0a463fbecb89", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-baseline-l2.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-candidate-l2.json", + "size_bytes": 553, + "sha256": "804835abf431ad926eb8d9828ae8618d9e4bc81113f1a0c55324154c86046e6e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-candidate-l2.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep", + "size_bytes": 74904976, + "sha256": "12ce74472ba25c4dfda1907bd054b7bb1637ce6f8701826013b6aa9e108b3d8a", + "artifact_class": "nsight_compute_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-randomized.json", + "size_bytes": 4207, + "sha256": "99eb067955796bd768da9b0659483d129b860d8f69f03a9f237f3b3b94ca885d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-randomized.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-real-parity.json", + "size_bytes": 12023, + "sha256": "378aac3a8754f2f6318c526798239ce58204caa5c186a416d2b71af7ddd6db19", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-real-parity.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json", + "size_bytes": 4348, + "sha256": "e3c7b0f56125a358025184ca5c2b188d61999f8c3fdbd82fea41ee987b931f00", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json", + "size_bytes": 5487, + "sha256": "74b69b08873f31e98a0d40c501186e1f5853176c653381a1d4e65ea5d9a28552", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vprep-canonical-repeat.json", + "size_bytes": 1265, + "sha256": "d88e0b897d233989b7ae644587da924ed91cd47803f201f0768464d520afde26", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vprep-canonical-repeat.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vprep-randomized.json", + "size_bytes": 7299, + "sha256": "0c05e04a46c2f36c98bd5d38fad1a7ed15227e5c441a3521a815b73e261e19e4", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vprep-randomized.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vs-sol-block24-1344x768-124f.json", + "size_bytes": 5181, + "sha256": "a879d96c45a8dcfcc3011c0354ba46f3de40701fb75be543e44609e3515ef64e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage2-vs-sol-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json", + "size_bytes": 3133, + "sha256": "2781c3110051f78ebd9c5deae6ca882532f38f30e1776e64f82c5ad1ef49e3b2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage-strided-nhd-blocks-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json", + "size_bytes": 6358, + "sha256": "de934e74527121f808bbc3d8baf510c47dea09d8116616aabd2bd70d9bbf5e9f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-memory-sage2-eager-1344x768-124f-2step.json", + "size_bytes": 452, + "sha256": "de4c112770af07d43e4a11f22030f6fd45f78ecfbc4f921dcb312f0c804be9ca", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-memory-sage2-eager-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-memory-sage2-fused-elementwise-1344x768-124f-2step.json", + "size_bytes": 452, + "sha256": "8ca6caa6b46ba2c113229ff7257c2f411dd453af2f95218b193569c1389a75ca", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-memory-sage2-fused-elementwise-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sage2-1344x768-124f-2step.json", + "size_bytes": 990, + "sha256": "e5f3be255d8f4891923329024b02081b59fffaceadb7e0fab316de9d304bb3c9", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sage2-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sol-bf16-1344x768-124f-2step.json", + "size_bytes": 997, + "sha256": "d6632c9649c2c530d237bf5a9f56761d5061411087e9af1a2db527e7f6708fe8", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sol-bf16-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sol-int8qk-1344x768-124f-2step.json", + "size_bytes": 996, + "sha256": "ea33165cec14838357a75f52d5376225b756b68f3f2abe3054e88e25fbee5ccb", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-stages-sol-int8qk-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-1344x768-124f-2step.json", + "size_bytes": 375, + "sha256": "77d8c9a297f336dfcff4c7370386abb13fe294d0a382203d3dd3159d65cc159b", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-eager-1344x768-124f-12step.json", + "size_bytes": 377, + "sha256": "db44e29e7606abd89e88d0b0d5dba9b71d085ef47d9d697233fd1f9046e2caa6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-eager-1344x768-124f-12step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-fused-elementwise-1344x768-124f-12step.json", + "size_bytes": 379, + "sha256": "cf7f8b6b1fffc683cf257d9ef48dfa4a1166d43d8987231c26f44f6390e0d35c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-fused-elementwise-1344x768-124f-12step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-fused-elementwise-1344x768-124f-2step.json", + "size_bytes": 375, + "sha256": "6a60144e76ff2eec6ca94acacc53a73dff46f79640b652923ba80c161597086f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sage2-fused-elementwise-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sol-bf16-1344x768-124f-2step.json", + "size_bytes": 380, + "sha256": "e4fac38219d11a0008220df7567353d6e367f11442a2448f3599074829456d45", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sol-bf16-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sol-int8qk-1344x768-124f-2step.json", + "size_bytes": 381, + "sha256": "3945851302951a9b9421dd38d5a7871609d990867a589b2601904dddaa3765e1", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sampling-uninstrumented-sol-int8qk-1344x768-124f-2step.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sol-layout-block24-1344x768-124f.json", + "size_bytes": 6749, + "sha256": "aa8190ad5da75259a0b441bcf5eaa59d40d1ee7a8d035c987f75484a0f085fe8", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-sol-layout-block24-1344x768-124f.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-strided-nhd-block-capture.json", + "size_bytes": 352, + "sha256": "39acf0c5f77c88f1d71e16e682011af401ee73bea289481dd1041c2938609574", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-strided-nhd-block-capture.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-strided-nhd-block.qdstrm", + "size_bytes": 2203453, + "sha256": "ffac3a138de191af3cb857bd807d34adc74d37e81e61a39a237a8e0f4846a41c", + "artifact_class": "nsight_qdstrm_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/gb10-strided-nhd-block.qdstrm" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-report-7cc03f3.json", + "size_bytes": 2551, + "sha256": "583346e66878b4ce7cd5124ff35fa6efdab334a0f3e40d059aed5f0a1c3a7741", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-report-7cc03f3.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-report-7cc03f3.md", + "size_bytes": 1975, + "sha256": "88f31df5d9fe98997b6cab7855c113544bf214629c4849cd95a7ad2fd75f5002", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-report-7cc03f3.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block0-7cc03f3.json", + "size_bytes": 4048, + "sha256": "f5eb405fb7ec6e9dd12c7bc27c455cb27646270f479ebcfce40c1fc6287243a3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block0-7cc03f3.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block24-7cc03f3.json", + "size_bytes": 4054, + "sha256": "1b9308604bde102edd7ae189aae90b8c1108d240d58afb0ddd2c30f0dd7de8b7", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block24-7cc03f3.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block49-7cc03f3.json", + "size_bytes": 4055, + "sha256": "f7caefd86f1f1d6044c8d4ef46b6d35c02fac9ef00a0d1b71eed1190946b8318", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sage2-block49-7cc03f3.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-smoke.json", + "size_bytes": 19733, + "sha256": "3d045c6853fa809ac42ff9e27b19d5aa89290dc20ad6950edfc104b2da510f72", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-smoke.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-vortex-50.json", + "size_bytes": 19793, + "sha256": "2c2681f5ea5707fefb4f29a62289998ec19311e15153493d1a36998dbd9293cd", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-vortex-50.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-vortex-50.md", + "size_bytes": 4912, + "sha256": "e0c4ac5deb8fe0faac7d2a70ce19eb0b36d6f581755d83ff31f9fde7ac15db28", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/h3-block-profile-sol-native-vortex-50.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.mp4", + "size_bytes": 52989, + "sha256": "931e023ad714ef43c609d5e2263e934fdbe4f898f32930c309fec439790333a6", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.pt", + "size_bytes": 398917, + "sha256": "0d848adf3bc140d9ec0850672d0d18e9f3238591e0dafaefa7515cf1bc92103a", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.wav", + "size_bytes": 118478, + "sha256": "2ed71552f9abb90d6aa6b6652228c4194141c8d67e8d435b99b8d7f7946e08b0", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-384x384x22.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.mp4", + "size_bytes": 769178, + "sha256": "bb9096a69a43333757b8d04eafd9aad425cab89cb57b2a2284db32ba2ac51fe9", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.pt", + "size_bytes": 7301453, + "sha256": "213647a18783e81c657dd6c9ef3e5137fa862842d54c829a2595cedd24f4ee44", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.wav", + "size_bytes": 662478, + "sha256": "ec05b14f31c04ea4a483c2d73b822d21895ca597bafca3b1b225977426a2a24a", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-ck-int8-960x544x124.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.mp4", + "size_bytes": 50075, + "sha256": "2da7940332daa951133c04902f190e0ccc3f9342b26d98be63b406643a1e287c", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.pt", + "size_bytes": 398901, + "sha256": "67050db6af74e50c296c2aac9a719f81613828e87e466db3a8a124f474e3de8c", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.wav", + "size_bytes": 118478, + "sha256": "8d4fe7d6589d7b74e2fefc4daf074397330b6f576b98571d96274c455709e890", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-384x384x22.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.mp4", + "size_bytes": 753470, + "sha256": "14352e051e68038b5726dd41598334ec20dfd5901d909e673f2fa57f835e7881", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.pt", + "size_bytes": 7301437, + "sha256": "4cd1bf19e214c9b5fc01863f2085b2ad572f295c6a42d96694a852a425a8ff8b", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.wav", + "size_bytes": 662478, + "sha256": "3c77ee5f239a769ded9634051ac926ab277bb857df8bfd9055468230dc6393d3", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-cudnn-960x544x124.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.mp4", + "size_bytes": 49915, + "sha256": "7a8ac2b0c88c377ee716c853d55a7c3ff1aabe04c64438f57264259470737bcb", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.pt", + "size_bytes": 398901, + "sha256": "9d2bc243aded0b4b3b1083ddcfe14023eea4ed76126102112fb20b4346d251f5", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.wav", + "size_bytes": 118478, + "sha256": "e92fae3e93b9b711bf08dc90456d1774fa0683b15b396bcf85db44ed0436afb0", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-384x384x22.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.mp4", + "size_bytes": 773439, + "sha256": "20b47bd9d5a444c34184d5d5a82e56c9a0e57f601cccd3c90991ae4305e95653", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.pt", + "size_bytes": 7301437, + "sha256": "0a750edd8db9ebed3fcfef9e7467370d5782e5e12fda558b2f81c2eb1d8f299a", + "artifact_class": "tensor_capture", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.pt" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.wav", + "size_bytes": 662478, + "sha256": "9f5392872dd2fa7dbeb0a6f77412471c0147ad9b59873e4455afd948a353109d", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-960x544x124.wav" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-960x544x124-contact.png", + "size_bytes": 4183739, + "sha256": "d6ccd0660cc63201f7854cbd6a0524e4ae1c205c5cd6908b9a4ccc790a3ebb8b", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-960x544x124-contact.png" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-960x544x124-side-by-side.mp4", + "size_bytes": 1893684, + "sha256": "6573801de9f19262aaadf5597bc9e00c50ec212203de3b94be6e53a825a6a1ac", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-960x544x124-side-by-side.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-side-by-side.mp4", + "size_bytes": 96666, + "sha256": "7211138b5c3710339377487ef371e17846bb53552aae695156930563279dc487", + "artifact_class": "media_output", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/hot-attn-sage2-cudnn-ck-side-by-side.mp4" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-block24-sage2-d6eabb1.json", + "size_bytes": 34788, + "sha256": "69c998d6df5c86f062df4be78b6f8186583826dfcb434d1647dfcd0ba10d1708", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-block24-sage2-d6eabb1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-quantizers-03dbe14.json", + "size_bytes": 67030, + "sha256": "1171b99692f25dd022102902657141618b70d03a341901f8af5b37593695d12f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-quantizers-03dbe14.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-native-d4c243c.json", + "size_bytes": 45983, + "sha256": "543294b2a08736f50e5ba5cefc537214e9953b309f8fd2d9efa404dd32a4d61a", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-native-d4c243c.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-0fda845.json", + "size_bytes": 45560, + "sha256": "18654fefc9016d6c8a3ea60db4b9b158554c2ea220aa821cb6d523a6145cba34", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-0fda845.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-684b645.json", + "size_bytes": 46531, + "sha256": "26573b9add2210363890b8e6cfa1df933b7551cc350547ba84be14c5b169af86", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-684b645.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-v1-1cc0ffd.json", + "size_bytes": 46497, + "sha256": "7dba70252a34db7720eea305ea885546d0d5fca3741598125db2748c6daf428f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-v1-1cc0ffd.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-v1-tuned-1fec77d.json", + "size_bytes": 46485, + "sha256": "41d05c56b2e87054e8de0dd78d137a3c41c672f173566b36665873fc3cfa38f0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-fc1-fc2-vortex-scale-v1-tuned-1fec77d.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-report-d6eabb1.json", + "size_bytes": 4645, + "sha256": "59d6503ae8bd50ca291176a2f9ee4a501acd169939c135e1a3fbc998b0d91588", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-report-d6eabb1.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-report-d6eabb1.md", + "size_bytes": 1949, + "sha256": "0f17930f4f9d73b83ff6e5f4e3a9f21be36b387fd85f889d1b3735c29828aae2", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-linear-profile-report-d6eabb1.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-8d90cec.json", + "size_bytes": 8238, + "sha256": "892e30eb039ccf266aca924e15436c5e0933ccc19c67701cff63336ad7e06640", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-8d90cec.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-report-8d90cec.json", + "size_bytes": 2743, + "sha256": "6a175798dc3237b4b603f6b6b17e6eac43f4a2751dc9b9f176d2beb8c56ff6e9", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-report-8d90cec.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-report-8d90cec.md", + "size_bytes": 2150, + "sha256": "e4eee8f4ae496297627c254517dd406220579868e3ba69e1d4a37eb1b4d30c62", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-pack-sweep-report-8d90cec.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-quantizer-comparison-report-03dbe14.json", + "size_bytes": 3405, + "sha256": "a3740edee7d70216d2fc0df89ae15ea785df0e90a7aba40e21afe56d2de1cf13", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-quantizer-comparison-report-03dbe14.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-quantizer-comparison-report-03dbe14.md", + "size_bytes": 1963, + "sha256": "faa573179ac4a77b984e956a0dea50380ebb5a0f3462aafaa4fc5418c568bc2a", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-quantizer-comparison-report-03dbe14.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-scale-geometry-sweep-1cc0ffd.json", + "size_bytes": 15373, + "sha256": "bdb3998fbc8115091399a48a9336fbbab73f43549f38121695e260f2fbf7136c", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-scale-geometry-sweep-1cc0ffd.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-native-report-d4c243c.json", + "size_bytes": 1637, + "sha256": "2a3cc59be101ed4f6d88c13a88927d812cb7e24a78404ffdb9ae5baa790be49e", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-native-report-d4c243c.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-native-report-d4c243c.md", + "size_bytes": 1744, + "sha256": "459cb1b2d8d3da06268abef385edfd3a3481f32103fee95eb8d05f6fc2acc8a0", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-native-report-d4c243c.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-report-684b645.json", + "size_bytes": 1665, + "sha256": "fcee3a89c335ff45dd97bd23ca95c6e582d2d6d6043a3b6c0845032a2afcafb0", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-report-684b645.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-report-684b645.md", + "size_bytes": 1793, + "sha256": "5407b7eb10a72ccc95fd4f2d4c8e0a934cb8b37278cf7742f1895939682d48ed", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-report-684b645.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-v1-report-1fec77d.json", + "size_bytes": 3015, + "sha256": "28da6fe6ad980e93770ef5a82e44f0dae37c4ba137e65729e6c8f77bbaf29bc2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-v1-report-1fec77d.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-v1-report-1fec77d.md", + "size_bytes": 2339, + "sha256": "2ee41deb428fb51b73915c7a799533e337a772056962806316b8e78c67f560de", + "artifact_class": "research_report", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/nvfp4-vortex-scale-v1-report-1fec77d.md" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-baseline-f6166ed.json", + "size_bytes": 17394, + "sha256": "e65942e1ab874988f11e4e31da214fd17cdebd8d64729f402339d5708bcb5e7f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-baseline-f6166ed.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fastops-pairedrope-tile384-ea7712b.json", + "size_bytes": 7222, + "sha256": "30b684c5e5aa82ba87218714393e48957e27e75052a71b68ef97e83893c3e65d", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fastops-pairedrope-tile384-ea7712b.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fastops-tile384-8595dd8.json", + "size_bytes": 7222, + "sha256": "10f2c5748dee137ceed004f7b65a8284416db45c9c824520c237c83f32596be7", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fastops-tile384-8595dd8.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-cli-tile256.json", + "size_bytes": 828, + "sha256": "2f6ded62b27977bce5f4266ca8f669c4bdbf24cf279a2f04eda3eeb2e70afad2", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-cli-tile256.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile256-ea7712b.json", + "size_bytes": 17259, + "sha256": "845880cd6bd5857d976ae5f296d1b2444adf6bb80fd6340317de758bf07f6569", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile256-ea7712b.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile256-output-ea7712b.json", + "size_bytes": 834, + "sha256": "1a6299946432b5811a247470abd9b47c0c0638d8e22866a5e06171086434ec21", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile256-output-ea7712b.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile384-ea7712b.json", + "size_bytes": 7235, + "sha256": "be220e25c5365addd432825b9aa0cd422a4f4bd1d2d85813b08436d7701846a6", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-fp16-tile384-ea7712b.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile320-f6166ed.json", + "size_bytes": 9434, + "sha256": "6e1084e91a08449fa0d7af237d5f839d4509f523875552d622fe433878e02687", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile320-f6166ed.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile384-f6166ed.json", + "size_bytes": 7226, + "sha256": "b0266510ef24af19341abe1459fbba6ee4ff63971ca05e5a9ac7ed1ce481bb0f", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile384-f6166ed.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile512-f6166ed.json", + "size_bytes": 5008, + "sha256": "057a1504305693b57afd6a0e183b2a1aa24e6da8f2084be6b162ce99e5a860e3", + "artifact_class": "benchmark_json", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/vae-direct-tile512-f6166ed.json" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/warmup-direct-fp16-1s.log", + "size_bytes": 3278, + "sha256": "638e27b098ad80d71db68a79c2248d9f6e8a4cc5b9e9cc65ddfc427c0d2f05f1", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/warmup-direct-fp16-1s.log" + }, + { + "scope": "spark", + "path": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/warmup-direct-sage3-fp16-1s.log", + "size_bytes": 3289, + "sha256": "9b711de3301244891d04465249db49a9233951928607a50a6ef9823a5f0fab58", + "artifact_class": "run_log", + "location": "/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks/warmup-direct-sage3-fp16-1s.log" + } + ] +} diff --git a/research/cute_nvfp4_ring/README.md b/research/cute_nvfp4_ring/README.md new file mode 100644 index 0000000..dc01e14 --- /dev/null +++ b/research/cute_nvfp4_ring/README.md @@ -0,0 +1,21 @@ +# CuTe NVFP4 and Ring Attempt Recovery + +The bounded-ring runtime module and validator-only CuTe experiments were already +isolated as untracked files, so they are preserved as add-file patches: + +- `patches/0001-cute-qkv-ring-module.patch` +- `patches/0002-conversion-contract.patch` +- `patches/0003-tile-producer.patch` +- `patches/0004-ring-validator.patch` +- `patches/0005-real-tile-validator.patch` +- `patches/0006-h3-gemm-validator.patch` +- `patches/0007-full-ring-validator.patch` +- `patches/0008-qkv-block-gate.patch` +- `patches/0009-qkv-runtime-gate.patch` + +The caller-owned `_into` CUDA producer needed by the ring is in the explicitly +mixed shared CUDA patch under `../shared_cuda_experimental_worktree/patches/`. +The CuTe kernel itself was referenced through an external path and is not in the +worktree or reachable Git objects; see `UNRECOVERABLE.md`. + +Nothing here is registered with production dispatch or build defaults. diff --git a/research/cute_nvfp4_ring/UNRECOVERABLE.md b/research/cute_nvfp4_ring/UNRECOVERABLE.md new file mode 100644 index 0000000..b6f2045 --- /dev/null +++ b/research/cute_nvfp4_ring/UNRECOVERABLE.md @@ -0,0 +1,12 @@ +# Unrecoverable CuTe Source + +The exact external file +`dense_blockscaled_gemm_persistent_cooperative_vortex_alpha.py` is not present in +this checkout, current untracked files, reachable history, reflogs, or dangling +Git objects. The recovered ring module refers to its deployed location under +`/opt/h3-blackwell-runtime/tools/`, and validators accepted a path argument, but +that does not recover the implementation. + +The direct streamed-A, persistent cooperative, alternating, and full-ring kernel +variants described by benchmark names likewise have no exact kernel source in +this tree. Reconstructing them from reports would not be defensible recovery. diff --git a/research/cute_nvfp4_ring/patches/0001-cute-qkv-ring-module.patch b/research/cute_nvfp4_ring/patches/0001-cute-qkv-ring-module.patch new file mode 100644 index 0000000..c601cd3 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0001-cute-qkv-ring-module.patch @@ -0,0 +1,288 @@ +diff --git a/src/h3_blackwell_runtime/cute_qkv_ring.py b/src/h3_blackwell_runtime/cute_qkv_ring.py +new file mode 100644 +index 0000000..5108a4d +--- /dev/null ++++ b/src/h3_blackwell_runtime/cute_qkv_ring.py +@@ -0,0 +1,282 @@ ++"""Opt-in CuTe bounded-ring backend for full-width H3 QKV projections.""" ++ ++from __future__ import annotations ++ ++import importlib.util ++import os ++import sys ++import threading ++import weakref ++from dataclasses import dataclass ++from pathlib import Path ++ ++import torch ++ ++from .nvfp4_quant import nvfp4_activation_scale, vortex_native_quantize_nvfp4_into ++ ++ ++def _enabled(name: str) -> bool: ++ return os.getenv(name, "").lower() in {"1", "true", "yes", "on"} ++ ++ ++def _output_tensor(storage: torch.Tensor): ++ from cutlass.cute.runtime import from_dlpack ++ ++ tensor = from_dlpack(storage.unsqueeze(-1), assumed_align=16) ++ return tensor.mark_compact_shape_dynamic( ++ mode=1, stride_order=(2, 0, 1), divisibility=1, ++ ) ++ ++ ++def _scale_tensor(storage: torch.Tensor): ++ import cutlass ++ from cutlass.cute.runtime import from_dlpack ++ ++ tensor = from_dlpack(storage.view(torch.uint8).unsqueeze(-1), assumed_align=16) ++ tensor.element_type = cutlass.Float8E4M3FN ++ return tensor.mark_layout_dynamic(leading_dim=1) ++ ++ ++def _weight_tensor(storage: torch.Tensor): ++ import cutlass ++ import cutlass.torch as cutlass_torch ++ ++ lookup = torch.tensor( ++ [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, ++ -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], ++ device=storage.device, ++ dtype=torch.float32, ++ ) ++ codes = torch.stack((storage >> 4, storage & 0x0F), dim=-1).reshape( ++ storage.shape[0], -1, ++ ) ++ logical = lookup[codes.long()].unsqueeze(-1) ++ tensor, backing = cutlass_torch.cute_tensor_like( ++ logical, ++ cutlass.Float4E2M1FN, ++ is_dynamic_layout=True, ++ assumed_align=16, ++ ) ++ return tensor, backing ++ ++ ++def _load_kernel(path: Path): ++ if not path.is_file(): ++ raise FileNotFoundError(f"CuTe QKV kernel not found: {path}") ++ sys.path.insert(0, str(path.parent)) ++ spec = importlib.util.spec_from_file_location("h3_cute_qkv_ring_kernel", path) ++ if spec is None or spec.loader is None: ++ raise ImportError(f"Cannot load CuTe QKV kernel: {path}") ++ module = importlib.util.module_from_spec(spec) ++ spec.loader.exec_module(module) ++ return module ++ ++ ++@dataclass ++class _PreparedWeight: ++ b: object ++ backing: torch.Tensor ++ sfb: object ++ ++ ++@dataclass ++class _Workspace: ++ a: object ++ a_backing: torch.Tensor ++ qdata: torch.Tensor ++ block_scale: torch.Tensor ++ sfa: object ++ alpha: torch.Tensor ++ alpha_argument: object ++ outputs: dict[int, tuple[torch.Tensor, list[object]]] ++ ++ ++class _QkvRingBackend: ++ def __init__(self) -> None: ++ self.capacity = int(os.getenv("H3_CUTE_QKV_RING_CAPACITY", "2048")) ++ if self.capacity <= 0 or self.capacity % 128: ++ raise ValueError("H3_CUTE_QKV_RING_CAPACITY must be a positive multiple of 128") ++ default_path = Path("/opt/h3-blackwell-runtime/tools/dense_blockscaled_gemm_persistent_cooperative_vortex_alpha.py") ++ self.kernel_path = Path(os.getenv("H3_CUTE_QKV_KERNEL", str(default_path))) ++ self.strict = _enabled("H3_CUTE_QKV_RING_STRICT") ++ self._lock = threading.Lock() ++ self._module = None ++ self._workspace: dict[int, _Workspace] = {} ++ self._weights: weakref.WeakKeyDictionary = weakref.WeakKeyDictionary() ++ self._compiled: dict[int, tuple[object, object]] = {} ++ self.disabled_reason: str | None = None ++ ++ def _ineligible(self, message: str): ++ if self.strict: ++ raise RuntimeError(message) ++ return None ++ ++ def _eligible(self, linear, x: torch.Tensor) -> str | None: ++ if linear.role != "h3_attn_qkv": ++ return "projection role is not H3 QKV" ++ if linear.in_features != 5376 or linear.out_features != 21504: ++ return "QKV projection is sharded or has an unsupported shape" ++ if linear.full_precision_matrix_mult or linear.pre_quant_scale is not None: ++ return "QKV projection uses an unsupported quantization policy" ++ if linear.bias is not None or linear.output_dtype != torch.bfloat16: ++ return "QKV bias/output dtype is unsupported" ++ if not x.is_cuda or x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous(): ++ return "QKV activation must be contiguous 2D CUDA BF16" ++ if x.requires_grad or torch.is_grad_enabled(): ++ return "QKV ring is inference-only" ++ major, _ = torch.cuda.get_device_capability(x.device) ++ if major != 12: ++ return "QKV ring is currently validated only on SM12x" ++ return None ++ ++ def _prepare_workspace(self, device: torch.device) -> _Workspace: ++ import cutlass ++ import cutlass.torch as cutlass_torch ++ from cutlass.cute.runtime import from_dlpack ++ ++ index = device.index or 0 ++ workspace = self._workspace.get(index) ++ if workspace is not None: ++ return workspace ++ a, a_backing = cutlass_torch.cute_tensor_like( ++ torch.zeros( ++ self.capacity, 5376, 1, device=device, dtype=torch.float32, ++ ), ++ cutlass.Float4E2M1FN, ++ is_dynamic_layout=True, ++ assumed_align=16, ++ ) ++ qdata = a_backing.view(torch.uint8).flatten()[ ++ : self.capacity * 5376 // 2 ++ ].reshape(self.capacity, 5376 // 2) ++ block_scale = torch.empty( ++ self.capacity, 5376 // 16, device=device, dtype=torch.float8_e4m3fn, ++ ) ++ alpha = torch.empty(1, device=device, dtype=torch.float32) ++ workspace = _Workspace( ++ a=a, ++ a_backing=a_backing, ++ qdata=qdata, ++ block_scale=block_scale, ++ sfa=_scale_tensor(block_scale), ++ alpha=alpha, ++ alpha_argument=from_dlpack(alpha, assumed_align=4), ++ outputs={}, ++ ) ++ self._workspace[index] = workspace ++ return workspace ++ ++ def _prepare_weight(self, linear) -> _PreparedWeight: ++ prepared = self._weights.get(linear) ++ if prepared is not None: ++ return prepared ++ b, backing = _weight_tensor(linear.weight) ++ prepared = _PreparedWeight( ++ b=b, ++ backing=backing, ++ sfb=_scale_tensor(linear.weight_scale), ++ ) ++ self._weights[linear] = prepared ++ return prepared ++ ++ def _output(self, workspace: _Workspace, rows: int): ++ padded_rows = ((rows + self.capacity - 1) // self.capacity) * self.capacity ++ cached = workspace.outputs.get(padded_rows) ++ if cached is not None: ++ return cached ++ output = torch.empty( ++ padded_rows, ++ 21504, ++ device=workspace.qdata.device, ++ dtype=torch.bfloat16, ++ ) ++ chunks = [ ++ _output_tensor(output[start : start + self.capacity]) ++ for start in range(0, padded_rows, self.capacity) ++ ] ++ cached = (output, chunks) ++ workspace.outputs[padded_rows] = cached ++ return cached ++ ++ def _compile(self, device: torch.device, workspace: _Workspace, weight: _PreparedWeight, c): ++ import cutlass ++ import cutlass.cute as cute ++ import cutlass.torch as cutlass_torch ++ ++ index = device.index or 0 ++ cached = self._compiled.get(index) ++ if cached is not None: ++ return cached ++ if self._module is None: ++ self._module = _load_kernel(self.kernel_path) ++ gemm = self._module.Sm120BlockScaledGemmKernel( ++ cutlass.Float32, 16, (128, 128, 128), (128, 128), ++ ) ++ stream = cutlass_torch.default_stream() ++ max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) ++ compiled = cute.compile( ++ gemm, ++ workspace.a, ++ weight.b, ++ workspace.sfa, ++ weight.sfb, ++ c, ++ workspace.alpha_argument, ++ max_active_clusters, ++ stream, ++ ) ++ cached = (compiled, stream) ++ self._compiled[index] = cached ++ return cached ++ ++ def __call__(self, linear, x: torch.Tensor): ++ reason = self._eligible(linear, x) ++ if reason is not None: ++ return self._ineligible(reason) ++ try: ++ with self._lock: ++ workspace = self._prepare_workspace(x.device) ++ weight = self._prepare_weight(linear) ++ output, c_chunks = self._output(workspace, x.shape[0]) ++ compiled, stream = self._compile( ++ x.device, workspace, weight, c_chunks[0], ++ ) ++ scale = nvfp4_activation_scale(x).float() ++ workspace.alpha.copy_(scale * linear.weight_scale_2.float()) ++ for index, start in enumerate(range(0, x.shape[0], self.capacity)): ++ stop = min(start + self.capacity, x.shape[0]) ++ vortex_native_quantize_nvfp4_into( ++ x[start:stop], ++ scale, ++ workspace.qdata, ++ workspace.block_scale, ++ hi_first=False, ++ ) ++ compiled( ++ workspace.a, ++ weight.b, ++ workspace.sfa, ++ weight.sfb, ++ c_chunks[index], ++ workspace.alpha_argument, ++ stream, ++ ) ++ return output[: x.shape[0], : linear.out_features] ++ except (ImportError, FileNotFoundError, RuntimeError) as error: ++ self.disabled_reason = str(error) ++ return self._ineligible(f"QKV ring initialization failed: {error}") ++ ++ ++_BACKEND: _QkvRingBackend | None = None ++ ++ ++def qkv_ring_linear(linear, x: torch.Tensor): ++ global _BACKEND ++ if _BACKEND is None: ++ try: ++ _BACKEND = _QkvRingBackend() ++ except (ImportError, ValueError) as error: ++ if _enabled("H3_CUTE_QKV_RING_STRICT"): ++ raise ++ return None ++ return _BACKEND(linear, x) diff --git a/research/cute_nvfp4_ring/patches/0002-conversion-contract.patch b/research/cute_nvfp4_ring/patches/0002-conversion-contract.patch new file mode 100644 index 0000000..6014d7d --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0002-conversion-contract.patch @@ -0,0 +1,120 @@ +diff --git a/tools/validate_cute_nvfp4_conversion.py b/tools/validate_cute_nvfp4_conversion.py +new file mode 100644 +index 0000000..6172960 +--- /dev/null ++++ b/tools/validate_cute_nvfp4_conversion.py +@@ -0,0 +1,114 @@ ++"""Validate CuTe DSL E2M1/E4M3 conversion boundaries used by the streamed producer.""" ++ ++from __future__ import annotations ++ ++import json ++from pathlib import Path ++ ++import cutlass ++import cutlass.cute as cute ++import cutlass.torch as cutlass_torch ++import torch ++from cutlass.cute.runtime import from_dlpack ++ ++ ++VALUES = ( ++ -6.1, -6.0, -5.0, -4.999, -3.5, -3.499, -2.5, -2.499, ++ -1.75, -1.749, -1.25, -1.249, -0.75, -0.749, -0.25, -0.249, ++ -0.0, 0.0, 0.249, 0.25, 0.251, 0.749, 0.75, 0.751, ++ 1.249, 1.25, 1.251, 1.749, 1.75, 1.751, 2.499, 2.5, ++ 2.501, 3.499, 3.5, 3.501, 4.999, 5.0, 5.001, 6.0, 6.1, ++) ++PADDED_COUNT = ((len(VALUES) + 15) // 16) * 16 ++ ++ ++@cute.kernel ++def conversion_kernel(source: cute.Tensor, fp4: cute.Tensor, fp8: cute.Tensor): ++ if cute.arch.thread_idx()[0] == 0: ++ fp4_tiles = cute.zipped_divide(fp4, (8,)) ++ fp8_tiles = cute.zipped_divide(fp8, (8,)) ++ fp4_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN) ++ fp8_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float8E4M3FN) ++ source_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) ++ fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) ++ fp8_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) ++ for fragment_index in cutlass.range_constexpr(PADDED_COUNT // 8): ++ offset = fragment_index * 8 ++ for element in cutlass.range_constexpr(8): ++ source_fragment[element] = source[offset + element] ++ values = source_fragment.load() ++ fp4_fragment.store(values.to(cutlass.Float4E2M1FN)) ++ fp8_fragment.store(values.to(cutlass.Float8E4M3FN)) ++ cute.copy(fp4_store, fp4_fragment, fp4_tiles[(None, fragment_index)]) ++ cute.copy(fp8_store, fp8_fragment, fp8_tiles[(None, fragment_index)]) ++ ++ ++@cute.jit ++def convert(source: cute.Tensor, fp4: cute.Tensor, fp8: cute.Tensor): ++ conversion_kernel(source, fp4, fp8).launch(grid=(1, 1, 1), block=(1, 1, 1)) ++ ++ ++def fp4_code(value: float) -> int: ++ negative = torch.signbit(torch.tensor(value)).item() ++ magnitude = abs(value) ++ if magnitude > 5.0: ++ code = 7 ++ elif magnitude >= 3.5: ++ code = 6 ++ elif magnitude > 2.5: ++ code = 5 ++ elif magnitude >= 1.75: ++ code = 4 ++ elif magnitude > 1.25: ++ code = 3 ++ elif magnitude >= 0.75: ++ code = 2 ++ elif magnitude > 0.25: ++ code = 1 ++ else: ++ code = 0 ++ return code | (8 if negative else 0) ++ ++ ++def main() -> None: ++ padded_values = VALUES + (0.0,) * (PADDED_COUNT - len(VALUES)) ++ source_torch = torch.tensor(padded_values, device="cuda", dtype=torch.float32) ++ source = from_dlpack(source_torch, assumed_align=4).mark_layout_dynamic() ++ fp4, fp4_torch = cutlass_torch.cute_tensor_like( ++ torch.zeros_like(source_torch), cutlass.Float4E2M1FN, is_dynamic_layout=True, assumed_align=16, ++ ) ++ fp8, fp8_torch = cutlass_torch.cute_tensor_like( ++ torch.zeros_like(source_torch), cutlass.Float8E4M3FN, is_dynamic_layout=True, assumed_align=16, ++ ) ++ compiled = cute.compile(convert, source, fp4, fp8) ++ compiled(source, fp4, fp8) ++ torch.cuda.synchronize() ++ ++ fp4_bytes = fp4_torch.view(torch.uint8).flatten().cpu().tolist() ++ expected_codes = [fp4_code(value) for value in padded_values] ++ expected_low_first = [ ++ expected_codes[index] | ((expected_codes[index + 1] if index + 1 < len(expected_codes) else 0) << 4) ++ for index in range(0, len(expected_codes), 2) ++ ] ++ fp8_bytes = fp8_torch.view(torch.uint8).flatten().cpu() ++ expected_fp8 = source_torch.to(torch.float8_e4m3fn).view(torch.uint8).cpu() ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "cutlass_dsl": "4.6.2", ++ "value_count": len(VALUES), ++ "padded_value_count": len(padded_values), ++ "fp4_torch_shape": list(fp4_torch.shape), ++ "fp4_bytes": fp4_bytes, ++ "expected_low_first_bytes": expected_low_first, ++ "fp4_prefix_equal": fp4_bytes[:len(expected_low_first)] == expected_low_first, ++ "fp8_equal": torch.equal(fp8_bytes[:len(padded_values)], expected_fp8), ++ "fp8_difference_count": int((fp8_bytes[:len(padded_values)] != expected_fp8).sum().item()), ++ } ++ output = Path("/output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-conversion-contract.json") ++ output.parent.mkdir(parents=True, exist_ok=True) ++ output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0003-tile-producer.patch b/research/cute_nvfp4_ring/patches/0003-tile-producer.patch new file mode 100644 index 0000000..dce5f58 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0003-tile-producer.patch @@ -0,0 +1,230 @@ +diff --git a/tools/validate_cute_nvfp4_tile_producer.py b/tools/validate_cute_nvfp4_tile_producer.py +new file mode 100644 +index 0000000..d03c79e +--- /dev/null ++++ b/tools/validate_cute_nvfp4_tile_producer.py +@@ -0,0 +1,224 @@ ++"""Validate a CuTe BF16-to-NVFP4 producer on one 128x128 activation tile.""" ++ ++from __future__ import annotations ++ ++import json ++from pathlib import Path ++from typing import Optional ++ ++import cutlass ++import cutlass.cute as cute ++import cutlass.torch as cutlass_torch ++import torch ++from cutlass import Float32 ++from cutlass.cutlass_dsl import dsl_user_op ++from cutlass._mlir import ir ++from cutlass._mlir.dialects import llvm ++from cutlass.cute.runtime import from_dlpack ++ ++from h3_blackwell_runtime.nvfp4_quant import nvfp4_activation_scale ++ ++ ++TILE = 128 ++BLOCK = 16 ++BLOCKS_PER_ROW = TILE // BLOCK ++JOBS = TILE * BLOCKS_PER_ROW ++ ++ ++@dsl_user_op ++def rcp_approx_ftz_f32( ++ x: Float32, ++ *, ++ loc: Optional[ir.Location] = None, ++ ip: Optional[ir.InsertionPoint] = None, ++) -> Float32: ++ result = llvm.inline_asm( ++ Float32.mlir_type, ++ [x.ir_value(loc=loc, ip=ip)], ++ "rcp.approx.ftz.f32 $0, $1;", ++ "=f,f", ++ has_side_effects=False, ++ asm_dialect=0, ++ loc=loc, ++ ip=ip, ++ ) ++ return Float32(result) ++ ++ ++@cute.kernel ++def tile_producer_kernel( ++ source: cute.Tensor, ++ tensor_scale: cute.Tensor, ++ fp4: cute.Tensor, ++ block_scales: cute.Tensor, ++ scalar_scales: cute.Tensor, ++): ++ row = cute.arch.thread_idx()[0] ++ fp4_linear = cute.make_tensor( ++ fp4.iterator, ++ cute.make_layout((TILE * TILE,), stride=(1,)), ++ ) ++ fp4_tiles = cute.zipped_divide(fp4_linear, (8,)) ++ scale_tiles = cute.zipped_divide(block_scales, (8,)) ++ fp4_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN) ++ fp8_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float8E4M3FN) ++ source_fragment = cute.make_rmem_tensor((16,), cutlass.Float32) ++ normalized_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) ++ fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) ++ scale_source = cute.make_rmem_tensor((8,), cutlass.Float32) ++ scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) ++ decoded_scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) ++ scale = tensor_scale[0] ++ ++ for block_column in cutlass.range_constexpr(BLOCKS_PER_ROW): ++ job = row * BLOCKS_PER_ROW + block_column ++ column = block_column * BLOCK ++ maximum = cutlass.Float32(0.0) ++ for element in cutlass.range_constexpr(BLOCK): ++ value = source[row, column + element] ++ source_fragment[element] = value ++ maximum = cutlass.max(cutlass.max(value, -value), maximum) ++ ++ raw_block_scale = (maximum / cutlass.Float32(6.0)) / scale ++ for element in cutlass.range_constexpr(8): ++ scale_source[element] = raw_block_scale ++ scale_values = scale_source.load() ++ scale_values = cute.where( ++ scale_values <= cutlass.Float32(448.0), ++ scale_values, ++ cutlass.Float32(448.0), ++ ) ++ scale_fragment.store(scale_values.to(cutlass.Float8E4M3FN)) ++ cute.copy(fp8_store, scale_fragment, scale_tiles[(None, job)]) ++ scalar_scales[job] = scale_fragment[0] ++ decoded_scale_fragment.store(scale_fragment.load().to(cutlass.Float32)) ++ decoded_scale = decoded_scale_fragment[0] ++ raw_encode_scale = rcp_approx_ftz_f32(decoded_scale * scale) ++ for element in cutlass.range_constexpr(8): ++ scale_source[element] = raw_encode_scale ++ encode_scale_values = scale_source.load() ++ encode_scale_values = cute.where( ++ encode_scale_values <= cutlass.Float32(3.402823466e38), ++ encode_scale_values, ++ cutlass.Float32(3.402823466e38), ++ ) ++ scale_source.store(encode_scale_values) ++ encode_scale = scale_source[0] ++ ++ for half in cutlass.range_constexpr(2): ++ for element in cutlass.range_constexpr(8): ++ normalized = source_fragment[half * 8 + element] * encode_scale ++ normalized_fragment[element] = normalized ++ fp4_fragment.store(normalized_fragment.load().to(cutlass.Float4E2M1FN)) ++ output_tile = job * 2 + half ++ cute.copy(fp4_store, fp4_fragment, fp4_tiles[(None, output_tile)]) ++ ++ ++@cute.jit ++def produce_tile( ++ source: cute.Tensor, ++ tensor_scale: cute.Tensor, ++ fp4: cute.Tensor, ++ block_scales: cute.Tensor, ++ scalar_scales: cute.Tensor, ++): ++ tile_producer_kernel(source, tensor_scale, fp4, block_scales, scalar_scales).launch( ++ grid=(1, 1, 1), block=(TILE, 1, 1), ++ ) ++ ++ ++def unswizzle_scales(physical: torch.Tensor) -> torch.Tensor: ++ rows, scale_columns = physical.shape ++ row = torch.arange(rows, device=physical.device).view(-1, 1) ++ block_column = torch.arange(scale_columns, device=physical.device).view(1, -1) ++ row_in_tile = row % 128 ++ tile = (row // 128) * (scale_columns // 4) + block_column // 4 ++ within = ( ++ ((row_in_tile % 32) // 2) * 32 ++ + block_column % 4 ++ + (row_in_tile // 32) * 4 ++ + (row_in_tile % 2) * 16 ++ ) ++ return physical.flatten()[(tile * 512 + within).long()] ++ ++ ++def main() -> None: ++ import comfy_kitchen as ck ++ ++ torch.manual_seed(440420) ++ random_values = torch.randn(TILE, TILE, device="cuda", dtype=torch.bfloat16) ++ zero_values = torch.zeros_like(random_values) ++ sparse_values = torch.zeros_like(random_values) ++ sparse_values.flatten()[:16] = torch.tensor( ++ [-100, -6, -4, -3, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 3, 6, 100], ++ device="cuda", ++ dtype=torch.bfloat16, ++ ) ++ source_torch = torch.empty_like(random_values) ++ scale_torch = torch.empty(1, device="cuda", dtype=torch.float32) ++ ++ source = from_dlpack(source_torch, assumed_align=16).mark_layout_dynamic(leading_dim=1) ++ scale = from_dlpack(scale_torch, assumed_align=4).mark_layout_dynamic() ++ fp4, fp4_torch = cutlass_torch.cute_tensor_like( ++ torch.zeros_like(source_torch, dtype=torch.float32), ++ cutlass.Float4E2M1FN, ++ is_dynamic_layout=True, ++ assumed_align=16, ++ ) ++ block_scales_torch = torch.zeros(JOBS, 8, device="cuda", dtype=torch.uint8) ++ block_scales = from_dlpack(block_scales_torch.flatten(), assumed_align=16) ++ block_scales.element_type = cutlass.Float8E4M3FN ++ block_scales = block_scales.mark_layout_dynamic() ++ scalar_scales_torch = torch.zeros(JOBS, device="cuda", dtype=torch.uint8) ++ scalar_scales = from_dlpack(scalar_scales_torch, assumed_align=16) ++ scalar_scales.element_type = cutlass.Float8E4M3FN ++ scalar_scales = scalar_scales.mark_layout_dynamic() ++ ++ compiled = cute.compile(produce_tile, source, scale, fp4, block_scales, scalar_scales) ++ cases = [] ++ for name, values in (("random", random_values), ("zeros", zero_values), ("sparse_extremes", sparse_values)): ++ source_torch.copy_(values) ++ scale_torch.copy_(nvfp4_activation_scale(source_torch).float().reshape(1)) ++ expected_qdata, expected_physical_scales = ck.quantize_nvfp4( ++ source_torch, scale_torch, pad_16x=False, ++ ) ++ expected_low_first = ((expected_qdata & 0x0F) << 4) | ((expected_qdata & 0xF0) >> 4) ++ expected_scales = unswizzle_scales(expected_physical_scales.view(torch.uint8)) ++ compiled(source, scale, fp4, block_scales, scalar_scales) ++ torch.cuda.synchronize() ++ actual_fp4 = fp4_torch.view(torch.uint8).flatten()[:TILE * TILE // 2].reshape(TILE, TILE // 2) ++ actual_scales = block_scales_torch[:, 0].reshape(TILE, BLOCKS_PER_ROW) ++ cases.append({ ++ "name": name, ++ "tensor_scale": scale_torch.item(), ++ "fp4_difference_count": int((actual_fp4 != expected_low_first).sum().item()), ++ "fp4_equal": torch.equal(actual_fp4, expected_low_first), ++ "block_scale_difference_count": int((actual_scales != expected_scales).sum().item()), ++ "block_scales_equal": torch.equal(actual_scales, expected_scales), ++ "scalar_block_scales_equal": torch.equal( ++ scalar_scales_torch.reshape(TILE, BLOCKS_PER_ROW), expected_scales, ++ ), ++ "actual_block_scale_bytes": actual_scales.unique().tolist(), ++ "expected_block_scale_bytes": expected_scales.unique().tolist(), ++ }) ++ ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "cutlass_dsl": "4.6.2", ++ "tile": [TILE, TILE], ++ "all_equal": all( ++ case["fp4_equal"] ++ and case["block_scales_equal"] ++ and case["scalar_block_scales_equal"] ++ for case in cases ++ ), ++ "cases": cases, ++ } ++ output = Path("/output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-tile-producer.json") ++ output.parent.mkdir(parents=True, exist_ok=True) ++ output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0004-ring-validator.patch b/research/cute_nvfp4_ring/patches/0004-ring-validator.patch new file mode 100644 index 0000000..1075030 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0004-ring-validator.patch @@ -0,0 +1,277 @@ +diff --git a/tools/validate_cute_nvfp4_ring.py b/tools/validate_cute_nvfp4_ring.py +new file mode 100644 +index 0000000..51217ec +--- /dev/null ++++ b/tools/validate_cute_nvfp4_ring.py +@@ -0,0 +1,271 @@ ++"""Validate and time a bounded 128-row NVFP4 packed-tile ring prototype.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import math ++from pathlib import Path ++ ++import cutlass ++import cutlass.cute as cute ++import cutlass.torch as cutlass_torch ++import torch ++import torch.nn.functional as functional ++from cutlass.cute.runtime import from_dlpack ++ ++from h3_blackwell_runtime.nvfp4_quant import ( ++ nvfp4_activation_scale, ++ vortex_native_quantize_nvfp4, ++ vortex_native_quantize_nvfp4_into, ++ vortex_quantize_nvfp4, ++) ++from profile_nvfp4_linear import module_for_name, representative_inputs ++from validate_cute_nvfp4_h3 import ( ++ fp4_tensor, ++ load_cutlass_example, ++ output_tensor, ++ scale_tensor, ++) ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--cutlass-example", type=Path, required=True) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument( ++ "--linear", ++ choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), ++ required=True, ++ ) ++ parser.add_argument("--warmup", type=int, default=5) ++ parser.add_argument("--iterations", type=int, default=20) ++ parser.add_argument("--rows", type=int, default=128) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def measure_cuda(fn, *, warmup: int, iterations: int) -> tuple[object, float]: ++ result = None ++ for _ in range(warmup): ++ result = fn() ++ torch.cuda.synchronize() ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ for _ in range(iterations): ++ result = fn() ++ finished.record() ++ finished.synchronize() ++ return result, started.elapsed_time(finished) / iterations ++ ++ ++def main() -> None: ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ import comfy_kitchen as ck ++ ++ args = parse_args() ++ if args.warmup < 0 or args.iterations <= 0: ++ raise ValueError("--warmup must be non-negative and --iterations must be positive") ++ if args.rows <= 0 or args.rows % 128: ++ raise ValueError("--rows must be a positive multiple of 128") ++ ++ example = load_cutlass_example(args.cutlass_example, fuse_alpha=True) ++ block, inputs, metadata = representative_inputs(args) ++ linear = module_for_name(block, args.linear) ++ full_activation = inputs[args.linear].reshape(-1, linear.in_features).contiguous() ++ if args.rows > full_activation.shape[0]: ++ raise ValueError(f"--rows exceeds the available {full_activation.shape[0]} rows") ++ activation = full_activation[:args.rows].contiguous() ++ if activation.shape[1] % 128: ++ raise ValueError(f"Ring prototype requires K divisible by 128, got {activation.shape}") ++ global_scale = nvfp4_activation_scale(full_activation).float() ++ ++ with torch.inference_mode(): ++ expected_packed = vortex_quantize_nvfp4(activation, scale=global_scale) ++ ring_packed = vortex_native_quantize_nvfp4(activation, scale=global_scale) ++ packed_weight = linear._packed_weight() ++ expected_qdata, expected_tensor_scale, expected_block_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(expected_packed) ++ ) ++ ring_qdata, ring_tensor_scale, ring_block_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(ring_packed) ++ ) ++ b_qdata, tensor_scale_b, b_block_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) ++ ) ++ reference = functional.linear(expected_packed, packed_weight, None) ++ ++ a, _ = fp4_tensor(ring_qdata, swap_nibbles=False, reencode=True) ++ b, _ = fp4_tensor(b_qdata, swap_nibbles=False, reencode=True) ++ sfa = scale_tensor(ring_block_scales) ++ sfb = scale_tensor(b_block_scales) ++ output_bf16 = torch.zeros( ++ args.rows, b_qdata.shape[0], device="cuda", dtype=torch.bfloat16, ++ ) ++ c = output_tensor(output_bf16) ++ alpha = ring_tensor_scale.float() * tensor_scale_b.float() ++ alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) ++ gemm = example.Sm120BlockScaledGemmKernel( ++ cutlass.Float32, 16, (128, 128, 128), (128, 128), ++ ) ++ max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) ++ stream = cutlass_torch.default_stream() ++ compiled_gemm = cute.compile( ++ gemm, a, b, sfa, sfb, c, alpha_argument, max_active_clusters, stream, ++ ) ++ ++ def run_consumer(): ++ return compiled_gemm(a, b, sfa, sfb, c, alpha_argument, stream) ++ ++ def run_pack_into(): ++ return vortex_native_quantize_nvfp4_into( ++ activation, global_scale, ring_qdata, ring_block_scales, ++ ) ++ ++ def run_cute_ring(): ++ run_pack_into() ++ return run_consumer() ++ ++ def run_comfy_consumer(): ++ return ck.scaled_mm_nvfp4( ++ ring_qdata, ++ b_qdata, ++ tensor_scale_a=ring_tensor_scale, ++ tensor_scale_b=tensor_scale_b, ++ block_scale_a=ring_block_scales, ++ block_scale_b=b_block_scales, ++ out_dtype=torch.bfloat16, ++ alpha=ring_tensor_scale.float() * tensor_scale_b.float(), ++ ) ++ ++ def run_comfy_ring(): ++ run_pack_into() ++ return run_comfy_consumer() ++ ++ run_consumer() ++ torch.cuda.synchronize() ++ candidate = output_bf16[:, : linear.out_features] ++ delta = candidate.float() - reference.float() ++ ++ _, scale_ms = measure_cuda( ++ lambda: nvfp4_activation_scale(full_activation), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ _, producer_ms = measure_cuda( ++ run_pack_into, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ _, consumer_ms = measure_cuda( ++ run_consumer, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ _, actual_ring_ms = measure_cuda( ++ run_cute_ring, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ comfy_candidate, comfy_consumer_ms = measure_cuda( ++ run_comfy_consumer, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ _, comfy_ring_ms = measure_cuda( ++ run_comfy_ring, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ _, reference_ms = measure_cuda( ++ lambda: functional.linear( ++ vortex_quantize_nvfp4(activation, scale=global_scale), ++ packed_weight, ++ None, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ ++ qdata_bytes = ring_qdata.numel() * ring_qdata.element_size() ++ sfa_bytes = ring_block_scales.numel() * ring_block_scales.element_size() ++ chunk_count = math.ceil(full_activation.shape[0] / args.rows) ++ modeled_cute_chunk_ms = producer_ms + consumer_ms ++ modeled_comfy_chunk_ms = producer_ms + comfy_consumer_ms ++ modeled_reference_canonical_ms = scale_ms + chunk_count * reference_ms ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "cutlass_dsl": "4.6.2", ++ "metadata": metadata, ++ "linear": args.linear, ++ "mnk": [args.rows, linear.out_features, activation.shape[1]], ++ "full_activation_rows": full_activation.shape[0], ++ "modeled_chunk_count": chunk_count, ++ "ring": { ++ "row_capacity": args.rows, ++ "producer": "vortex_native_quantize_nvfp4", ++ "qdata_bytes": qdata_bytes, ++ "sfa_bytes": sfa_bytes, ++ "logical_bytes": qdata_bytes + sfa_bytes, ++ }, ++ "parity": { ++ "tensor_scale_equal": torch.equal( ++ ring_tensor_scale, expected_tensor_scale, ++ ), ++ "fp4_difference_count": int( ++ (ring_qdata != expected_qdata).sum().item() ++ ), ++ "block_scale_difference_count": int( ++ (ring_block_scales.view(torch.uint8) ++ != expected_block_scales.view(torch.uint8)).sum().item() ++ ), ++ "output_equal": torch.equal(candidate, reference), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ "comfy_output_equal": torch.equal( ++ comfy_candidate[: args.rows, : linear.out_features], reference, ++ ), ++ }, ++ "timing": { ++ "warmup": args.warmup, ++ "iterations": args.iterations, ++ "producer_ms": producer_ms, ++ "global_scale_ms": scale_ms, ++ "cute_consumer_ms": consumer_ms, ++ "modeled_cute_chunk_ms": modeled_cute_chunk_ms, ++ "modeled_comfy_chunk_ms": modeled_comfy_chunk_ms, ++ "actual_into_ring_cute_gemm_ms": actual_ring_ms, ++ "comfy_consumer_ms": comfy_consumer_ms, ++ "actual_into_ring_comfy_gemm_ms": comfy_ring_ms, ++ "reference_vortex_scale_comfy_pack_gemm_ms": reference_ms, ++ "modeled_canonical_reference_ms": modeled_reference_canonical_ms, ++ "modeled_canonical_cute_ring_ms": scale_ms + chunk_count * actual_ring_ms, ++ "modeled_canonical_comfy_ring_ms": scale_ms + chunk_count * comfy_ring_ms, ++ "actual_ring_vs_reference": actual_ring_ms / reference_ms, ++ "comfy_ring_vs_reference": comfy_ring_ms / reference_ms, ++ "modeled_canonical_cute_ring_vs_reference": ( ++ scale_ms + chunk_count * actual_ring_ms ++ ) / modeled_reference_canonical_ms, ++ "modeled_canonical_comfy_ring_vs_reference": ( ++ scale_ms + chunk_count * comfy_ring_ms ++ ) / modeled_reference_canonical_ms, ++ }, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0005-real-tile-validator.patch b/research/cute_nvfp4_ring/patches/0005-real-tile-validator.patch new file mode 100644 index 0000000..18275b2 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0005-real-tile-validator.patch @@ -0,0 +1,160 @@ +diff --git a/tools/validate_cute_nvfp4_real_tiles.py b/tools/validate_cute_nvfp4_real_tiles.py +new file mode 100644 +index 0000000..be72bd2 +--- /dev/null ++++ b/tools/validate_cute_nvfp4_real_tiles.py +@@ -0,0 +1,154 @@ ++"""Compare the CuTe tile producer with every tile of a real H3 activation.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++from pathlib import Path ++ ++import cutlass ++import cutlass.cute as cute ++import cutlass.torch as cutlass_torch ++import torch ++from cutlass.cute.runtime import from_dlpack ++ ++from h3_blackwell_runtime.nvfp4_quant import vortex_quantize_nvfp4 ++from profile_nvfp4_linear import module_for_name, representative_inputs ++from validate_cute_nvfp4_tile_producer import ( ++ BLOCKS_PER_ROW, ++ JOBS, ++ TILE, ++ produce_tile, ++ unswizzle_scales, ++) ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument( ++ "--linear", ++ choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), ++ required=True, ++ ) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ args = parse_args() ++ block, inputs, metadata = representative_inputs(args) ++ linear = module_for_name(block, args.linear) ++ activation = inputs[args.linear].reshape(-1, linear.in_features)[:TILE].contiguous() ++ if activation.shape != (TILE, linear.in_features) or activation.shape[1] % TILE: ++ raise ValueError(f"Expected a 128-row activation with K divisible by 128, got {activation.shape}") ++ ++ packed = vortex_quantize_nvfp4(activation) ++ expected_qdata, tensor_scale, expected_physical_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(packed) ++ ) ++ expected_fp4 = ((expected_qdata & 0x0F) << 4) | ((expected_qdata & 0xF0) >> 4) ++ expected_scales = unswizzle_scales(expected_physical_scales.view(torch.uint8)) ++ ++ source_torch = torch.empty(TILE, TILE, device="cuda", dtype=torch.bfloat16) ++ source = from_dlpack(source_torch, assumed_align=16).mark_layout_dynamic(leading_dim=1) ++ scale_torch = tensor_scale.float().reshape(1).contiguous() ++ scale = from_dlpack(scale_torch, assumed_align=4).mark_layout_dynamic() ++ fp4, fp4_torch = cutlass_torch.cute_tensor_like( ++ torch.zeros_like(source_torch, dtype=torch.float32), ++ cutlass.Float4E2M1FN, ++ is_dynamic_layout=True, ++ assumed_align=16, ++ ) ++ block_scales_torch = torch.zeros(JOBS, 8, device="cuda", dtype=torch.uint8) ++ block_scales = from_dlpack(block_scales_torch.flatten(), assumed_align=16) ++ block_scales.element_type = cutlass.Float8E4M3FN ++ block_scales = block_scales.mark_layout_dynamic() ++ scalar_scales_torch = torch.zeros(JOBS, device="cuda", dtype=torch.uint8) ++ scalar_scales = from_dlpack(scalar_scales_torch, assumed_align=16) ++ scalar_scales.element_type = cutlass.Float8E4M3FN ++ scalar_scales = scalar_scales.mark_layout_dynamic() ++ ++ compiled = cute.compile(produce_tile, source, scale, fp4, block_scales, scalar_scales) ++ tile_reports = [] ++ examples = [] ++ total_fp4_differences = 0 ++ total_scale_differences = 0 ++ for k_start in range(0, activation.shape[1], TILE): ++ source_torch.copy_(activation[:, k_start : k_start + TILE]) ++ compiled(source, scale, fp4, block_scales, scalar_scales) ++ torch.cuda.synchronize() ++ actual_fp4 = fp4_torch.view(torch.uint8).flatten()[: TILE * TILE // 2].reshape(TILE, TILE // 2) ++ actual_scales = scalar_scales_torch.reshape(TILE, BLOCKS_PER_ROW) ++ fp4_reference = expected_fp4[:, k_start // 2 : (k_start + TILE) // 2] ++ scale_reference = expected_scales[:, k_start // 16 : (k_start + TILE) // 16] ++ fp4_differences = int((actual_fp4 != fp4_reference).sum().item()) ++ scale_differences = int((actual_scales != scale_reference).sum().item()) ++ total_fp4_differences += fp4_differences ++ total_scale_differences += scale_differences ++ if fp4_differences or scale_differences: ++ tile_reports.append({ ++ "k_start": k_start, ++ "fp4_difference_count": fp4_differences, ++ "block_scale_difference_count": scale_differences, ++ }) ++ if fp4_differences and len(examples) < 20: ++ for row, packed_column in (actual_fp4 != fp4_reference).nonzero().tolist(): ++ global_column = k_start + packed_column * 2 ++ block_scale_byte = scale_reference[row, packed_column // 8].reshape(1) ++ decoded_scale = block_scale_byte.view(torch.float8_e4m3fn).float() ++ encode_scale = torch.minimum( ++ torch.ones_like(decoded_scale) / (decoded_scale * scale_torch), ++ torch.full_like(decoded_scale, torch.finfo(torch.float32).max), ++ ) ++ normalized = activation[row, global_column : global_column + 2].float() * encode_scale ++ examples.append({ ++ "row": row, ++ "global_column": global_column, ++ "source": [ ++ float(activation[row, global_column].float().item()), ++ float(activation[row, global_column + 1].float().item()), ++ ], ++ "actual_byte": int(actual_fp4[row, packed_column].item()), ++ "expected_byte": int(fp4_reference[row, packed_column].item()), ++ "block_scale_byte": int(block_scale_byte.item()), ++ "decoded_block_scale": float(decoded_scale.item()), ++ "encode_scale": float(encode_scale.item()), ++ "torch_normalized": normalized.tolist(), ++ }) ++ if len(examples) == 20: ++ break ++ ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "cutlass_dsl": "4.6.2", ++ "metadata": metadata, ++ "linear": args.linear, ++ "activation_shape": list(activation.shape), ++ "tensor_scale": scale_torch.item(), ++ "tile_count": activation.shape[1] // TILE, ++ "fp4_difference_count": total_fp4_differences, ++ "block_scale_difference_count": total_scale_differences, ++ "equal": total_fp4_differences == 0 and total_scale_differences == 0, ++ "differing_tiles": tile_reports, ++ "difference_examples": examples, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0006-h3-gemm-validator.patch b/research/cute_nvfp4_ring/patches/0006-h3-gemm-validator.patch new file mode 100644 index 0000000..5638b76 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0006-h3-gemm-validator.patch @@ -0,0 +1,543 @@ +diff --git a/tools/validate_cute_nvfp4_h3.py b/tools/validate_cute_nvfp4_h3.py +new file mode 100644 +index 0000000..f7a3aa3 +--- /dev/null ++++ b/tools/validate_cute_nvfp4_h3.py +@@ -0,0 +1,537 @@ ++"""Compare the CUTLASS DSL SM121 block-scaled GEMM with real H3 NVFP4 tensors.""" ++ ++from __future__ import annotations ++ ++import argparse ++import importlib.util ++import json ++import sys ++from pathlib import Path ++ ++import torch ++import torch.nn.functional as functional ++ ++from h3_blackwell_runtime.nvfp4_quant import vortex_quantize_nvfp4 ++from profile_nvfp4_linear import module_for_name, representative_inputs ++ ++ ++def _replace_once(source: str, old: str, new: str, name: str) -> str: ++ if source.count(old) != 1: ++ raise RuntimeError( ++ f"CUTLASS {name} patch expected one occurrence, found {source.count(old)}: {old!r}" ++ ) ++ return source.replace(old, new) ++ ++ ++def _patch_streaming_a(source: str) -> str: ++ source = _replace_once( ++ source, ++ "import cutlass\nimport cutlass.cute as cute", ++ """import cutlass ++import cutlass.cute as cute ++from cutlass import Float32 ++from cutlass.cutlass_dsl import dsl_user_op ++from cutlass._mlir import ir ++from cutlass._mlir.dialects import llvm ++ ++ ++@dsl_user_op ++def vortex_rcp_approx_ftz_f32( ++ x: Float32, ++ *, ++ loc: Optional[ir.Location] = None, ++ ip: Optional[ir.InsertionPoint] = None, ++) -> Float32: ++ result = llvm.inline_asm( ++ Float32.mlir_type, ++ [x.ir_value(loc=loc, ip=ip)], ++ "rcp.approx.ftz.f32 $0, $1;", ++ "=f,f", ++ has_side_effects=False, ++ asm_dialect=0, ++ loc=loc, ++ ip=ip, ++ ) ++ return Float32(result)""", ++ "streaming-A reciprocal", ++ ) ++ replacements = ( ++ ( ++ " self.a_dtype = a.element_type\n self.b_dtype = b.element_type\n self.c_dtype = c.element_type\n self.sf_dtype = sfa.element_type", ++ " self.a_source_dtype = a.element_type\n self.a_dtype = cutlass.Float4E2M1FN\n self.b_dtype = b.element_type\n self.c_dtype = c.element_type\n self.sf_dtype = cutlass.Float8E4M3FN", ++ ), ++ ( ++ " self.sfa_layout = blockscaled_utils.tile_atom_to_shape_SF(\n a.shape, self.sf_vec_size\n )\n sfa_tensor = cute.make_tensor(sfa.iterator, self.sfa_layout)\n", ++ "", ++ ), ++ ( ++ " tma_atom_a, tma_tensor_a = self._make_tma_atoms_and_tensors(\n a,\n self.a_smem_layout_staged,\n (self.tile_shape_mnk[0], self.tile_shape_mnk[2]),\n 1,\n internal_type=self.tma_internal_a_dtype,\n )\n\n", ++ "", ++ ), ++ ( ++ " tma_atom_sfa, tma_tensor_sfa = self._make_tma_atoms_and_tensors(\n sfa_tensor,\n self.sfa_smem_layout_staged,\n (self.tile_shape_mnk[0], self.tile_shape_mnk[2]),\n 1,\n internal_type=cutlass.Int16,\n )\n\n", ++ "", ++ ), ++ ( ++ " self.kernel(\n tma_atom_a,\n tma_tensor_a,\n tma_atom_b,\n tma_tensor_b,\n tma_atom_sfa,\n tma_tensor_sfa,\n tma_atom_sfb,", ++ " self.kernel(\n a,\n sfa,\n tma_atom_b,\n tma_tensor_b,\n tma_atom_sfb,", ++ ), ++ ( ++ " tma_atom_a: cute.CopyAtom,\n mA_mkl: cute.Tensor,\n tma_atom_b: cute.CopyAtom,\n mB_nkl: cute.Tensor,\n tma_atom_sfa: cute.CopyAtom,\n mSFA_mkl: cute.Tensor,\n tma_atom_sfb: cute.CopyAtom,", ++ " mA_mkl: cute.Tensor,\n tensor_scale_a: cute.Tensor,\n tma_atom_b: cute.CopyAtom,\n mB_nkl: cute.Tensor,\n tma_atom_sfb: cute.CopyAtom,", ++ ), ++ ( ++ " cpasync.prefetch_descriptor(tma_atom_a)\n cpasync.prefetch_descriptor(tma_atom_b)\n cpasync.prefetch_descriptor(tma_atom_sfa)\n cpasync.prefetch_descriptor(tma_atom_sfb)", ++ " cpasync.prefetch_descriptor(tma_atom_b)\n cpasync.prefetch_descriptor(tma_atom_sfb)", ++ ), ++ ( ++ " tma_copy_bytes = (\n cute.size_in_bytes(self.a_dtype, a_smem_layout)\n + cute.size_in_bytes(self.b_dtype, b_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfa_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfb_smem_layout)\n )", ++ " tma_copy_bytes = (\n cute.size_in_bytes(self.b_dtype, b_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfb_smem_layout)\n )", ++ ), ++ ( ++ " # (tM, tK, loopM, loopK, loopL)\n gSFA_mkl = cute.local_tile(\n mSFA_mkl,\n cute.slice_(self.tile_shape_mnk, (None, 0, None)),\n (None, None, None),\n )\n", ++ "", ++ ), ++ ( ++ " # TMA load A partition_S/D\n a_cta_layout = cute.make_layout(cute.slice_(cta_layout_mnk, (0, None, 0)).shape)\n a_cta_crd = cluster_coord_mnk[1]\n tAsA, tAgA = cpasync.tma_partition(\n tma_atom_a,\n a_cta_crd,\n a_cta_layout,\n cute.group_modes(sA, 0, 2),\n cute.group_modes(gA_mkl, 0, 2),\n )\n\n", ++ "", ++ ), ++ ( ++ " tAsSFA, tAgSFA = cpasync.tma_partition(\n tma_atom_sfa,\n a_cta_crd,\n a_cta_layout,\n cute.group_modes(sSFA, 0, 2),\n cute.group_modes(gSFA_mkl, 0, 2),\n )\n tAsSFA = cute.filter_zeros(tAsSFA)\n tAgSFA = cute.filter_zeros(tAgSFA)\n\n", ++ "", ++ ), ++ ( ++ " cute.arch.setmaxregister_decrease(self.load_register_requirement)\n\n while work_tile.is_valid_tile:", ++ """ cute.arch.setmaxregister_decrease(self.load_register_requirement) ++ producer_lane = tidx - self.tma_load_warp_id * self.num_threads_per_warp ++ fp4_store = cute.make_copy_atom( ++ cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN ++ ) ++ source_fragment = cute.make_rmem_tensor((16,), cutlass.Float32) ++ normalized_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) ++ fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) ++ scale_source = cute.make_rmem_tensor((8,), cutlass.Float32) ++ scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) ++ decoded_scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) ++ while work_tile.is_valid_tile:""", ++ ), ++ ( ++ " tAgA_mkl = tAgA[(None, tile_coord_mnl[0], None, tile_coord_mnl[2])]\n tBgB_nkl = tBgB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]\n tAgSFA_mkl = tAgSFA[(None, tile_coord_mnl[0], None, tile_coord_mnl[2])]\n tBgSFB_nkl = tBgSFB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]", ++ " tBgB_nkl = tBgB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]\n tBgSFB_nkl = tBgSFB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]", ++ ), ++ ( ++ " tAgA_k = tAgA_mkl[(None, mainloop_producer_state.count)]\n tAsA_pipe = tAsA[(None, mainloop_producer_state.index)]\n\n tBgB_k = tBgB_nkl[(None, mainloop_producer_state.count)]\n tBsB_pipe = tBsB[(None, mainloop_producer_state.index)]\n\n tAgSFA_k = tAgSFA_mkl[(None, mainloop_producer_state.count)]\n tAsSFA_pipe = tAsSFA[(None, mainloop_producer_state.index)]\n\n tBgSFB_k = tBgSFB_nkl[(None, mainloop_producer_state.count)]", ++ " tBgB_k = tBgB_nkl[(None, mainloop_producer_state.count)]\n tBsB_pipe = tBsB[(None, mainloop_producer_state.index)]\n\n tBgSFB_k = tBgSFB_nkl[(None, mainloop_producer_state.count)]", ++ ), ++ ( ++ """ cute.copy( ++ tma_atom_a, ++ tAgA_k, ++ tAsA_pipe, ++ tma_bar_ptr=mainloop_pipeline.producer_get_barrier( ++ mainloop_producer_state ++ ), ++ ) ++""", ++ """ scale = tensor_scale_a[0] ++ stage = mainloop_producer_state.index ++ source_k_base = mainloop_producer_state.count * self.tile_shape_mnk[2] ++ source_m_base = tile_coord_mnl[0] * self.tile_shape_mnk[0] ++ for row_group in cutlass.range_constexpr(4): ++ row = producer_lane + row_group * self.num_threads_per_warp ++ source_row = source_m_base + row ++ sA_row = sA[(row, None, stage)] ++ sA_tiles = cute.zipped_divide(sA_row, (8,)) ++ for block_column in cutlass.range_constexpr(8): ++ source_column = source_k_base + block_column * 16 ++ maximum = cutlass.Float32(0.0) ++ for element in cutlass.range_constexpr(16): ++ value = mA_mkl[ ++ source_row, ++ source_column + element, ++ tile_coord_mnl[2], ++ ] ++ source_fragment[element] = value ++ maximum = cutlass.max( ++ cutlass.max(value, -value), maximum ++ ) ++ ++ raw_block_scale = ( ++ maximum / cutlass.Float32(6.0) ++ ) / scale ++ for element in cutlass.range_constexpr(8): ++ scale_source[element] = raw_block_scale ++ scale_values = scale_source.load() ++ scale_values = cute.where( ++ scale_values <= cutlass.Float32(448.0), ++ scale_values, ++ cutlass.Float32(448.0), ++ ) ++ scale_fragment.store( ++ scale_values.to(cutlass.Float8E4M3FN) ++ ) ++ sSFA[row, block_column * 16, stage] = scale_fragment[0] ++ decoded_scale_fragment.store( ++ scale_fragment.load().to(cutlass.Float32) ++ ) ++ raw_encode_scale = vortex_rcp_approx_ftz_f32( ++ decoded_scale_fragment[0] * scale ++ ) ++ for element in cutlass.range_constexpr(8): ++ scale_source[element] = raw_encode_scale ++ encode_scale_values = scale_source.load() ++ encode_scale_values = cute.where( ++ encode_scale_values ++ <= cutlass.Float32(3.402823466e38), ++ encode_scale_values, ++ cutlass.Float32(3.402823466e38), ++ ) ++ scale_source.store(encode_scale_values) ++ encode_scale = scale_source[0] ++ ++ for half in cutlass.range_constexpr(2): ++ for element in cutlass.range_constexpr(8): ++ normalized_fragment[element] = source_fragment[ ++ half * 8 + element ++ ] * encode_scale ++ fp4_fragment.store( ++ normalized_fragment.load().to( ++ cutlass.Float4E2M1FN ++ ) ++ ) ++ cute.copy( ++ fp4_store, ++ fp4_fragment, ++ sA_tiles[(None, block_column * 2 + half)], ++ ) ++ cute.arch.fence_proxy("async.shared", space="cta") ++""", ++ ), ++ ( ++ """ cute.copy( ++ tma_atom_sfa, ++ tAgSFA_k, ++ tAsSFA_pipe, ++ tma_bar_ptr=mainloop_pipeline.producer_get_barrier( ++ mainloop_producer_state ++ ), ++ ) ++""", ++ "", ++ ), ++ ) ++ for index, (old, new) in enumerate(replacements): ++ source = _replace_once(source, old, new, f"streaming-A[{index}]") ++ return source ++ ++ ++def load_cutlass_example(path: Path, *, fuse_alpha: bool, stream_a: bool = False): ++ if not path.is_file(): ++ raise FileNotFoundError(f"CUTLASS DSL example not found: {path}") ++ sys.path.insert(0, str(path.parent)) ++ source = path.read_text(encoding="utf-8") ++ if stream_a: ++ source = _patch_streaming_a(source) ++ if fuse_alpha: ++ replacements = ( ++ (" c: cute.Tensor,\n max_active_clusters: cutlass.Constexpr,", " c: cute.Tensor,\n alpha: cute.Tensor,\n max_active_clusters: cutlass.Constexpr,"), ++ (" mC_mnl: cute.Tensor,\n tiled_mma: cute.TiledMma,", " mC_mnl: cute.Tensor,\n alpha: cute.Tensor,\n tiled_mma: cute.TiledMma,"), ++ (" tma_tensor_c,\n self.tiled_mma,", " tma_tensor_c,\n alpha,\n self.tiled_mma,"), ++ (" tRS_rD_out.store(acc_vec.to(self.c_dtype))", " tRS_rD_out.store((acc_vec * alpha[0]).to(self.c_dtype))"), ++ ) ++ for old, new in replacements: ++ source = _replace_once(source, old, new, "alpha") ++ if stream_a or fuse_alpha: ++ suffix = "_vortex_stream_a" if stream_a else "_vortex" ++ suffix += "_alpha" if fuse_alpha else "" ++ load_path = path.with_name(f"{path.stem}{suffix}.py") ++ load_path.write_text(source, encoding="utf-8") ++ else: ++ load_path = path ++ spec = importlib.util.spec_from_file_location("vortex_cutlass_blockscaled", load_path) ++ if spec is None or spec.loader is None: ++ raise ImportError(f"Cannot load CUTLASS DSL example: {load_path}") ++ module = importlib.util.module_from_spec(spec) ++ spec.loader.exec_module(module) ++ return module ++ ++ ++def decode_comfy_fp4(storage: torch.Tensor) -> torch.Tensor: ++ lookup = torch.tensor( ++ [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], ++ device=storage.device, ++ dtype=torch.float32, ++ ) ++ codes = torch.stack((storage >> 4, storage & 0x0F), dim=-1).reshape(storage.shape[0], -1) ++ return lookup[codes.long()] ++ ++ ++def fp4_tensor(storage: torch.Tensor, *, swap_nibbles: bool, reencode: bool): ++ import cutlass ++ import cutlass.torch as cutlass_torch ++ from cutlass.cute.runtime import from_dlpack ++ ++ rows, packed_columns = storage.shape ++ if reencode: ++ logical = decode_comfy_fp4(storage).unsqueeze(-1) ++ return cutlass_torch.cute_tensor_like( ++ logical, cutlass.Float4E2M1FN, is_dynamic_layout=True, assumed_align=16, ++ ) ++ if swap_nibbles: ++ storage = ((storage & 0x0F) << 4) | ((storage & 0xF0) >> 4) ++ # DLPack cannot export Torch's packed FP4 dtype yet. Build the desired ++ # logical layout from an oversized uint8 allocation, then reinterpret its ++ # iterator as FP4 and populate only the packed storage that the layout uses. ++ backing = torch.empty( ++ (rows, packed_columns * 2, 1), device=storage.device, dtype=torch.uint8, ++ ) ++ backing.zero_() ++ backing[:, :packed_columns, 0].copy_(storage) ++ tensor = from_dlpack(backing, assumed_align=16) ++ tensor.element_type = cutlass.Float4E2M1FN ++ tensor = tensor.mark_layout_dynamic(leading_dim=1) ++ return tensor, backing ++ ++ ++def output_tensor(storage: torch.Tensor): ++ from cutlass.cute.runtime import from_dlpack ++ ++ tensor = from_dlpack(storage.unsqueeze(-1), assumed_align=16) ++ tensor = tensor.mark_compact_shape_dynamic(mode=1, stride_order=(2, 0, 1), divisibility=1) ++ return tensor ++ ++ ++def scale_tensor(storage: torch.Tensor): ++ import cutlass ++ from cutlass.cute.runtime import from_dlpack ++ ++ tensor = from_dlpack(storage.view(torch.uint8).unsqueeze(-1), assumed_align=16) ++ tensor.element_type = cutlass.Float8E4M3FN ++ return tensor.mark_layout_dynamic(leading_dim=1) ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--cutlass-example", type=Path, required=True) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--linear", choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1", "mlp_fc2"), default="attn_qkv_proj") ++ parser.add_argument("--rows", type=int, default=128) ++ parser.add_argument("--tile-k", type=int, choices=(128, 256), default=128) ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument("--device", default="cuda") ++ parser.add_argument("--swap-nibbles", action="store_true") ++ parser.add_argument("--reencode-fp4", action="store_true") ++ parser.add_argument("--zero-a", action="store_true") ++ parser.add_argument("--fuse-alpha", action="store_true") ++ parser.add_argument("--stream-a", action="store_true") ++ parser.add_argument("--warmup", type=int, default=0) ++ parser.add_argument("--iterations", type=int, default=0) ++ parser.add_argument("--benchmark-reference", action="store_true") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ if args.rows <= 0 or args.rows % 128: ++ raise ValueError("--rows must be a positive multiple of 128") ++ if args.stream_a and args.linear == "mlp_fc2": ++ raise ValueError("--stream-a deliberately excludes mlp_fc2; retain the reference cuBLAS path") ++ if args.warmup < 0 or args.iterations < 0: ++ raise ValueError("--warmup and --iterations must be non-negative") ++ ++ import cutlass ++ import cutlass.cute as cute ++ import cutlass.torch as cutlass_torch ++ import comfy_kitchen as ck ++ from cutlass.cute.runtime import from_dlpack ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ example = load_cutlass_example( ++ args.cutlass_example, fuse_alpha=args.fuse_alpha, stream_a=args.stream_a, ++ ) ++ block, inputs, metadata = representative_inputs(args) ++ linear = module_for_name(block, args.linear) ++ activation = inputs[args.linear].reshape(-1, linear.in_features)[:args.rows].contiguous() ++ ++ with torch.inference_mode(): ++ packed_activation = vortex_quantize_nvfp4(activation) ++ packed_weight = linear._packed_weight() ++ a_qdata, a_tensor_scale, a_block_scale = TensorCoreNVFP4Layout.get_plain_tensors(packed_activation) ++ b_qdata, b_tensor_scale, b_block_scale = TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) ++ reference = functional.linear(packed_activation, packed_weight, None)[:args.rows, :linear.out_features] ++ raw_reference = ck.scaled_mm_nvfp4( ++ a_qdata, ++ b_qdata, ++ tensor_scale_a=a_tensor_scale, ++ tensor_scale_b=b_tensor_scale, ++ block_scale_a=a_block_scale, ++ block_scale_b=b_block_scale, ++ out_dtype=torch.bfloat16, ++ alpha=torch.ones(1, device=activation.device, dtype=torch.float32), ++ )[:args.rows, :linear.out_features] ++ if args.zero_a: ++ a_qdata = torch.zeros_like(a_qdata) ++ ++ output_bf16 = torch.zeros((a_qdata.shape[0], b_qdata.shape[0]), device=activation.device, dtype=torch.bfloat16) ++ if args.stream_a: ++ if args.rows != 128 or activation.shape[1] % 128: ++ raise ValueError("--stream-a currently requires exactly 128 rows and K divisible by 128") ++ a = output_tensor(activation) ++ a_backing = activation ++ else: ++ a, a_backing = fp4_tensor(a_qdata, swap_nibbles=args.swap_nibbles, reencode=args.reencode_fp4) ++ b, b_backing = fp4_tensor(b_qdata, swap_nibbles=args.swap_nibbles, reencode=args.reencode_fp4) ++ sfa = ( ++ from_dlpack(a_tensor_scale.float().reshape(1).contiguous(), assumed_align=4) ++ if args.stream_a ++ else scale_tensor(a_block_scale) ++ ) ++ sfb = scale_tensor(b_block_scale) ++ c = output_tensor(output_bf16) ++ ++ gemm = example.Sm120BlockScaledGemmKernel( ++ cutlass.Float32, ++ 16, ++ (128, 128, args.tile_k), ++ (128, 128), ++ ) ++ hardware_info = cutlass.utils.HardwareInfo() ++ max_active_clusters = hardware_info.get_max_active_clusters(1) ++ stream = cutlass_torch.default_stream() ++ alpha = a_tensor_scale.float() * b_tensor_scale.float() ++ alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) ++ if args.fuse_alpha: ++ compiled = cute.compile(gemm, a, b, sfa, sfb, c, alpha_argument, max_active_clusters, stream) ++ arguments = (a, b, sfa, sfb, c, alpha_argument, stream) ++ else: ++ compiled = cute.compile(gemm, a, b, sfa, sfb, c, max_active_clusters, stream) ++ arguments = (a, b, sfa, sfb, c, stream) ++ compiled(*arguments) ++ torch.cuda.synchronize() ++ ++ timing = None ++ if args.iterations: ++ for _ in range(args.warmup): ++ compiled(*arguments) ++ torch.cuda.synchronize() ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ for _ in range(args.iterations): ++ compiled(*arguments) ++ finished.record() ++ finished.synchronize() ++ total_ms = started.elapsed_time(finished) ++ timing = { ++ "warmup": args.warmup, ++ "iterations": args.iterations, ++ "total_ms": total_ms, ++ "mean_ms": total_ms / args.iterations, ++ } ++ ++ reference_timing = None ++ if args.benchmark_reference: ++ if not args.iterations: ++ raise ValueError("--benchmark-reference requires --iterations") ++ ++ def measure_cuda(fn): ++ result = None ++ for _ in range(args.warmup): ++ result = fn() ++ torch.cuda.synchronize() ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ for _ in range(args.iterations): ++ result = fn() ++ finished.record() ++ finished.synchronize() ++ total = started.elapsed_time(finished) ++ return result, total / args.iterations ++ ++ _, quantize_ms = measure_cuda(lambda: vortex_quantize_nvfp4(activation)) ++ _, complete_ms = measure_cuda( ++ lambda: functional.linear( ++ vortex_quantize_nvfp4(activation), packed_weight, None, ++ ) ++ ) ++ reference_timing = { ++ "backend": "vortex_scale_plus_comfy_pack_gemm", ++ "activation_quantize_mean_ms": quantize_ms, ++ "complete_projection_mean_ms": complete_ms, ++ } ++ ++ candidate = ( ++ output_bf16[:args.rows, :linear.out_features] ++ if args.fuse_alpha ++ else (output_bf16[:args.rows, :linear.out_features].float() * alpha).to(reference.dtype) ++ ) ++ raw_candidate = output_bf16[:args.rows, :linear.out_features] ++ raw_delta = raw_candidate.float() - raw_reference.float() ++ delta = candidate.float() - reference.float() ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "torch": torch.__version__, ++ "cutlass_dsl": "4.6.2", ++ "metadata": metadata, ++ "linear": args.linear, ++ "rows": args.rows, ++ "mnk": [args.rows, linear.out_features, linear.in_features], ++ "tile_shape_mnk": [128, 128, args.tile_k], ++ "swap_nibbles": args.swap_nibbles, ++ "reencode_fp4": args.reencode_fp4, ++ "zero_a": args.zero_a, ++ "fuse_alpha": args.fuse_alpha, ++ "stream_a": args.stream_a, ++ "streamed_activation_materialization": ( ++ {"global_qdata": False, "global_sfa": False} ++ if args.stream_a ++ else None ++ ), ++ "timing": timing, ++ "reference_timing": reference_timing, ++ "cute_shapes": {"a": str(a.shape), "b": str(b.shape), "sfa": str(sfa.shape), "sfb": str(sfb.shape), "c": str(c.shape)}, ++ "tensor_scales": {"a": a_tensor_scale.float().item(), "b": b_tensor_scale.float().item(), "alpha": alpha.item()}, ++ "raw_output": { ++ "dtype": str(output_bf16.dtype), ++ "checksum": output_bf16.float().sum().item(), ++ "max_abs": output_bf16.float().abs().max().item(), ++ "nonzero": int(torch.count_nonzero(output_bf16).item()), ++ "finite": bool(torch.isfinite(output_bf16).all().item()), ++ }, ++ "raw_blockscaled_parity": { ++ "applicable": not args.fuse_alpha, ++ "reference_checksum": raw_reference.float().sum().item(), ++ "candidate_checksum": raw_candidate.float().sum().item(), ++ "equal": torch.equal(raw_candidate, raw_reference), ++ "max_abs": raw_delta.abs().max().item(), ++ "mean_abs": raw_delta.abs().mean().item(), ++ }, ++ "reference_checksum": reference.float().sum().item(), ++ "candidate_checksum": candidate.float().sum().item(), ++ "equal": torch.equal(candidate, reference), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ "relative_l2": (delta.norm() / reference.float().norm().clamp_min(1e-12)).item(), ++ "numerical_note": ( ++ "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." ++ if args.fuse_alpha ++ else "The stock SM121 kernel rounds before the external global-scale product; exact H3 integration requires the fused-alpha epilogue." ++ ), ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0007-full-ring-validator.patch b/research/cute_nvfp4_ring/patches/0007-full-ring-validator.patch new file mode 100644 index 0000000..7a31765 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0007-full-ring-validator.patch @@ -0,0 +1,284 @@ +diff --git a/tools/validate_cute_nvfp4_ring_full.py b/tools/validate_cute_nvfp4_ring_full.py +new file mode 100644 +index 0000000..f8774fc +--- /dev/null ++++ b/tools/validate_cute_nvfp4_ring_full.py +@@ -0,0 +1,278 @@ ++"""Validate complete H3 projections through a reusable NVFP4 row ring.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import math ++from pathlib import Path ++ ++import cutlass ++import cutlass.cute as cute ++import cutlass.torch as cutlass_torch ++import torch ++import torch.nn.functional as functional ++from cutlass.cute.runtime import from_dlpack ++ ++from h3_blackwell_runtime.nvfp4_quant import ( ++ nvfp4_activation_scale, ++ vortex_native_quantize_nvfp4, ++ vortex_native_quantize_nvfp4_into, ++ vortex_quantize_nvfp4, ++) ++from profile_nvfp4_linear import module_for_name, representative_inputs ++from validate_cute_nvfp4_h3 import ( ++ fp4_tensor, ++ load_cutlass_example, ++ output_tensor, ++ scale_tensor, ++) ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--cutlass-example", type=Path, required=True) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument( ++ "--linears", ++ nargs="+", ++ choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), ++ default=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), ++ ) ++ parser.add_argument("--capacity", type=int, default=2048) ++ parser.add_argument("--warmup", type=int, default=1) ++ parser.add_argument("--iterations", type=int, default=3) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def measure_cuda(fn, *, warmup: int, iterations: int) -> tuple[object, float]: ++ result = None ++ for _ in range(warmup): ++ result = fn() ++ torch.cuda.synchronize() ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ for _ in range(iterations): ++ result = fn() ++ finished.record() ++ finished.synchronize() ++ return result, started.elapsed_time(finished) / iterations ++ ++ ++def validate_linear(args, example, block, inputs, name: str) -> dict: ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ linear = module_for_name(block, name) ++ activation = inputs[name].reshape(-1, linear.in_features).contiguous() ++ rows, features = activation.shape ++ global_scale = nvfp4_activation_scale(activation).float() ++ packed_weight = linear._packed_weight() ++ b_qdata, tensor_scale_b, b_block_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) ++ ) ++ ++ with torch.inference_mode(): ++ reference_packed = vortex_quantize_nvfp4( ++ activation, scale=global_scale, ++ ) ++ reference = functional.linear(reference_packed, packed_weight, None) ++ ring_seed = vortex_native_quantize_nvfp4( ++ activation[: args.capacity], scale=global_scale, ++ ) ++ _, ring_tensor_scale, ring_block_scales = ( ++ TensorCoreNVFP4Layout.get_plain_tensors(ring_seed) ++ ) ++ ++ a, a_backing = cutlass_torch.cute_tensor_like( ++ torch.zeros( ++ args.capacity, ++ features, ++ 1, ++ device="cuda", ++ dtype=torch.float32, ++ ), ++ cutlass.Float4E2M1FN, ++ is_dynamic_layout=True, ++ assumed_align=16, ++ ) ++ ring_qdata = a_backing.view(torch.uint8).flatten()[ ++ : args.capacity * features // 2 ++ ].reshape(args.capacity, features // 2) ++ b, _ = fp4_tensor(b_qdata, swap_nibbles=False, reencode=True) ++ sfa = scale_tensor(ring_block_scales) ++ sfb = scale_tensor(b_block_scales) ++ chunks = [ ++ (start, min(start + args.capacity, rows)) ++ for start in range(0, rows, args.capacity) ++ ] ++ padded_rows = len(chunks) * args.capacity ++ candidate_padded = torch.zeros( ++ padded_rows, b_qdata.shape[0], device="cuda", dtype=torch.bfloat16, ++ ) ++ c_chunks = [ ++ output_tensor( ++ candidate_padded[ ++ index * args.capacity : (index + 1) * args.capacity ++ ] ++ ) ++ for index in range(len(chunks)) ++ ] ++ alpha = ring_tensor_scale.float() * tensor_scale_b.float() ++ alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) ++ gemm = example.Sm120BlockScaledGemmKernel( ++ cutlass.Float32, 16, (128, 128, 128), (128, 128), ++ ) ++ max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) ++ stream = cutlass_torch.default_stream() ++ compiled_gemm = cute.compile( ++ gemm, a, b, sfa, sfb, c_chunks[0], alpha_argument, ++ max_active_clusters, stream, ++ ) ++ ++ def run_chunks(): ++ for index, (start, end) in enumerate(chunks): ++ vortex_native_quantize_nvfp4_into( ++ activation[start:end], ++ global_scale, ++ ring_qdata, ++ ring_block_scales, ++ hi_first=False, ++ ) ++ compiled_gemm( ++ a, b, sfa, sfb, c_chunks[index], alpha_argument, stream, ++ ) ++ return candidate_padded ++ ++ def run_complete_ring(): ++ nvfp4_activation_scale(activation) ++ return run_chunks() ++ ++ chunk_reports = [] ++ for index, (start, end) in enumerate(chunks): ++ vortex_native_quantize_nvfp4_into( ++ activation[start:end], ++ global_scale, ++ ring_qdata, ++ ring_block_scales, ++ hi_first=False, ++ ) ++ compiled_gemm( ++ a, b, sfa, sfb, c_chunks[index], alpha_argument, stream, ++ ) ++ torch.cuda.synchronize() ++ candidate = candidate_padded[start:end, : linear.out_features] ++ expected = reference[start:end] ++ delta = candidate.float() - expected.float() ++ chunk_reports.append({ ++ "start": start, ++ "rows": end - start, ++ "equal": torch.equal(candidate, expected), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ }) ++ ++ complete_candidate = candidate_padded[:rows, : linear.out_features] ++ reference_checksum = reference.float().sum().item() ++ candidate_checksum = complete_candidate.float().sum().item() ++ output_bytes = candidate_padded.numel() * candidate_padded.element_size() ++ del reference ++ del reference_packed ++ torch.cuda.empty_cache() ++ ++ _, ring_ms = measure_cuda( ++ run_complete_ring, warmup=args.warmup, iterations=args.iterations, ++ ) ++ del candidate ++ del complete_candidate ++ del c_chunks ++ del candidate_padded ++ torch.cuda.empty_cache() ++ _, reference_ms = measure_cuda( ++ lambda: functional.linear( ++ vortex_quantize_nvfp4(activation), packed_weight, None, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ ++ qdata_bytes = ring_qdata.numel() * ring_qdata.element_size() ++ sfa_bytes = ring_block_scales.numel() * ring_block_scales.element_size() ++ return { ++ "name": name, ++ "mnk": [rows, linear.out_features, features], ++ "capacity": args.capacity, ++ "chunk_count": len(chunks), ++ "final_chunk_rows": chunks[-1][1] - chunks[-1][0], ++ "ring_bytes": qdata_bytes + sfa_bytes, ++ "parity": { ++ "all_chunks_equal": all(chunk["equal"] for chunk in chunk_reports), ++ "max_abs": max(chunk["max_abs"] for chunk in chunk_reports), ++ "mean_abs_max": max(chunk["mean_abs"] for chunk in chunk_reports), ++ "reference_checksum": reference_checksum, ++ "candidate_checksum": candidate_checksum, ++ }, ++ "timing": { ++ "warmup": args.warmup, ++ "iterations": args.iterations, ++ "ring_complete_ms": ring_ms, ++ "reference_complete_ms": reference_ms, ++ "ring_vs_reference": ring_ms / reference_ms, ++ "improvement_percent": (1.0 - ring_ms / reference_ms) * 100.0, ++ }, ++ "chunks": chunk_reports, ++ "output_bytes": output_bytes, ++ } ++ ++ ++def main() -> None: ++ args = parse_args() ++ if args.capacity <= 0 or args.capacity % 128: ++ raise ValueError("--capacity must be a positive multiple of 128") ++ if args.warmup < 0 or args.iterations <= 0: ++ raise ValueError("--warmup must be non-negative and --iterations positive") ++ ++ example = load_cutlass_example(args.cutlass_example, fuse_alpha=True) ++ block, inputs, metadata = representative_inputs(args) ++ results = [ ++ validate_linear(args, example, block, inputs, name) ++ for name in args.linears ++ ] ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "cutlass_dsl": "4.6.2", ++ "metadata": metadata, ++ "block_index": args.block_index, ++ "capacity": args.capacity, ++ "all_equal": all(result["parity"]["all_chunks_equal"] for result in results), ++ "projection_reference_total_ms": sum( ++ result["timing"]["reference_complete_ms"] for result in results ++ ), ++ "projection_ring_total_ms": sum( ++ result["timing"]["ring_complete_ms"] for result in results ++ ), ++ "results": results, ++ } ++ report["projection_total_improvement_percent"] = ( ++ 1.0 ++ - report["projection_ring_total_ms"] ++ / report["projection_reference_total_ms"] ++ ) * 100.0 ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0008-qkv-block-gate.patch b/research/cute_nvfp4_ring/patches/0008-qkv-block-gate.patch new file mode 100644 index 0000000..f562b7d --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0008-qkv-block-gate.patch @@ -0,0 +1,163 @@ +diff --git a/tools/validate_cute_qkv_block.py b/tools/validate_cute_qkv_block.py +new file mode 100644 +index 0000000..84811c1 +--- /dev/null ++++ b/tools/validate_cute_qkv_block.py +@@ -0,0 +1,157 @@ ++"""Alternate baseline and CuTe-QKV execution inside one loaded H3 block.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import os ++import time ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.adaln import H3CurveAdaLN ++from h3_blackwell_runtime.block import H3DiTBlock ++from h3_blackwell_runtime.checkpoint import H3Checkpoint ++from h3_blackwell_runtime.packing import H3PromptPacker ++from h3_blackwell_runtime.rope import h3_rope_rotation ++from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas ++from h3_blackwell_runtime.t2v import random_av_latents ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--block-index", type=int, required=True) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument( ++ "--feature", choices=("qkv_ring", "modulate_fusion", "swiglu_fusion"), default="qkv_ring", ++ ) ++ parser.add_argument("--warmup", type=int, default=2) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def sync() -> None: ++ torch.cuda.synchronize() ++ ++ ++def summarize(values: list[float]) -> dict[str, float]: ++ ordered = sorted(values) ++ middle = len(ordered) // 2 ++ median = ( ++ ordered[middle] ++ if len(ordered) % 2 ++ else (ordered[middle - 1] + ordered[middle]) / 2 ++ ) ++ return { ++ "mean_s": sum(values) / len(values), ++ "p50_s": median, ++ "min_s": ordered[0], ++ "max_s": ordered[-1], ++ } ++ ++ ++def main() -> None: ++ args = parse_args() ++ torch.manual_seed(args.seed) ++ checkpoint = H3Checkpoint(args.model_path, device=args.device) ++ block = H3DiTBlock.from_checkpoint( ++ checkpoint, args.block_index, attention_backend=args.attention, ++ ).eval() ++ adaln = H3CurveAdaLN.from_checkpoint( ++ checkpoint, f"blocks.{args.block_index}.adaln_proj", ++ ).eval() ++ packer = H3PromptPacker(checkpoint) ++ video, audio, _ = random_av_latents( ++ args.width, args.height, args.frames, args.seed, device=args.device, ++ ) ++ sigmas = beta_sigmas(args.steps, device=args.device) ++ sigma = sigmas[args.sampler_step - 1] ++ native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) ++ text = torch.randn( ++ 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, ++ ) ++ hidden, timesteps, segments, positions, _, _ = packer( ++ text, video, native_audio, _model_sigma(sigma), ++ ) ++ rotation = h3_rope_rotation( ++ positions.to(args.device), ++ checkpoint.tensor("rope.inv_freq", dtype=torch.float32), ++ hidden.dtype, ++ ) ++ adaln_values = tuple(value.detach() for value in adaln(timesteps)) ++ ++ def run(enabled: bool): ++ if args.feature == "modulate_fusion": ++ block.fused_nvfp4_modulation = enabled ++ elif args.feature == "swiglu_fusion": ++ block.mlp.fused_nvfp4_swiglu = enabled ++ else: ++ if enabled: ++ os.environ["H3_CUTE_QKV_RING"] = "1" ++ else: ++ os.environ.pop("H3_CUTE_QKV_RING", None) ++ return block(hidden, rotation, *adaln_values, segments) ++ ++ with torch.inference_mode(): ++ reference = run(False) ++ candidate = run(True) ++ sync() ++ delta = candidate.float() - reference.float() ++ for _ in range(args.warmup): ++ run(False) ++ run(True) ++ sync() ++ baseline_times = [] ++ candidate_times = [] ++ last_reference = reference ++ last_candidate = candidate ++ for _ in range(args.iterations): ++ sync() ++ started = time.perf_counter() ++ last_reference = run(False) ++ sync() ++ baseline_times.append(time.perf_counter() - started) ++ ++ sync() ++ started = time.perf_counter() ++ last_candidate = run(True) ++ sync() ++ candidate_times.append(time.perf_counter() - started) ++ ++ baseline = summarize(baseline_times) ++ candidate_timing = summarize(candidate_times) ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "block_index": args.block_index, ++ "feature": args.feature, ++ "hidden_shape": list(hidden.shape), ++ "iterations": args.iterations, ++ "equal": torch.equal(reference, candidate), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ "reference_checksum": last_reference.float().sum().item(), ++ "candidate_checksum": last_candidate.float().sum().item(), ++ "baseline": baseline, ++ "candidate": candidate_timing, ++ "p50_improvement_percent": ( ++ 1.0 - candidate_timing["p50_s"] / baseline["p50_s"] ++ ) * 100.0, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/cute_nvfp4_ring/patches/0009-qkv-runtime-gate.patch b/research/cute_nvfp4_ring/patches/0009-qkv-runtime-gate.patch new file mode 100644 index 0000000..41f1da2 --- /dev/null +++ b/research/cute_nvfp4_ring/patches/0009-qkv-runtime-gate.patch @@ -0,0 +1,96 @@ +diff --git a/tools/validate_cute_qkv_runtime.py b/tools/validate_cute_qkv_runtime.py +new file mode 100644 +index 0000000..b6b2e74 +--- /dev/null ++++ b/tools/validate_cute_qkv_runtime.py +@@ -0,0 +1,90 @@ ++"""Validate the opt-in Nvfp4Linear CuTe QKV runtime dispatch.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import os ++from pathlib import Path ++ ++import torch ++ ++from profile_nvfp4_linear import representative_inputs ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--rows", type=int, default=2048) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--attention", default="sage2") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def measure(fn, warmup: int, iterations: int) -> float: ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ start = torch.cuda.Event(enable_timing=True) ++ end = torch.cuda.Event(enable_timing=True) ++ start.record() ++ for _ in range(iterations): ++ fn() ++ end.record() ++ end.synchronize() ++ return start.elapsed_time(end) / iterations ++ ++ ++def main() -> None: ++ args = parse_args() ++ block, inputs, metadata = representative_inputs(args) ++ linear = block.attention.qkv_proj ++ x = inputs["attn_qkv_proj"][: args.rows].contiguous() ++ if linear.role != "h3_attn_qkv": ++ raise RuntimeError(f"Expected h3_attn_qkv role, got {linear.role!r}") ++ ++ os.environ.pop("H3_CUTE_QKV_RING", None) ++ with torch.inference_mode(): ++ reference = linear(x) ++ os.environ["H3_CUTE_QKV_RING"] = "1" ++ with torch.inference_mode(): ++ candidate = linear(x) ++ torch.cuda.synchronize() ++ delta = candidate.float() - reference.float() ++ ++ with torch.inference_mode(): ++ ring_ms = measure(lambda: linear(x), args.warmup, args.iterations) ++ os.environ.pop("H3_CUTE_QKV_RING", None) ++ reference_ms = measure(lambda: linear(x), args.warmup, args.iterations) ++ ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "metadata": metadata, ++ "block_index": args.block_index, ++ "rows": args.rows, ++ "role": linear.role, ++ "equal": torch.equal(candidate, reference), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ "ring_ms": ring_ms, ++ "reference_ms": reference_ms, ++ "improvement_percent": (1.0 - ring_ms / reference_ms) * 100.0, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/experiment_registry.json b/research/experiment_registry.json new file mode 100644 index 0000000..ffd85da --- /dev/null +++ b/research/experiment_registry.json @@ -0,0 +1,917 @@ +{ + "schema": "h3-blackwell-runtime-experiment-registry", + "version": "1.0.0", + "generated_at": "2026-08-25", + "allowed_statuses": [ + "production_accepted", + "research_retained", + "architecture_rejected", + "performance_rejected", + "quality_rejected", + "temporarily_blocked", + "incomplete" + ], + "experiments": [ + { + "id": "elem-fused-residual-rmsnorm", + "name": "Fused residual + RMSNorm", + "family": "elementwise-fusion", + "status": "production_accepted", + "hypothesis": "Combining exact H3 normalization/modulation and residual-gate work reduces tensor traffic without changing model arithmetic.", + "implementation_strategy": "Use the deployed fused H3 elementwise kernels while preserving explicit BF16 boundaries.", + "source_locations": ["src/h3_blackwell_runtime/block.py", "src/h3_blackwell_runtime/h3_fusion.py", "PERFORMANCE_ROADMAP.md"], + "active_source_location": "src/h3_blackwell_runtime/h3_fusion.py", + "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "dirty working tree at registry generation"}, + "metrics": {"result": "accepted; exact measurements remain in linked artifacts"}, + "correctness_evidence": ["Repository current-state documentation records bit-exact blocks and checksum-identical trajectories."], + "decision_rationale": "The exact path passed block, trajectory, deployment, and memory gates and is selected for Spark production.", + "reproducer_commands": [], + "timestamp": null, + "evidence_missing": ["Portable validation outside GB10/SM121"], + "production_behavior": "Enabled for the selected Spark runtime through the fused elementwise path.", + "source_recovery": "Active implementation and validator remain in the working tree." + }, + { + "id": "nvfp4-quantize-alone", + "name": "NVFP4 quantize alone", + "family": "nvfp4-projection", + "status": "research_retained", + "hypothesis": "An owned standalone quantizer can match the reference packing contract and expose scale/packing cost.", + "implementation_strategy": "Discover scales and emit native QDATA/SFA independently of GEMM.", + "source_locations": ["src/h3_blackwell_runtime/nvfp4_quant.py", "tools/profile_nvfp4_linear.py", "PERFORMANCE_ROADMAP.md"], + "active_source_location": "src/h3_blackwell_runtime/nvfp4_quant.py", + "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json", "exists": true}], + "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "prototype infrastructure in dirty working tree"}, + "metrics": {"conclusion": "packing contract established; standalone speed not consistently favorable"}, + "correctness_evidence": ["Current roadmap records packed-bit exactness at H3 widths."], + "decision_rationale": "Useful as reference infrastructure, but not selected as an independent production optimization.", + "reproducer_commands": [], "timestamp": null, + "evidence_missing": ["Standalone canonical deployment gate"], + "production_behavior": "Not dispatched as a standalone replacement.", + "source_recovery": "Implementation is present in nvfp4_quant.py and historical commits named in git log." + }, + { + "id": "nvfp4-quantize-gemm", + "name": "NVFP4 quantize + GEMM", + "family": "nvfp4-projection", + "status": "research_retained", + "hypothesis": "Pairing the owned packing contract with the reference GEMM establishes an exact projection baseline.", + "implementation_strategy": "Wrap native packed activations for Comfy Kitchen/CUBLAS NVFP4 GEMM and compare outputs.", + "source_locations": ["src/h3_blackwell_runtime/nvfp4.py", "src/h3_blackwell_runtime/nvfp4_quant.py", "benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json"], + "active_source_location": "src/h3_blackwell_runtime/nvfp4.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "reference/prototype path"}, + "metrics": {"conclusion": "exact baseline retained; no standalone production win documented"}, + "correctness_evidence": ["Roadmap records exact outputs at all four H3 projection widths after swizzle correction."], + "decision_rationale": "Retained as the exact baseline and seam for producer fusion rather than promoted as a separate backend.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Isolated accepted performance gate"], + "production_behavior": "Production continues to use the established NVFP4 linear dispatch.", + "source_recovery": "Active wrapper and quantizer are present." + }, + { + "id": "nvfp4-prequant-swiglu-gemm", + "name": "Fused prequantized SiLU x up + quantize + GEMM", + "family": "nvfp4-producer-fusion", + "status": "production_accepted", + "hypothesis": "Recomputing exact SwiGLU BF16 boundaries inside NVFP4 production removes the complete FC2 activation intermediate.", + "implementation_strategy": "Fuse SiLU/multiply into scale and packing while retaining the exact Comfy FC2 GEMM.", + "source_locations": ["src/h3_blackwell_runtime/block.py", "src/h3_blackwell_runtime/nvfp4_quant.py", "NVFP4_SWIGLU_FUSION_DESIGN.md"], + "active_source_location": "src/h3_blackwell_runtime/nvfp4_quant.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-nvfp4-swiglu-fusion-summary.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-fully-fused-fresh-nsight-summary.json", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": "sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83", "commit_context": "dirty working tree at registry generation"}, + "metrics": {"result": "accepted; measurements are in the linked summary"}, + "correctness_evidence": ["Producer bytes and warmed trajectory outputs are documented as bit-identical."], + "decision_rationale": "Passed exactness, block, canonical trajectory, and deployment smoke gates.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Non-GB10 architecture validation"], + "production_behavior": "Selected Spark deployments enable H3_NVFP4_SWIGLU_FUSION=1 for eligible base requests.", + "source_recovery": "Active source and design document are present." + }, + { + "id": "nvfp4-prequant-modulate-gemm", + "name": "Fused modulation + quantize + GEMM", + "family": "nvfp4-producer-fusion", + "status": "production_accepted", + "hypothesis": "Recomputing the exact BF16 AdaLN modulation boundary inside NVFP4 scale discovery and packing removes the materialized modulated QKV and FC1 inputs.", + "implementation_strategy": "Fuse segment-specific modulation into native QDATA/SFA production while retaining the unchanged Comfy NVFP4 GEMMs.", + "source_locations": ["src/h3_blackwell_runtime/block.py", "src/h3_blackwell_runtime/nvfp4.py", "src/h3_blackwell_runtime/nvfp4_quant.py", "NVFP4_MODULATE_FUSION_DESIGN.md"], + "active_source_location": "src/h3_blackwell_runtime/nvfp4_quant.py", + "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-nvfp4-modulate-fusion-summary.json", "exists": true}, {"path": "benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json", "exists": true}], + "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": "sha256:5f879c43374bcedb89745971d7d95d82afc8fcf9c41d30f11b257c2863b9fe28", "commit_context": "active source and production manifests"}, + "metrics": {"packed_tokens": 37810, "block_24_producer_p50_ms": 6.018611951731145, "alternating_block_p50_improvement_percent": {"block_0": 0.2794895723218782, "block_24": 0.6017027236613792, "block_49": 0.7939711616055223}, "warmed_two_step_seconds": {"baseline": 47.66834012803156, "candidate": 47.41923755296739, "improvement_percent": 0.5225744684944189}, "warmed_twelve_step_seconds": {"baseline": 285.99590051098494, "candidate": 284.4078275830252, "improvement_percent": 0.5552782138213641}}, + "correctness_evidence": ["The linked summary records equal tensor scales, zero QDATA and SFA differences for blocks 0/24/49, and bit-identical warmed two-step and twelve-step video/audio tensors."], + "decision_rationale": "The exact producer fusion improved all three alternating block medians and warmed two-step and canonical twelve-step trajectories, then passed the deployment smoke test.", + "reproducer_commands": [], + "timestamp": null, + "evidence_missing": ["Distributed, chunked-MLP, full-precision, pre-scale, autograd, and active-LoRA cases retain the materialized fallback"], + "production_behavior": "Spark single-GPU manifests enable H3_NVFP4_MODULATE_FUSION=1; ineligible cases retain the materialized fallback.", + "source_recovery": "Active dispatch, wrapper, quantizer, validators, design document, and benchmark artifacts remain in the working tree." + }, + { + "id": "lora-fused-down-activation", + "name": "Fused LoRA down + activation", + "family": "lora-fusion", "status": "incomplete", + "hypothesis": "Fusing LoRA down projection with activation production may avoid a materialized BF16 input.", + "implementation_strategy": "Produce LoRA-down operands at the activation boundary without changing LoRA arithmetic.", + "source_locations": ["src/h3_blackwell_runtime/lora.py", "TURBO.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "The repository documents materialized LoRA fallback behavior but no isolated conclusion for this fusion boundary.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Recoverable candidate source", "Benchmark artifact", "Parity evidence"], + "production_behavior": "Not selected; active LoRA uses the materialized fallback.", + "source_recovery": "Search history and external experiment storage; no candidate source was found in this checkout." + }, + { + "id": "lora-fused-up-residual", + "name": "Fused LoRA up + residual", + "family": "lora-fusion", "status": "incomplete", + "hypothesis": "Applying LoRA-up output directly at the residual boundary may eliminate an intermediate.", + "implementation_strategy": "Fuse LoRA up projection accumulation with the exact residual update order.", + "source_locations": ["src/h3_blackwell_runtime/lora.py", "TURBO.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "No defensible performance or exactness conclusion is present for this materially distinct boundary.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Benchmark", "Residual-order parity"], + "production_behavior": "Not selected; production applies LoRA through the existing fallback.", + "source_recovery": "No candidate source found in current tree or searched git history." + }, + { + "id": "lora-fused-full-down-up", + "name": "Fused full LoRA down/up", + "family": "lora-fusion", "status": "incomplete", + "hypothesis": "A single fused down/up path can reduce launch and materialization overhead for active adapters.", + "implementation_strategy": "Execute both low-rank projections in one owned path while preserving rank, alpha, and BF16 boundaries.", + "source_locations": ["src/h3_blackwell_runtime/lora.py", "TURBO.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No complete candidate or conclusive artifact was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Implementation", "Timing", "Turbo trajectory parity"], + "production_behavior": "Not selected.", "source_recovery": "Recover from external research checkout if it existed." + }, + { + "id": "lora-direct-materialization-gemm-prepass", + "name": "Direct LoRA materialization into GEMM prepass memory", + "family": "lora-fusion", "status": "performance_rejected", + "hypothesis": "Materializing exact BF16 LoRA input inside fused producer prepass memory can retain producer fusion for Turbo requests.", + "implementation_strategy": "Add optional BF16 materialization while modulation/SwiGLU producers pack NVFP4.", + "source_locations": ["src/h3_blackwell_runtime/nvfp4_quant.py", "PERFORMANCE_ROADMAP.md", "CURRENT_STATE.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "prototype reverted from deployment"}, + "metrics": {"result": "bit-exact but slower; exact measurements remain in artifact"}, + "correctness_evidence": ["Canonical Turbo-4 trajectory documented as bit-exact."], + "decision_rationale": "Extra BF16 writes cost more than the removed standalone producer.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Profiler report isolating write cost"], + "production_behavior": "Rejected; active LoRA retains exact materialized fallback and disables producer fusion.", + "source_recovery": "Historical behavior is documented; prototype is not an active dispatch." + }, + { + "id": "fc2-fused-activation-residual", + "name": "Fused FC2 activation/residual", + "family": "projection-epilogue", "status": "incomplete", + "hypothesis": "An exact FC2 epilogue can combine activation projection, gate, and residual while reducing traffic.", + "implementation_strategy": "Round GEMM output to BF16, apply gate with the reference boundary, then add BF16 residual in reference order.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "NVFP4_SWIGLU_FUSION_DESIGN.md", "src/h3_blackwell_runtime/block.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "The numerical contract is documented, but no integrated epilogue candidate passed a gate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Owned FC2 epilogue source", "Randomized parity", "Block and trajectory benchmarks"], + "production_behavior": "FC2 GEMM and residual gate remain separate production operations.", + "source_recovery": "Design requirements are in PERFORMANCE_ROADMAP.md." + }, + { + "id": "qkv-fused-projection-layout", + "name": "Fused QKV projection + layout", + "family": "qkv-layout", "status": "architecture_rejected", + "hypothesis": "A post-projection fused layout kernel can remove three QKV materializations.", + "implementation_strategy": "Write or copy interleaved projection output into backend-specific Q/K/V layouts.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/attention.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "superseded by views"}, + "metrics": {"conclusion": "post-GEMM copy does not remove the projection boundary"}, + "correctness_evidence": ["The accepted strided-NHD path demonstrates copies are unnecessary for Sage2."], + "decision_rationale": "A separate layout writer is architecturally inferior to direct strided views or a true GEMM epilogue.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Historical candidate-specific benchmark"], + "production_behavior": "Production uses projection-strided NHD views instead.", + "source_recovery": "Superseded concept; use current attention.py for the accepted architecture." + }, + { + "id": "qkv-fused-projection-rmsnorm", + "name": "Fused QKV + RMSNorm", + "family": "qkv-preparation", "status": "incomplete", + "hypothesis": "Fusing QKV output production with Q/K RMSNorm can avoid reading and writing full Q/K tensors.", + "implementation_strategy": "Apply per-head RMSNorm as part of projection output handling before attention layout formatting.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/attention.py", "SAGE2_BLACKWELL_DESIGN.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No projection-integrated implementation or gate was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Projection-integrated parity", "Timing"], + "production_behavior": "Production runs the established fused RMSNorm/RoPE operator after projection.", + "source_recovery": "Only the design target is recoverable from current docs." + }, + { + "id": "qkv-direct-sageattention-layout", + "name": "Direct QKV output into SageAttention layout", + "family": "qkv-layout", "status": "production_accepted", + "hypothesis": "Sage2 can consume projection-strided NHD Q/K/V views without QKV copies.", + "implementation_strategy": "View interleaved projection output as NHD, normalize/RoPE in place, and call Sage2 NHD.", + "source_locations": ["src/h3_blackwell_runtime/attention.py", "CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": "src/h3_blackwell_runtime/attention.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json", "exists": true}, {"path": "benchmarks/gb10-sage-strided-nhd-contract-1344x768-124f-seed440420.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "active source in dirty working tree"}, + "metrics": {"result": "accepted; measurements in linked summary"}, + "correctness_evidence": ["Blocks 0/24/49 and two-step/canonical trajectories documented as bit-exact."], + "decision_rationale": "Removes three large copies and passed production gates.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Distributed-path equivalent"], + "production_behavior": "Selected for single-GPU Sage2 with H3_SAGE_QKV_LAYOUT=strided_nhd.", + "source_recovery": "Active in H3SageAttention.forward." + }, + { + "id": "qkv-to-sage-fused", + "name": "QKV-to-Sage fused", + "family": "qkv-preparation", "status": "performance_rejected", + "hypothesis": "Fusing direct strided Q/K preparation through Sage2 Q quantization can remove an entry launch and traffic.", + "implementation_strategy": "Fuse NHD Q/K RMSNorm, RoPE, and Q INT8 quantization while retaining K/V preparation and mainloop.", + "source_locations": ["src/h3_blackwell_runtime/sage2_entry.py", "tools/validate_sage2_entry_fusion.py", "SAGE2_BLACKWELL_DESIGN.md", "CURRENT_STATE.md", "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "runtime branch removed"}, + "metrics": {"result": "exact but below complete-block gate"}, + "correctness_evidence": ["Randomized lengths, real tensors, quantized buffers, attention output, and blocks are documented bit-exact."], + "decision_rationale": "Entry speedup did not reach the required complete-block improvement.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Two-step and canonical trajectory runs were intentionally skipped"], + "production_behavior": "Callable wrapper/export and validator source remain, but production integration and dispatch are absent; production preparation is unchanged.", + "source_recovery": "Callable prepare_qk/attention_nhd source remains in sage2_entry.py and the validator remains in tools/validate_sage2_entry_fusion.py; evidence artifacts are retained, but production integration is absent." + }, + { + "id": "qknorm-fused", + "name": "qknorm fused", + "family": "qkv-preparation", "status": "research_retained", + "hypothesis": "Fusing Q/K RMSNorm and split-half RoPE reduces preparation launches while preserving H3 arithmetic.", + "implementation_strategy": "Use the Comfy Kitchen in-place rms_rope_split_half operator.", + "source_locations": ["src/h3_blackwell_runtime/attention.py", "CURRENT_STATE.md"], "active_source_location": "src/h3_blackwell_runtime/attention.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-post-optimization-sage2-nhd-components.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "active dependency operator"}, + "metrics": {"role": "current exact preparation reference"}, + "correctness_evidence": ["Used by the accepted exact Sage2 path."], + "decision_rationale": "Retained as the current exact preparation operation; it is not independently claimed as a production experiment win.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Standalone gate specific to qknorm fusion"], + "production_behavior": "Called by production attention preparation.", "source_recovery": "Invocation remains in attention.py; implementation belongs to Comfy Kitchen." + }, + { + "id": "qknorm-p0-fused-qk-normalization", + "name": "Fused P0 Q/K normalization", + "family": "qkv-preparation", "status": "incomplete", + "hypothesis": "A P0-specific Q/K normalization kernel may reduce preparation overhead before broader entry fusion.", + "implementation_strategy": "Fuse Q and K normalization at the earliest preparation checkpoint.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "tools/profile_attention_components.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "The current checkout has no candidate-specific artifact separating this label from later P1 entry fusion.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Distinct benchmark", "Decision record"], + "production_behavior": "No distinct P0 normalization dispatch.", "source_recovery": "Check external or deleted research branches." + }, + { + "id": "qkv-fused-rope-output-format", + "name": "Fused RoPE + QKV output formatting", + "family": "qkv-preparation", "status": "performance_rejected", + "hypothesis": "Combining RoPE with attention-format stores removes a Q/K pass.", + "implementation_strategy": "Apply split-half RoPE while writing prepared Q/K into the Sage entry representation.", + "source_locations": ["src/h3_blackwell_runtime/sage2_entry.py", "tools/validate_sage2_entry_fusion.py", "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json", "SAGE2_BLACKWELL_DESIGN.md", "src/h3_blackwell_runtime/attention.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "covered by rejected P1 entry candidate"}, + "metrics": {"result": "component was exact within P1 but complete-block gate failed"}, + "correctness_evidence": ["P1 prepared Q/K and complete output parity."], + "decision_rationale": "As implemented in the fused P1 boundary, formatting fusion did not deliver enough block-level benefit.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Standalone ablation"], + "production_behavior": "Callable P1 wrapper/export and validator source remain without production integration; production keeps the existing RMSNorm/RoPE preparation on strided views.", "source_recovery": "The P1 prepare_qk wrapper remains in sage2_entry.py and its validator remains in tools/validate_sage2_entry_fusion.py; production dispatch is absent." + }, + { + "id": "gemm-persistent", + "name": "Persistent GEMM", + "family": "cute-nvfp4", "status": "incomplete", + "hypothesis": "A persistent scheduler can amortize activation production and launch overhead across output tiles.", + "implementation_strategy": "Use a persistent work queue with bounded tile reuse rather than one producer per output-N CTA.", + "source_locations": ["NVFP4_STREAMING_DESIGN.md", "PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/cute_qkv_ring.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "proposed successor to rejected ring schedule"}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "The bounded ring was tested, but the distinct launch-fused persistent scheduler remains unimplemented or unconcluded.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Persistent scheduler source", "Block gate", "Trajectory validation"], + "production_behavior": "No persistent GEMM production dispatch.", "source_recovery": "Design target remains in roadmap; ring prototype is not equivalent." + }, + { + "id": "gemm-warp-specialized", + "name": "Warp-specialized GEMM", + "family": "cute-nvfp4", "status": "incomplete", + "hypothesis": "Separating producer and consumer warps can overlap packing with tensor-core GEMM.", + "implementation_strategy": "Assign DMA/packing and MMA roles to different warps with staged shared-memory handoff.", + "source_locations": ["NVFP4_STREAMING_DESIGN.md", "benchmarks/gb10-cute-p1-stream-a-summary.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-p1-stream-a-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "validator prototype"}, + "metrics": {"result": "fixed-tile contract demonstrated; complete warp-specialized runtime conclusion absent"}, + "correctness_evidence": ["P1 shared-memory producer contract is documented bit-exact."], + "decision_rationale": "The available evidence covers a fixed tile and a rejected per-CTA schedule, not a complete production candidate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Canonical runtime implementation", "Integrated benchmark"], + "production_behavior": "Not selected.", "source_recovery": "Validator evidence remains under benchmarks." + }, + { + "id": "runtime-event-overhead", + "name": "Event-overhead experiment", + "family": "runtime-overhead", "status": "research_retained", + "hypothesis": "CUDA event instrumentation may distort small kernel and launch-boundary measurements.", + "implementation_strategy": "Compare synchronized component probes with uninstrumented end-to-end timing.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "tools/profile_h3_block.py", "tools/profile_attention_components.py"], "active_source_location": "tools/profile_h3_block.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-fully-fused-fresh-nsight-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "profiling methodology"}, + "metrics": {"conclusion": "component probes are attribution-only; final claims require separate uninstrumented runs"}, + "correctness_evidence": [], "decision_rationale": "Retained as measurement methodology, not a production feature.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Dedicated event-overhead artifact"], + "production_behavior": "No production behavior change.", "source_recovery": "Profiling tools and methodology remain." + }, + { + "id": "attention-full-cuda-graph-capture", + "name": "Full-attention CUDA graph/capture", + "family": "cuda-graph", "status": "incomplete", + "hypothesis": "Capturing full attention can reduce repeated launch overhead for stable shape buckets.", + "implementation_strategy": "Capture the complete attention preparation/mainloop/output path with preallocated buffers.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "CUDA graph work is explicitly listed as missing and deferred until shape/kernel policies stabilize.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Capture implementation", "Graph safety validation", "Timing"], + "production_behavior": "No full-attention graph capture.", "source_recovery": "Roadmap-only target." + }, + { + "id": "cute-conversion-contract", + "name": "CuTe NVFP4 conversion contract", + "family": "cute-nvfp4", "status": "research_retained", + "hypothesis": "CUTLASS DSL can consume the same logical H3 block-scaled FP4 data with an exact epilogue policy.", + "implementation_strategy": "Validate E2M1/E4M3 conversion, scale layouts, and alpha-before-BF16 epilogue on real tiles.", + "source_locations": ["tools/validate_cute_nvfp4_conversion.py", "NVFP4_STREAMING_DESIGN.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": "tools/validate_cute_nvfp4_conversion.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-nvfp4-conversion-contract.json", "exists": true}, {"path": "benchmarks/gb10-cute-p0-h3-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "validator-only"}, + "metrics": {"conclusion": "QKV, attention output, and FC1 exact; FC2 policy differs"}, + "correctness_evidence": ["Real-tile packed data, scales, and outputs are documented exact for selected roles."], + "decision_rationale": "Retained as foundational interoperability evidence, not runtime dispatch.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Packaged runtime kernel"], + "production_behavior": "No CuTe production dispatch.", "source_recovery": "Validation tools and artifacts are present." + }, + { + "id": "cute-tiny-tile", + "name": "CuTe tiny-tile attempt", + "family": "cute-nvfp4", "status": "research_retained", + "hypothesis": "A fixed tiny real tile can prove the producer-consumer conversion contract before full shapes.", + "implementation_strategy": "Replace A/SFA TMA input for a fixed 128-row tile and compare every packed byte and output.", + "source_locations": ["tools/validate_cute_nvfp4_tile_producer.py", "benchmarks/gb10-cute-nvfp4-tile-producer.json", "PERFORMANCE_ROADMAP.md"], "active_source_location": "tools/validate_cute_nvfp4_tile_producer.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-nvfp4-tile-producer.json", "exists": true}, {"path": "benchmarks/gb10-cute-p1-stream-a-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "validator-only"}, + "metrics": {"scope": "fixed 128-row checkpoint"}, + "correctness_evidence": ["Every real 128-K tile and complete selected-role output documented bit-exact."], + "decision_rationale": "Successful research checkpoint, but deliberately not a full runtime candidate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Canonical M/padding runtime support"], + "production_behavior": "Validator only.", "source_recovery": "Tool and benchmark artifacts remain." + }, + { + "id": "cute-temporary-output", + "name": "CuTe temporary-output attempt", + "family": "cute-nvfp4", "status": "architecture_rejected", + "hypothesis": "A temporary global output can bridge an owned CuTe GEMM to existing downstream code.", + "implementation_strategy": "Materialize complete intermediate output and copy/reshape after GEMM.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md", "NVFP4_STREAMING_DESIGN.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "Complete temporary materialization fails the stated boundary-removal architecture even if numerically correct.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Historical candidate artifact"], + "production_behavior": "Not selected.", "source_recovery": "No source found; architecture is superseded by direct/bounded producer designs." + }, + { + "id": "cute-failed-builds", + "name": "CuTe failed-build attempts", + "family": "cute-toolchain", "status": "temporarily_blocked", + "hypothesis": "Candidate CuTe kernels can be compiled for SM121 with the available DSL/toolchain.", + "implementation_strategy": "Build generated CUTLASS DSL kernels for the target architecture.", + "source_locations": ["CURRENT_STATE.md", "SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-post-optimization-profile-summary.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-post-optimization-profile-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "build failures summarized, logs external or absent"}, + "metrics": {"result": "one or more candidate builds rejected"}, + "correctness_evidence": [], "decision_rationale": "Build/toolchain failure prevents a defensible architecture or performance decision.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Per-attempt build logs", "Exact compiler versions", "Candidate source mapping"], + "production_behavior": "No failed candidate is dispatched.", "source_recovery": "Recover run-specific build directories from external Spark storage." + }, + { + "id": "cute-compile-blowups", + "name": "CuTe compile-blowup attempts", + "family": "cute-toolchain", "status": "temporarily_blocked", + "hypothesis": "Larger static schedules can be generated without prohibitive compile-time/resource growth.", + "implementation_strategy": "Instantiate wider tile/schedule variants in CUTLASS DSL.", + "source_locations": ["NVFP4_STREAMING_DESIGN.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "Compiler resource blowup is a temporary implementation/toolchain blocker, not proof the algorithm is wrong.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Compiler log", "Exact variant dimensions", "Toolchain versions"], + "production_behavior": "Not selected.", "source_recovery": "No local build log found; recover external build workspace if available." + }, + { + "id": "cute-oom-vram", + "name": "CuTe OOM/VRAM-failure attempts", + "family": "cute-resource-limit", "status": "temporarily_blocked", + "hypothesis": "Full-shape duplicate-model or workspace trials fit available GB10 memory.", + "implementation_strategy": "Run full projection and model-level variants with large outputs/workspaces.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "benchmarks/gb10-cute-p2-ring-full-projection-summary.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-p2-ring-full-projection-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "duplicate-model harness memory pressure"}, + "metrics": {"result": "FC1 timing unusable; memory pressure caused variance and a container stop"}, + "correctness_evidence": [], "decision_rationale": "Resource failure blocked reliable timing and does not establish a candidate quality or architecture decision.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Peak VRAM measurement", "Container failure log"], + "production_behavior": "No affected candidate selected.", "source_recovery": "Summary artifact remains; detailed container logs are absent." + }, + { + "id": "cute-bounded-ring", + "name": "CuTe bounded ring-attention/projection attempts", + "family": "cute-ring", "status": "performance_rejected", + "hypothesis": "A bounded global packed-tile ring can produce each activation tile once and reuse it across N consumers.", + "implementation_strategy": "Use caller-owned QDATA/SFA ring slots with allocation-free production and chunked GEMM consumption.", + "source_locations": ["src/h3_blackwell_runtime/cute_qkv_ring.py", "tools/validate_cute_nvfp4_ring.py", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-p2-ring-capacity-summary.json", "exists": true}, {"path": "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "opt-in disabled prototype"}, + "metrics": {"result": "projection parity passed; alternating complete-block gate regressed"}, + "correctness_evidence": ["Full projections in blocks 0/24/49, including tail chunk, documented bit-exact."], + "decision_rationale": "Isolated projection gains did not survive chunk launch and scheduler overhead at the block gate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Trajectory validation intentionally skipped"], + "production_behavior": "Opt-in dispatchable code remains in cute_qkv_ring.py and nvfp4.py, but H3_CUTE_QKV_RING is disabled by default and absent from production manifests.", "source_recovery": "Dispatchable prototype source remains in src/h3_blackwell_runtime/cute_qkv_ring.py and src/h3_blackwell_runtime/nvfp4.py; validation artifacts are also present." + }, + { + "id": "cute-p1-per-output-cta-streaming", + "name": "CuTe P1 per-output-N-CTA streaming", + "family": "cute-nvfp4", "status": "performance_rejected", + "hypothesis": "Each output-N CTA can produce its own A/SFA tile cheaply enough to eliminate global activation packing.", + "implementation_strategy": "Use the DMA warp to convert BF16 A directly into staged E2M1/SFA shared memory independently in every output-N CTA.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "benchmarks/gb10-cute-p1-stream-a-timing-summary.json", "tools/validate_cute_nvfp4_real_tiles.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-p1-stream-a-timing-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "validator/timing prototype"}, + "metrics": {"result": "exact selected-role outputs, but repeated producer work made the schedule much slower"}, + "correctness_evidence": ["Real QKV, attention-output, and FC1 tile/output identity is documented in the P1 artifacts."], + "decision_rationale": "Every output-N CTA rereads and repacks A; required N reuse is not viable under the accumulator/register budget.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate kernel source is external to this checkout"], + "production_behavior": "Rejected schedule is not dispatched.", "source_recovery": "Timing and contract artifacts remain; recover external CuTe kernel checkout for source." + }, + { + "id": "cute-p2-full-workspace-ring", + "name": "CuTe P2 full-workspace ring", + "family": "cute-ring", "status": "performance_rejected", + "hypothesis": "A full-activation-sized ring/workspace removes chunk recycling overhead and improves the complete block.", + "implementation_strategy": "Expand ring capacity to cover the canonical activation while retaining caller-owned packed buffers.", + "source_locations": ["src/h3_blackwell_runtime/cute_qkv_ring.py", "benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json", "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json", "exists": true}, {"path": "benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "opt-in capacity variant"}, + "metrics": {"result": "exact capacity variant with a documented complete-block regression"}, + "correctness_evidence": ["Alternating block artifact records equality for the candidate path."], + "decision_rationale": "Removing recycling did not rescue the architecture and substantially worsened the controlled block result.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Trajectory validation intentionally skipped"], + "production_behavior": "Not selected; bounded-ring dispatch remains disabled.", "source_recovery": "Runtime prototype and capacity artifact remain." + }, + { + "id": "cute-fc2-streaming-policy", + "name": "CuTe FC2 streamed producer attempt", + "family": "cute-nvfp4", "status": "quality_rejected", + "hypothesis": "The same streamed CuTe conversion/epilogue policy used for QKV, attention output, and FC1 can preserve FC2 exactly.", + "implementation_strategy": "Run FC2 through the owned E2M1/E4M3 producer and alpha-before-BF16 epilogue contract.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json", "benchmarks/gb10-cute-p0-h3-summary.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json", "exists": true}, {"path": "benchmarks/gb10-cute-p0-h3-summary.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "P0 numerical-contract experiment"}, + "metrics": {"result": "non-exact because FC2 reference reduction policy differs"}, + "correctness_evidence": ["P0 artifacts isolate the FC2 mismatch while other selected roles match."], + "decision_rationale": "The candidate fails the exact FC2 numerical contract; FC2 streaming is explicitly excluded.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["No accepted alternative FC2 reduction implementation"], + "production_behavior": "FC2 retains the Comfy/CUBLAS fallback.", "source_recovery": "P0 artifacts and fallback policy are retained in the roadmap." + }, + { + "id": "layout-direct-temporal-output", + "name": "Direct-to-temporal output", + "family": "attention-layout", "status": "incomplete", + "hypothesis": "Writing QKV or attention results directly to a temporal consumer layout avoids a copy.", + "implementation_strategy": "Use consumer-native temporal indexing in the producer store path.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "VORTEX_RESEARCH_AGENDA.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No source or artifact supports a final decision for this named layout.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Layout contract", "Candidate source", "Benchmark"], + "production_behavior": "Not selected.", "source_recovery": "Search external research branches/artifact storage." + }, + { + "id": "layout-direct-sage-output", + "name": "Direct-to-Sage output", + "family": "attention-layout", "status": "incomplete", + "hypothesis": "A QKV producer can write a Sage-specific output layout directly rather than expose views over interleaved projection output.", + "implementation_strategy": "Change the producer store contract to emit consumer-native Sage buffers without a separate formatting copy.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/attention.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "historical producer/output-layout concept distinct from accepted strided views"}, + "metrics": {}, + "correctness_evidence": [], "decision_rationale": "No distinct producer-store implementation or candidate-specific evidence was found. The accepted no-copy strided-NHD view contract is recorded once under qkv-direct-sageattention-layout and is not evidence that this producer/output-layout concept was accepted.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Distinct producer-store source", "Candidate-specific correctness evidence", "Candidate-specific benchmark"], + "production_behavior": "No separate direct-to-Sage producer/output-layout acceptance; production uses the strided-NHD view contract recorded by qkv-direct-sageattention-layout.", "source_recovery": "Only the historical concept and the separate accepted view implementation are recoverable in this checkout." + }, + { + "id": "layout-direct-q-padding-32", + "name": "Direct output into q_padding=32 buffers", + "family": "attention-layout", "status": "architecture_rejected", + "hypothesis": "Writing directly into padded Q buffers can remove a later padding copy.", + "implementation_strategy": "Allocate/store producer output in buffers with q_padding=32.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "Durable padded producer buffers conflict with the accepted strided-view/direct-consumer architecture and no contrary evidence was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Historical benchmark", "Candidate source"], + "production_behavior": "Production does not allocate this direct padded-Q boundary.", "source_recovery": "No candidate source found." + }, + { + "id": "layout-dynamic-to-hot-allocation", + "name": "Dynamic .to(...) hot-path allocation", + "family": "runtime-allocation", "status": "architecture_rejected", + "hypothesis": "On-demand device/dtype conversion in the hot path is acceptable for layout preparation.", + "implementation_strategy": "Call dynamic tensor .to(...) operations during repeated attention execution.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/attention.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], + "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], + "decision_rationale": "Repeated dynamic allocation/conversion is incompatible with allocation-free hot-path and graph-capture goals.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Historical candidate-specific timing"], + "production_behavior": "Not an accepted optimization strategy.", "source_recovery": "Conceptual rejection; no retained candidate source located." + }, + { + "id": "layout-q-coalesced-16b-store", + "name": "Coalesced-16-byte Q layout store", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "Coalesced 16-byte Q stores improve layout-writer throughput.", + "implementation_strategy": "Vectorize Q output stores to aligned 16-byte transactions.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/csrc/nvfp4_scale.cu"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No named artifact or recoverable ablation was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Source mapping", "Alignment proof", "Benchmark"], + "production_behavior": "No distinct dispatch.", "source_recovery": "Inspect external CUDA experiment source if available." + }, + { + "id": "layout-qkv-all-16b-stores", + "name": "All-16-byte Q/K/V stores", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "Using aligned 16-byte stores for Q, K, and V maximizes layout write bandwidth.", + "implementation_strategy": "Vectorize every Q/K/V output path to 16-byte transactions.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "src/h3_blackwell_runtime/csrc/nvfp4_scale.cu"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No defensible conclusion survives in repository evidence.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Correctness", "Timing"], + "production_behavior": "No distinct dispatch.", "source_recovery": "External/deleted experiment source required." + }, + { + "id": "layout-xor-swizzle-q", + "name": "XOR-swizzle Q variant", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "XOR swizzling Q addresses reduces partition/bank conflicts.", + "implementation_strategy": "Apply an XOR mapping to Q store coordinates.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "src/h3_blackwell_runtime/csrc/nvfp4_scale.cu"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No candidate-specific evidence was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Swizzle definition", "Profiler counters", "Parity"], + "production_behavior": "Not selected.", "source_recovery": "Search external experiment source." + }, + { + "id": "layout-xor-swizzle-qk", + "name": "XOR-swizzle Q/K variant", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "Applying the XOR mapping to both Q and K improves aggregate preparation throughput.", + "implementation_strategy": "Swizzle Q and K stores while retaining V layout.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "src/h3_blackwell_runtime/csrc/nvfp4_scale.cu"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "Materially distinct from Q-only, but no retained evidence supports a conclusion.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Benchmark", "Parity"], + "production_behavior": "Not selected.", "source_recovery": "Search external experiment source." + }, + { + "id": "layout-swizzle-all", + "name": "Swizzle-all variants", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "Swizzling all output components avoids systemic store conflicts.", + "implementation_strategy": "Apply the candidate swizzle across Q/K/V and associated padded regions.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "src/h3_blackwell_runtime/csrc/nvfp4_scale.cu"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No retained source or benchmark identifies which swizzle-all variant was viable.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Variant enumeration", "Source", "Counters", "Timing"], + "production_behavior": "Not selected.", "source_recovery": "Recover external experiment notes/builds." + }, + { + "id": "layout-vectorized-padded-q-copy", + "name": "Vectorized-padded-Q copy", + "family": "attention-layout-copy", "status": "architecture_rejected", + "hypothesis": "A wide copy into padded Q reduces the cost of unavoidable preparation.", + "implementation_strategy": "Vector-copy Q into a separate padded destination.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "superseded by direct strided views"}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "Optimizing a complete copy is architecturally superseded by the accepted no-copy Sage layout.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Historical timing"], + "production_behavior": "No padded-Q copy on accepted direct path.", "source_recovery": "No candidate source retained." + }, + { + "id": "layout-q-copy-wide-store", + "name": "Q-copy wide-store variants", + "family": "attention-layout-copy", "status": "architecture_rejected", + "hypothesis": "Wider Q copy stores reduce standalone layout conversion latency.", + "implementation_strategy": "Sweep vector widths and block geometry for a separate Q copy kernel.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "superseded by no-copy path"}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "A faster standalone copy still fails the boundary-elimination objective.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Variant timings", "Source"], + "production_behavior": "Not selected.", "source_recovery": "Historical microbenchmarks were not found." + }, + { + "id": "layout-q-copy-block128x2", + "name": "Q-copy wide-store block128x2", + "family": "attention-layout-copy", "status": "architecture_rejected", + "hypothesis": "The block128x2 geometry improves wide-store Q copy occupancy/coalescing.", + "implementation_strategy": "Use a 128-by-2 block mapping for the Q copy writer.", + "source_locations": ["CURRENT_STATE.md", "PERFORMANCE_ROADMAP.md"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "superseded microvariant"}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "Materially distinct geometry, but still a complete-copy architecture superseded by strided views.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Exact geometry source", "Timing", "Parity"], + "production_behavior": "Not selected.", "source_recovery": "No source found in current tree/history search." + }, + { + "id": "layout-grid-kv-writers", + "name": "Grid-style KV writers", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "A 2D grid writer improves K/V store coalescing and load balance.", + "implementation_strategy": "Map K/V rows and heads across a grid instead of a linear writer.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "tools/profile_attention_components.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No retained artifact supports a decision.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Grid geometry", "Candidate source", "Benchmark"], + "production_behavior": "No distinct dispatch.", "source_recovery": "External experiment source required." + }, + { + "id": "layout-vectorized-kv-staging", + "name": "Vectorized K/V staging", + "family": "attention-layout-microvariant", "status": "incomplete", + "hypothesis": "Vectorized K/V staging reduces preparation memory transactions.", + "implementation_strategy": "Use aligned vector loads/stores during K/V preparation.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-block24-mainloop-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "The profiler identifies V staging behavior, but not a conclusive vectorized K/V candidate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Ablation timing", "Counters"], + "production_behavior": "Production Sage2 staging unchanged.", "source_recovery": "Use mainloop analysis as context; candidate source absent." + }, + { + "id": "sage2-online-pv", + "name": "Online-PV Sage2", + "family": "sage2-mainloop", "status": "incomplete", + "hypothesis": "An online PV schedule can overlap or reduce score materialization while preserving Sage2 accumulation semantics.", + "implementation_strategy": "Integrate PV consumption into online-softmax progress without changing per-warp numerical order.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-block24-mainloop-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No distinct online-PV implementation or final gate was found; P3 temporal pairing is recorded separately.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source", "Parity", "Mainloop timing"], + "production_behavior": "Production uses the public Sage2 mainloop.", "source_recovery": "Check isolated SageAttention worktrees." + }, + { + "id": "sage-scheduler-v1", + "name": "Sage scheduler/register-pressure v1", + "family": "sage2-scheduler", "status": "incomplete", + "hypothesis": "A first scheduler rewrite can reduce dependency stalls without increasing spills.", + "implementation_strategy": "Reorder QK/PV/softmax work and narrow live ranges.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "The aggregate P0 analysis survives, but it does not map v1 to an exact retained variant.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["v1 source mapping", "v1-specific timing", "v1 parity"], + "production_behavior": "Not selected.", "source_recovery": "Recover original SageAttention variant worktree/build logs." + }, + { + "id": "sage-scheduler-v2", + "name": "Sage scheduler/register-pressure v2", + "family": "sage2-scheduler", "status": "incomplete", + "hypothesis": "A second scheduling/register-pressure balance improves eligible-warp availability.", + "implementation_strategy": "Revise live ranges and instruction ordering relative to v1.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No variant-specific artifact permits a conclusion independent of aggregate P0 rejection.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["v2 source mapping", "v2 timing", "v2 counters"], + "production_behavior": "Not selected.", "source_recovery": "Recover isolated variant checkout if available." + }, + { + "id": "sage-scheduler-v3", + "name": "Sage scheduler/register-pressure v3", + "family": "sage2-scheduler", "status": "incomplete", + "hypothesis": "A third scheduler variant can cross the mainloop gate after earlier register-pressure tradeoffs.", + "implementation_strategy": "Apply the final recorded scheduling/register allocation variant.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "The name is required for archival completeness, but exact evidence mapping is missing.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["v3 source", "v3 benchmark", "v3 decision"], + "production_behavior": "Not selected.", "source_recovery": "Recover external experiment notes/builds." + }, + { + "id": "sage-p0-register-caps", + "name": "Sage P0 register-cap variants", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Capping registers can increase CTA residency enough to hide dependency stalls.", + "implementation_strategy": "Sweep register caps through the occupancy cliff and measure spills/latency.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-register-cap-latency-interleaved.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-sage2-p0-cap168-counters.ncu-rep", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated SageAttention P0"}, + "metrics": {"result": "occupancy gain at the cliff was overwhelmed by spills"}, + "correctness_evidence": ["P0 variants documented byte-exact where timed."], "decision_rationale": "No cap crossed the mainloop gate; the residency-changing cap caused severe spill traffic.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source in this checkout"], + "production_behavior": "Production compile policy unchanged.", "source_recovery": "NCU and latency artifacts retained." + }, + { + "id": "sage-p0-narrow-scopes", + "name": "Sage P0 narrowed scopes", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Narrowing C++ variable scopes reduces live ranges, spills, and scheduler stalls.", + "implementation_strategy": "Refactor exact mainloop scopes without changing arithmetic.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-register-scheduler-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-scopes-latency-interleaved.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-sage2-p0-scopes-counters.ncu-rep", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated P0 variant"}, + "metrics": {"result": "spill reduction without material latency gain"}, + "correctness_evidence": ["Documented byte-exact."], "decision_rationale": "Reduced spills did not improve mainloop enough and scheduler eligibility worsened.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source in current checkout"], + "production_behavior": "Not integrated.", "source_recovery": "Artifacts remain; source lived in an isolated checkout." + }, + { + "id": "sage-p0-inplace-scores", + "name": "Sage P0 in-place scores", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Reusing score storage in place lowers register pressure and data movement.", + "implementation_strategy": "Alias/reuse exact score fragments while preserving operation order.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-inplace-scores-latency-interleaved.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated P0 variant"}, + "metrics": {"result": "neutral or slower"}, "correctness_evidence": ["Documented byte-exact."], + "decision_rationale": "Did not cross the required mainloop improvement gate.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source"], + "production_behavior": "Not integrated.", "source_recovery": "Latency artifact retained." + }, + { + "id": "sage-p0-early-k-prefetch", + "name": "Sage P0 early-K prefetch", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Prefetching K earlier overlaps staging with compute.", + "implementation_strategy": "Move exact K staging ahead in the mainloop schedule.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-early-k-latency-interleaved.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated P0 variant"}, + "metrics": {"result": "slower or below gate"}, "correctness_evidence": ["Documented byte-exact."], + "decision_rationale": "No material latency gain was measured.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source"], + "production_behavior": "Not integrated.", "source_recovery": "Latency artifact retained." + }, + { + "id": "sage-p0-softmax-interleave", + "name": "Sage P0 independent softmax-chain interleaving", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Interleaving independent softmax chains hides scalar dependency latency.", + "implementation_strategy": "Reorder independent online-softmax chains without changing each chain's arithmetic.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-interleave-softmax-latency.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-interleave-softmax-latency.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated P0 variant"}, + "metrics": {"result": "neutral or slower"}, "correctness_evidence": ["Documented byte-exact."], + "decision_rationale": "Did not alleviate the measured scheduler bottleneck enough to pass.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source"], + "production_behavior": "Not integrated.", "source_recovery": "Latency artifact retained." + }, + { + "id": "sage-p0-padded-v-smem", + "name": "Sage P0 padded-V shared-memory layout", + "family": "sage2-scheduler", "status": "performance_rejected", + "hypothesis": "Padding V shared-memory stride removes excessive wavefronts and improves latency.", + "implementation_strategy": "Increase V shared stride to a 128-byte padded layout.", + "source_locations": ["SAGE2_BLACKWELL_DESIGN.md", "benchmarks/gb10-sage2-p0-pad-v-smem-latency.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p0-pad-v-smem-latency.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-sage2-p0-pad-v-smem-wavefronts.ncu-rep", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated P0 variant"}, + "metrics": {"result": "excess wavefront count unchanged; no material latency gain"}, + "correctness_evidence": ["Exact mainloop contract retained."], "decision_rationale": "The targeted counter was unaffected and latency gate failed.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Candidate source"], + "production_behavior": "Original Sage2 shared layout retained.", "source_recovery": "Latency and NCU artifacts retained." + }, + { + "id": "backend-flashinfer", + "name": "FlashInfer", + "family": "attention-backend", "status": "incomplete", + "hypothesis": "FlashInfer may provide a faster compatible dense attention backend for H3 shapes.", + "implementation_strategy": "Adapt H3 Q/K/V layout and non-causal head-dimension-128 contract to FlashInfer.", + "source_locations": ["src/h3_blackwell_runtime/attention.py", "tools/compare_attention_backends.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "FlashInfer is not an available backend in current source and no artifact was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Adapter", "Version", "Benchmark", "Quality evidence"], + "production_behavior": "Unavailable.", "source_recovery": "No source/history match found in this checkout." + }, + { + "id": "backend-flashattention-3", + "name": "FlashAttention 3", + "family": "attention-backend", "status": "incomplete", + "hypothesis": "FlashAttention 3 may outperform generic SDPA while retaining dense attention quality.", + "implementation_strategy": "Integrate the FA3 API for H3 BF16 Q/K/V and compare against Sage2.", + "source_locations": ["src/h3_blackwell_runtime/attention.py", "FLASH4.md", "tools/compare_attention_backends.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "Current source integrates FlashAttention 4, not FlashAttention 3; no FA3 evidence was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["FA3 source", "Package version", "Benchmark", "Quality evidence"], + "production_behavior": "Unavailable; FlashAttention 4 is a separate optional backend.", "source_recovery": "No FA3 source/history match found." + }, + { + "id": "backend-spargeattention", + "name": "SpargeAttention", + "family": "attention-backend", "status": "incomplete", + "hypothesis": "SpargeAttention sparsity may reduce long-sequence H3 attention cost with acceptable quality.", + "implementation_strategy": "Integrate backend routing and perform full latent/audio/visual quality gates.", + "source_locations": ["PERFORMANCE_ROADMAP.md", "tools/compare_attention_backends.py"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [], "profiler_artifacts": [], "environment": {"gpu": null, "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": null}, + "metrics": {}, "correctness_evidence": [], "decision_rationale": "No adapter, benchmark, or quality run was found.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Implementation", "Version", "Performance", "Media quality gate"], + "production_behavior": "Unavailable.", "source_recovery": "No source/history match found in this checkout." + }, + { + "id": "backend-sol", + "name": "Sol", + "family": "attention-backend", "status": "quality_rejected", + "hypothesis": "Sparse Sol routing can materially accelerate long H3 attention while preserving acceptable output quality.", + "implementation_strategy": "Route sparse non-causal attention with configurable threshold and optional residual INT8 Q/K.", + "source_locations": ["src/h3_blackwell_runtime/attention.py", "PERFORMANCE_ROADMAP.md", "VORTEX_RESEARCH_AGENDA.md"], "active_source_location": "src/h3_blackwell_runtime/attention.py", "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-37810-token-optimization-profile-seed440420.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "experimental backend available"}, + "metrics": {"conclusion": "no measured setting was both faster and numerically Sage-like"}, + "correctness_evidence": ["Relative-output comparisons against Sage2 are summarized in PERFORMANCE_ROADMAP.md."], + "decision_rationale": "Performance potential does not satisfy parity/quality requirements; complete media gates are absent.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Full latent/audio/lip-sync/visual quality sweep"], + "production_behavior": "Experimental opt-in only with fallback; Sage2 remains default.", "source_recovery": "Adapter remains active as an experimental backend." + }, + { + "id": "sage2-p1-entry-fusion-final", + "name": "Finalized Sage2 P1 entry fusion", + "family": "sage2-finalized", "status": "performance_rejected", + "hypothesis": "One exact entry kernel can fuse strided Q/K RMSNorm, RoPE, and Q quantization for a block-level gain.", + "implementation_strategy": "Leave K/V preparation and SM89 mainloop unchanged while replacing the entry sequence.", + "source_locations": ["src/h3_blackwell_runtime/sage2_entry.py", "tools/validate_sage2_entry_fusion.py", "SAGE2_BLACKWELL_DESIGN.md", "CURRENT_STATE.md", "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p1-entry-fusion-analysis.json", "exists": true}, {"path": "benchmarks/gb10-sage2-p1-real-parity.json", "exists": true}], + "profiler_artifacts": [{"path": "benchmarks/gb10-sage2-p1-blocks-candidate-l2.ncu-rep", "exists": true}], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "candidate runtime branch removed"}, + "metrics": {"result": "entry improved, but no complete block reached required gate"}, + "correctness_evidence": ["Randomized and real prepared tensors, quantization, Sage output, and complete blocks documented bit-exact."], + "decision_rationale": "Failed the required complete-block median improvement despite exactness.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Trajectory validation intentionally skipped"], + "production_behavior": "Callable wrapper/export and validator source remain, but production integration and dispatch are absent; the public Sage2 entry remains.", "source_recovery": "Callable prepare_qk/attention_nhd source remains in sage2_entry.py and the validator remains in tools/validate_sage2_entry_fusion.py; analysis and parity artifacts are retained, but production integration is absent." + }, + { + "id": "sage2-p2-direct-v-preparation-final", + "name": "Finalized Sage2 P2 direct V preparation", + "family": "sage2-finalized", "status": "performance_rejected", + "hypothesis": "Direct NHD BF16-to-padded/permuted FP8 V preparation can remove the full BF16 transpose and save enough block time.", + "implementation_strategy": "Use a three-stage maxima, scale-finalization, and direct E4M3 writer.", + "source_locations": ["src/h3_blackwell_runtime/sage2_entry.py", "tools/validate_sage2_vprep.py", "SAGE2_BLACKWELL_DESIGN.md", "CURRENT_STATE.md", "benchmarks/gb10-sage2-p2-vprep-analysis.json"], "active_source_location": null, "commit_hash": null, + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p2-vprep-analysis.json", "exists": true}, {"path": "benchmarks/gb10-sage2-vprep-randomized.json", "exists": true}, {"path": "benchmarks/gb10-sage2-vprep-canonical-repeat.json", "exists": true}], "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "validator-only candidate"}, + "metrics": {"result": "exact and faster in isolation, but missed absolute go threshold"}, + "correctness_evidence": ["FP8 bytes and FP32 scales documented exact across boundary lengths."], + "decision_rationale": "Absolute saving could not reliably clear the complete-block gate, so integration was skipped.", + "reproducer_commands": [], "timestamp": null, "evidence_missing": ["Complete-block and trajectory validation intentionally skipped"], + "production_behavior": "Callable wrapper/export and validator source remain, but production integration and dispatch are absent; production V preparation is unchanged.", "source_recovery": "Callable prepare_v source remains in sage2_entry.py and the validator remains in tools/validate_sage2_vprep.py; evidence artifacts are retained, but production integration is absent." + }, + { + "id": "sage2-p3-temporal-pair-final", + "name": "Finalized Sage2 P3 temporal pair", + "family": "sage2-finalized", "status": "performance_rejected", + "hypothesis": "Two warp pairs alternating QK/online-softmax and prior-tile PV can overlap INT and FP tensor work while preserving exact per-warp state.", + "implementation_strategy": "Keep private warp scores/state/accumulators and alternate pair phases with exact barriers.", + "source_locations": ["research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch", "research/sage2_temporal_pair/README.md", "benchmarks/gb10-sage2-p3-temporal-pair-analysis.json"], + "active_source_location": null, + "commit_hash": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "benchmark_artifacts": [{"path": "benchmarks/gb10-sage2-p3-temporal-pair-analysis.json", "exists": true}, {"path": "benchmarks/gb10-sage2-temporal-pair-short-20260825-p0.json", "exists": true}, {"path": "benchmarks/gb10-sage2-temporal-pair-timing-20260825-p0.json", "exists": true}], + "profiler_artifacts": [], + "environment": {"gpu": "NVIDIA GB10", "cuda": null, "driver": null, "pytorch": null, "triton": null, "container_image": null, "commit_context": "isolated SageAttention source commit recorded by experiment"}, + "metrics": {"result": "sanitizer-clean and exact, but effectively neutral and above gate"}, + "correctness_evidence": ["Thirteen short shapes and real block-24 output SHA documented exact; sanitizer errors zero."], + "decision_rationale": "Temporal separation did not produce material overlap or meet the absolute mainloop gate.", + "reproducer_commands": ["bash research/sage2_temporal_pair/run_spark_experiment.sh"], + "timestamp": null, + "evidence_missing": ["NCU, block integration, and trajectory validation intentionally skipped"], + "production_behavior": "No production integration; public Sage2 mainloop remains.", + "source_recovery": "Patch, runner, validator, README, and result artifacts are retained under research/ and benchmarks/." + } + ] +} diff --git a/research/fc2_activation_residual/README.md b/research/fc2_activation_residual/README.md new file mode 100644 index 0000000..8aebef7 --- /dev/null +++ b/research/fc2_activation_residual/README.md @@ -0,0 +1,5 @@ +# FC2 Activation-Residual Experiment + +The tree contains the recovered SwiGLU-to-FC2 NVFP4 producer under +`research/nvfp4_fused_variants/`, but no exact fused FC2 activation-residual or +residual-gate epilogue implementation. See `UNRECOVERABLE.md`. diff --git a/research/fc2_activation_residual/UNRECOVERABLE.md b/research/fc2_activation_residual/UNRECOVERABLE.md new file mode 100644 index 0000000..b1beac5 --- /dev/null +++ b/research/fc2_activation_residual/UNRECOVERABLE.md @@ -0,0 +1,7 @@ +# Unrecoverable FC2 Activation-Residual Source + +No exact FC2 activation-residual or residual-gate epilogue kernel source exists +in the worktree diff, untracked files, reachable history, reflogs, or dangling +objects. Current prose specifies the required BF16 operation order and identifies +FC2 Stream-K behavior as a future target, but that is design evidence rather +than recoverable implementation. diff --git a/research/nvfp4_fused_variants/README.md b/research/nvfp4_fused_variants/README.md new file mode 100644 index 0000000..b9a2674 --- /dev/null +++ b/research/nvfp4_fused_variants/README.md @@ -0,0 +1,18 @@ +# NVFP4 Fused Variant Recovery + +This capsule preserves the modulation-to-pack and SwiGLU-to-FC2-pack variants. + +- `patches/0001-python-wrappers.patch`: NVFP4 Python wrappers and opt-in linear + methods from the current worktree. +- `patches/0002-block-call-sites.patch`: current block call sites and fallback + rules. The patch also includes related fused elementwise call sites and is + explicitly not a kernel-only patch. +- `patches/0003-modulation-validator.patch`: standalone producer validator. +- `patches/0004-swiglu-validator.patch`: standalone producer validator. +- `patches/0005-elementwise-support.patch`: standalone untracked Triton + modulation and residual-gate support required by the block call-site patch. +- `../shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch`: + CUDA producer implementations and bindings; mixed with other experiments. + +These artifacts preserve source only. They do not enable environment flags or +change runtime/build defaults. diff --git a/research/nvfp4_fused_variants/patches/0001-python-wrappers.patch b/research/nvfp4_fused_variants/patches/0001-python-wrappers.patch new file mode 100644 index 0000000..865091a --- /dev/null +++ b/research/nvfp4_fused_variants/patches/0001-python-wrappers.patch @@ -0,0 +1,131 @@ +diff --git a/src/h3_blackwell_runtime/nvfp4.py b/src/h3_blackwell_runtime/nvfp4.py +index a479c87..2c2d0ef 100644 +--- a/src/h3_blackwell_runtime/nvfp4.py ++++ b/src/h3_blackwell_runtime/nvfp4.py +@@ -1,6 +1,7 @@ + """Standalone Blackwell NVFP4 linear adapter for Comfy-format checkpoints.""" + + import json ++import os + from dataclasses import dataclass + + import torch +@@ -33,7 +34,7 @@ def parse_quant_sidecar(sidecar: torch.Tensor) -> dict: + class Nvfp4Linear(DynamicLoraMixin, nn.Module): + """Execute a packed Comfy NVFP4 linear with Comfy Kitchen's CUDA 13 kernel.""" + +- def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16): ++ def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16, *, role: str = "other"): + super().__init__() + if tensors.weight.dtype != torch.uint8 or tensors.weight.ndim != 2: + raise ValueError("NVFP4 weights must be a rank-2 packed uint8 tensor.") +@@ -45,6 +46,7 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module): + self.in_features = tensors.in_features + self.out_features = tensors.out_features + self.output_dtype = output_dtype ++ self.role = role + self.full_precision_matrix_mult = tensors.full_precision_matrix_mult + self.register_buffer("weight", tensors.weight.contiguous(), persistent=False) + self.register_buffer("weight_scale", tensors.weight_scale.view(torch.float8_e4m3fn).contiguous(), persistent=False) +@@ -88,13 +90,91 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module): + else: + if x.dtype == torch.float32: + raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.") +- packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout") +- output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] ++ ring_output = None ++ if self.role == "h3_attn_qkv" and os.getenv("H3_CUTE_QKV_RING", "").lower() in { ++ "1", "true", "yes", "on", ++ }: ++ from .cute_qkv_ring import qkv_ring_linear ++ ++ ring_output = qkv_ring_linear(self, flat_x) ++ if ring_output is not None: ++ output = ring_output ++ elif os.getenv("H3_NVFP4_SCALE_BACKEND", "torch").lower() == "torch": ++ packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout") ++ output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] ++ else: ++ from .nvfp4_quant import vortex_quantize_nvfp4 ++ ++ packed_x = vortex_quantize_nvfp4(flat_x) ++ output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] + base = output.reshape(*original_shape, self.out_features) + return self._apply_lora(original_x, base) + +- +-def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear: ++ def forward_modulated( ++ self, ++ x: torch.Tensor, ++ shift: torch.Tensor, ++ scale: torch.Tensor, ++ row_index: torch.Tensor, ++ ) -> torch.Tensor: ++ """Fuse exact H3 modulation into activation packing for QKV or FC1.""" ++ if self.role not in {"h3_attn_qkv", "h3_mlp_fc1"}: ++ raise ValueError("modulated NVFP4 dispatch is restricted to H3 QKV and FC1") ++ if ( ++ self.full_precision_matrix_mult ++ or self.pre_quant_scale is not None ++ or self.active_lora is not None ++ or self.lora_strength != 0.0 ++ or torch.is_grad_enabled() ++ or x.requires_grad ++ ): ++ from .h3_fusion import fused_modulate_ ++ ++ return self(fused_modulate_(x, shift, scale, row_index)) ++ if x.shape != (row_index.numel(), self.in_features): ++ raise ValueError("modulated NVFP4 input and row index have incompatible shapes") ++ if x.dtype != torch.bfloat16 or not x.is_cuda or not x.is_contiguous(): ++ raise ValueError("modulated NVFP4 dispatch requires contiguous CUDA BF16 input") ++ ++ from .nvfp4_quant import vortex_quantize_modulated_nvfp4 ++ ++ packed_x = vortex_quantize_modulated_nvfp4(x, shift, scale, row_index) ++ bias = self.bias.to(x) if self.bias is not None else None ++ output = functional.linear(packed_x, self._packed_weight(), bias)[ ++ : x.shape[0], : self.out_features ++ ] ++ return output ++ ++ def forward_swiglu(self, gate_up: torch.Tensor) -> torch.Tensor: ++ """Fuse exact BF16 SwiGLU into activation packing for H3 FC2.""" ++ if self.role != "h3_mlp_fc2": ++ raise ValueError("SwiGLU NVFP4 dispatch is restricted to H3 FC2") ++ if gate_up.shape[-1] != self.in_features * 2: ++ raise ValueError("SwiGLU input width must be twice the FC2 input width") ++ if ( ++ self.full_precision_matrix_mult ++ or self.pre_quant_scale is not None ++ or self.active_lora is not None ++ or self.lora_strength != 0.0 ++ or torch.is_grad_enabled() ++ or gate_up.requires_grad ++ ): ++ gate, up = gate_up.chunk(2, dim=-1) ++ return self(torch.nn.functional.silu(gate).mul_(up)) ++ if gate_up.dtype != torch.bfloat16 or not gate_up.is_cuda or not gate_up.is_contiguous(): ++ raise ValueError("SwiGLU NVFP4 dispatch requires contiguous CUDA BF16 input") ++ ++ from .nvfp4_quant import vortex_quantize_swiglu_nvfp4 ++ ++ packed_x = vortex_quantize_swiglu_nvfp4(gate_up) ++ bias = self.bias.to(gate_up) if self.bias is not None else None ++ output = functional.linear(packed_x, self._packed_weight(), bias)[ ++ : gate_up.shape[0], : self.out_features ++ ] ++ return output ++ ++ ++def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16, role: str = "other") -> Nvfp4Linear: + """Load one Comfy-format NVFP4 linear from a safetensors tensor mapping.""" + sidecar_key = f"{prefix}.comfy_quant" + metadata = parse_quant_sidecar(tensors[sidecar_key]) +@@ -110,4 +190,4 @@ def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_d + in_features=in_features, + out_features=weight.shape[0], + ) +- return Nvfp4Linear(packed, output_dtype=output_dtype) ++ return Nvfp4Linear(packed, output_dtype=output_dtype, role=role) diff --git a/research/nvfp4_fused_variants/patches/0002-block-call-sites.patch b/research/nvfp4_fused_variants/patches/0002-block-call-sites.patch new file mode 100644 index 0000000..ba964da --- /dev/null +++ b/research/nvfp4_fused_variants/patches/0002-block-call-sites.patch @@ -0,0 +1,119 @@ +diff --git a/src/h3_blackwell_runtime/block.py b/src/h3_blackwell_runtime/block.py +index 6177e73..2c3650a 100644 +--- a/src/h3_blackwell_runtime/block.py ++++ b/src/h3_blackwell_runtime/block.py +@@ -1,5 +1,7 @@ + """Direct MiniMax H3 DiT block over the standalone Sage3 attention unit.""" + ++import os ++ + import torch + from torch import nn + from typing import TYPE_CHECKING +@@ -45,12 +47,15 @@ class H3SwiGLU(nn.Module): + self.fc2 = fc2 + self.chunks = 1 + self.chunk_threshold = 4096 ++ self.fused_nvfp4_swiglu = os.getenv("H3_NVFP4_SWIGLU_FUSION", "").lower() in { ++ "1", "true", "yes", "on", ++ } + + @classmethod + def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16): + return cls( +- checkpoint.nvfp4_linear(f"{prefix}.fc1", output_dtype=output_dtype), +- checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype), ++ checkpoint.nvfp4_linear(f"{prefix}.fc1", output_dtype=output_dtype, role="h3_mlp_fc1"), ++ checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype, role="h3_mlp_fc2"), + ) + + def forward( +@@ -71,7 +76,27 @@ class H3SwiGLU(nn.Module): + return self._forward_chunk(x) + + def _forward_chunk(self, x: torch.Tensor) -> torch.Tensor: +- gate, up = self.fc1(x).chunk(2, dim=-1) ++ gate_up = self.fc1(x) ++ if self.fused_nvfp4_swiglu: ++ return self.fc2.forward_swiglu(gate_up) ++ gate, up = gate_up.chunk(2, dim=-1) ++ return self.fc2(torch.nn.functional.silu(gate).mul_(up)) ++ ++ def forward_modulated( ++ self, ++ x: torch.Tensor, ++ shift: torch.Tensor, ++ scale: torch.Tensor, ++ row_index: torch.Tensor, ++ ) -> torch.Tensor: ++ if self.chunks != 1: ++ from .h3_fusion import fused_modulate_ ++ ++ return self(fused_modulate_(x, shift, scale, row_index)) ++ gate_up = self.fc1.forward_modulated(x, shift, scale, row_index) ++ if self.fused_nvfp4_swiglu: ++ return self.fc2.forward_swiglu(gate_up) ++ gate, up = gate_up.chunk(2, dim=-1) + return self.fc2(torch.nn.functional.silu(gate).mul_(up)) + + +@@ -101,6 +126,12 @@ class H3DiTBlock(nn.Module): + self.attention = attention + self.mlp = mlp + self.norm_eps = norm_eps ++ self.fused_elementwise = os.getenv("H3_FUSED_ELEMENTWISE", "").lower() in { ++ "1", "true", "yes", "on", ++ } ++ self.fused_nvfp4_modulation = os.getenv("H3_NVFP4_MODULATE_FUSION", "").lower() in { ++ "1", "true", "yes", "on", ++ } + self.register_buffer("norm1_weight", norm1_weight, persistent=False) + self.register_buffer("norm2_weight", norm2_weight, persistent=False) + +@@ -128,6 +159,46 @@ class H3DiTBlock(nn.Module): + sequence_parallel: "SequenceParallelContext | None" = None, + tensor_parallel: "SequenceParallelContext | None" = None, + ) -> torch.Tensor: ++ if self.fused_elementwise and x.is_cuda and x.dtype == torch.bfloat16 and not x.requires_grad: ++ from .h3_fusion import fused_gate_add_, fused_modulate_, segment_index ++ ++ row_index = segment_index(x.shape[0], segments, x.device) ++ if ( ++ self.fused_nvfp4_modulation ++ and sequence_parallel is None ++ and tensor_parallel is None ++ ): ++ h = rms_norm(x, self.norm1_weight, self.norm_eps) ++ x = fused_gate_add_( ++ x, ++ self.attention( ++ h, ++ rope_rotation, ++ modulation=(shift_msa, scale_msa, row_index), ++ ), ++ gate_msa, ++ row_index, ++ ) ++ h = rms_norm(x, self.norm2_weight, self.norm_eps) ++ return fused_gate_add_( ++ x, ++ self.mlp.forward_modulated(h, shift_mlp, scale_mlp, row_index), ++ gate_mlp, ++ row_index, ++ ) ++ h = fused_modulate_( ++ rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, row_index, ++ ) ++ x = fused_gate_add_( ++ x, ++ self.attention(h, rope_rotation, sequence_parallel, tensor_parallel), ++ gate_msa, ++ row_index, ++ ) ++ h = fused_modulate_( ++ rms_norm(x, self.norm2_weight, self.norm_eps), shift_mlp, scale_mlp, row_index, ++ ) ++ return fused_gate_add_(x, self.mlp(h, tensor_parallel), gate_mlp, row_index) + h = modulate_segments(rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, segments) + x = gate_segments( + x, diff --git a/research/nvfp4_fused_variants/patches/0003-modulation-validator.patch b/research/nvfp4_fused_variants/patches/0003-modulation-validator.patch new file mode 100644 index 0000000..cdefc94 --- /dev/null +++ b/research/nvfp4_fused_variants/patches/0003-modulation-validator.patch @@ -0,0 +1,146 @@ +diff --git a/tools/validate_nvfp4_modulate_producer.py b/tools/validate_nvfp4_modulate_producer.py +new file mode 100644 +index 0000000..a17b7f2 +--- /dev/null ++++ b/tools/validate_nvfp4_modulate_producer.py +@@ -0,0 +1,140 @@ ++"""Validate fused H3 modulation and native NVFP4 production.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import time ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.adaln import H3CurveAdaLN ++from h3_blackwell_runtime.attention import rms_norm ++from h3_blackwell_runtime.block import H3DiTBlock, modulate_segments ++from h3_blackwell_runtime.checkpoint import H3Checkpoint ++from h3_blackwell_runtime.h3_fusion import segment_index ++from h3_blackwell_runtime.nvfp4_quant import ( ++ nvfp4_activation_scale, ++ vortex_native_quantize_modulated_nvfp4, ++) ++from h3_blackwell_runtime.packing import H3PromptPacker ++from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas ++from h3_blackwell_runtime.t2v import random_av_latents ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--rows", type=int, default=2048) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ torch.manual_seed(args.seed) ++ checkpoint = H3Checkpoint(args.model_path, device=args.device) ++ block = H3DiTBlock.from_checkpoint(checkpoint, args.block_index, attention_backend="sage2").eval() ++ adaln = H3CurveAdaLN.from_checkpoint( ++ checkpoint, f"blocks.{args.block_index}.adaln_proj", ++ ).eval() ++ packer = H3PromptPacker(checkpoint) ++ video, audio, _ = random_av_latents( ++ args.width, args.height, args.frames, args.seed, device=args.device, ++ ) ++ sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] ++ native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) ++ text = torch.randn( ++ 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, ++ ) ++ hidden, timesteps, segments, _, _, _ = packer( ++ text, video, native_audio, _model_sigma(sigma), ++ ) ++ rows = min(args.rows, hidden.shape[0]) ++ hidden = hidden[:rows].contiguous() ++ clipped_segments = [] ++ for start, stop, table_row in segments: ++ if start >= rows: ++ break ++ clipped_segments.append((start, min(stop, rows), table_row)) ++ shift_msa, scale_msa, *_ = (value.detach() for value in adaln(timesteps)) ++ ++ with torch.inference_mode(): ++ normalized = rms_norm(hidden, block.norm1_weight, block.norm_eps) ++ materialized = modulate_segments( ++ normalized, shift_msa, scale_msa, clipped_segments, ++ ) ++ reference_scale = nvfp4_activation_scale(materialized).float() ++ import comfy_kitchen as ck ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ reference_qdata, reference_sfa = ck.quantize_nvfp4( ++ materialized, ++ reference_scale, ++ pad_16x=TensorCoreNVFP4Layout.get_padded_shape(tuple(materialized.shape)) ++ != tuple(materialized.shape), ++ ) ++ fused_scale, fused_qdata, fused_sfa = vortex_native_quantize_modulated_nvfp4( ++ normalized, ++ shift_msa.contiguous(), ++ scale_msa.contiguous(), ++ segment_index(rows, clipped_segments, hidden.device), ++ ) ++ torch.cuda.synchronize() ++ ++ def run_fused(): ++ return vortex_native_quantize_modulated_nvfp4( ++ normalized, ++ shift_msa.contiguous(), ++ scale_msa.contiguous(), ++ segment_index(rows, clipped_segments, hidden.device), ++ ) ++ ++ for _ in range(args.warmup): ++ run_fused() ++ fused_times = [] ++ for _ in range(args.iterations): ++ torch.cuda.synchronize() ++ started = time.perf_counter() ++ run_fused() ++ torch.cuda.synchronize() ++ fused_times.append(time.perf_counter() - started) ++ ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "block_index": args.block_index, ++ "rows": rows, ++ "width": hidden.shape[1], ++ "scale_equal": torch.equal(fused_scale, reference_scale), ++ "scale_reference": reference_scale.item(), ++ "scale_fused": fused_scale.item(), ++ "qdata_equal": torch.equal(fused_qdata, reference_qdata), ++ "qdata_differences": torch.count_nonzero(fused_qdata != reference_qdata).item(), ++ "sfa_equal": torch.equal(fused_sfa.view(torch.uint8), reference_sfa.view(torch.uint8)), ++ "sfa_differences": torch.count_nonzero( ++ fused_sfa.view(torch.uint8) != reference_sfa.view(torch.uint8) ++ ).item(), ++ "fused_producer_p50_ms": sorted(fused_times)[len(fused_times) // 2] * 1000.0, ++ } ++ report["equal"] = report["scale_equal"] and report["qdata_equal"] and report["sfa_equal"] ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if not report["equal"]: ++ raise RuntimeError("fused modulation producer is not byte-exact") ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/nvfp4_fused_variants/patches/0004-swiglu-validator.patch b/research/nvfp4_fused_variants/patches/0004-swiglu-validator.patch new file mode 100644 index 0000000..e5851cd --- /dev/null +++ b/research/nvfp4_fused_variants/patches/0004-swiglu-validator.patch @@ -0,0 +1,141 @@ +diff --git a/tools/validate_nvfp4_swiglu_producer.py b/tools/validate_nvfp4_swiglu_producer.py +new file mode 100644 +index 0000000..7650115 +--- /dev/null ++++ b/tools/validate_nvfp4_swiglu_producer.py +@@ -0,0 +1,135 @@ ++"""Validate fused SwiGLU and native NVFP4 production on real H3 FC1 output.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import time ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.adaln import H3CurveAdaLN ++from h3_blackwell_runtime.block import H3DiTBlock ++from h3_blackwell_runtime.checkpoint import H3Checkpoint ++from h3_blackwell_runtime.nvfp4_quant import ( ++ nvfp4_activation_scale, ++ vortex_native_quantize_swiglu_nvfp4, ++) ++from h3_blackwell_runtime.packing import H3PromptPacker ++from h3_blackwell_runtime.rope import h3_rope_rotation ++from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas ++from h3_blackwell_runtime.t2v import random_av_latents ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--block-index", type=int, required=True) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ torch.manual_seed(args.seed) ++ checkpoint = H3Checkpoint(args.model_path, device=args.device) ++ block = H3DiTBlock.from_checkpoint( ++ checkpoint, args.block_index, attention_backend="sage2", ++ ).eval() ++ adaln = H3CurveAdaLN.from_checkpoint( ++ checkpoint, f"blocks.{args.block_index}.adaln_proj", ++ ).eval() ++ packer = H3PromptPacker(checkpoint) ++ video, audio, _ = random_av_latents( ++ args.width, args.height, args.frames, args.seed, device=args.device, ++ ) ++ sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] ++ native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) ++ text = torch.randn( ++ 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, ++ ) ++ hidden, timesteps, segments, positions, _, _ = packer( ++ text, video, native_audio, _model_sigma(sigma), ++ ) ++ rotation = h3_rope_rotation( ++ positions.to(args.device), ++ checkpoint.tensor("rope.inv_freq", dtype=torch.float32), ++ hidden.dtype, ++ ) ++ modulation = tuple(value.detach() for value in adaln(timesteps)) ++ captured = [] ++ block.fused_nvfp4_modulation = False ++ hook = block.mlp.fc1.register_forward_hook( ++ lambda _module, _inputs, output: captured.append(output.detach()), ++ ) ++ with torch.inference_mode(): ++ block(hidden, rotation, *modulation, segments) ++ hook.remove() ++ gate_up = captured[-1] ++ ++ with torch.inference_mode(): ++ gate, up = gate_up.chunk(2, dim=-1) ++ materialized = torch.nn.functional.silu(gate).mul_(up) ++ reference_scale = nvfp4_activation_scale(materialized).float() ++ import comfy_kitchen as ck ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ reference_qdata, reference_sfa = ck.quantize_nvfp4( ++ materialized, ++ reference_scale, ++ pad_16x=TensorCoreNVFP4Layout.get_padded_shape(tuple(materialized.shape)) ++ != tuple(materialized.shape), ++ ) ++ actual_scale, actual_qdata, actual_sfa = vortex_native_quantize_swiglu_nvfp4( ++ gate_up, ++ ) ++ torch.cuda.synchronize() ++ for _ in range(args.warmup): ++ vortex_native_quantize_swiglu_nvfp4(gate_up) ++ times = [] ++ for _ in range(args.iterations): ++ torch.cuda.synchronize() ++ started = time.perf_counter() ++ vortex_native_quantize_swiglu_nvfp4(gate_up) ++ torch.cuda.synchronize() ++ times.append(time.perf_counter() - started) ++ ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "block_index": args.block_index, ++ "input_shape": list(gate_up.shape), ++ "output_shape": list(materialized.shape), ++ "scale_equal": torch.equal(actual_scale, reference_scale), ++ "scale_reference": reference_scale.item(), ++ "scale_actual": actual_scale.item(), ++ "qdata_differences": torch.count_nonzero(actual_qdata != reference_qdata).item(), ++ "sfa_differences": torch.count_nonzero( ++ actual_sfa.view(torch.uint8) != reference_sfa.view(torch.uint8) ++ ).item(), ++ "producer_p50_ms": sorted(times)[len(times) // 2] * 1000.0, ++ } ++ report["equal"] = ( ++ report["scale_equal"] ++ and report["qdata_differences"] == 0 ++ and report["sfa_differences"] == 0 ++ ) ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if not report["equal"]: ++ raise RuntimeError("fused SwiGLU producer is not byte-exact") ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/nvfp4_fused_variants/patches/0005-elementwise-support.patch b/research/nvfp4_fused_variants/patches/0005-elementwise-support.patch new file mode 100644 index 0000000..d7d4a89 --- /dev/null +++ b/research/nvfp4_fused_variants/patches/0005-elementwise-support.patch @@ -0,0 +1,212 @@ +diff --git a/src/h3_blackwell_runtime/h3_fusion.py b/src/h3_blackwell_runtime/h3_fusion.py +new file mode 100644 +index 0000000..0ee54db +--- /dev/null ++++ b/src/h3_blackwell_runtime/h3_fusion.py +@@ -0,0 +1,206 @@ ++"""Triton fusion for H3's segmented BF16 modulation and residual gates.""" ++ ++from __future__ import annotations ++ ++import torch ++import triton ++import triton.language as tl ++ ++ ++_TILE = 1024 ++_segment_cache_key = None ++_segment_cache_value = None ++ ++ ++@triton.jit ++def _round_bf16_fp32(value): ++ """Apply round-to-nearest-even BF16 precision while retaining FP32.""" ++ bits = value.to(tl.int32, bitcast=True) ++ bits = bits + 0x7FFF + ((bits >> 16) & 1) ++ return (bits & -65536).to(tl.float32, bitcast=True) ++ ++ ++@triton.jit ++def _modulate_kernel( ++ x_ptr, ++ scale_ptr, ++ shift_ptr, ++ row_index_ptr, ++ tokens, ++ width, ++ sx_t, ++ sx_d, ++ ss_t, ++ ss_d, ++ sh_t, ++ sh_d, ++ BLOCK: tl.constexpr, ++): ++ row = tl.program_id(0) ++ columns = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) ++ valid = (row < tokens) & (columns < width) ++ table_row = tl.load(row_index_ptr + row) ++ x = tl.load(x_ptr + row * sx_t + columns * sx_d, mask=valid).to(tl.float32) ++ scale = tl.load( ++ scale_ptr + table_row * ss_t + columns * ss_d, mask=valid, ++ ).to(tl.float32) ++ shift = tl.load( ++ shift_ptr + table_row * sh_t + columns * sh_d, mask=valid, ++ ).to(tl.float32) ++ ++ scale = _round_bf16_fp32(scale) ++ shift = _round_bf16_fp32(shift) ++ multiplied = _round_bf16_fp32(x * _round_bf16_fp32(1.0 + scale)) ++ result = _round_bf16_fp32(multiplied + shift) ++ tl.store(x_ptr + row * sx_t + columns * sx_d, result.to(tl.bfloat16), mask=valid) ++ ++ ++@triton.jit ++def _gate_add_kernel( ++ residual_ptr, ++ gate_ptr, ++ update_ptr, ++ row_index_ptr, ++ tokens, ++ width, ++ sr_t, ++ sr_d, ++ sg_t, ++ sg_d, ++ su_t, ++ su_d, ++ BLOCK: tl.constexpr, ++): ++ row = tl.program_id(0) ++ columns = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) ++ valid = (row < tokens) & (columns < width) ++ table_row = tl.load(row_index_ptr + row) ++ residual = tl.load( ++ residual_ptr + row * sr_t + columns * sr_d, mask=valid, ++ ).to(tl.float32) ++ update = tl.load( ++ update_ptr + row * su_t + columns * su_d, mask=valid, ++ ).to(tl.float32) ++ gate = tl.load( ++ gate_ptr + table_row * sg_t + columns * sg_d, mask=valid, ++ ).to(tl.float32) ++ ++ result = residual + update * _round_bf16_fp32(gate) ++ tl.store( ++ residual_ptr + row * sr_t + columns * sr_d, ++ result.to(tl.bfloat16), ++ mask=valid, ++ ) ++ ++ ++def segment_index( ++ tokens: int, ++ segments: list[tuple[int, int, int]], ++ device: torch.device, ++) -> torch.Tensor: ++ """Return one cached row-to-modulation-table lookup for a packed layout.""" ++ global _segment_cache_key, _segment_cache_value ++ normalized = tuple((int(start), int(stop), int(row)) for start, stop, row in segments) ++ key = (str(device), int(tokens), normalized) ++ if key == _segment_cache_key: ++ return _segment_cache_value ++ host = torch.empty(tokens, dtype=torch.int32) ++ cursor = 0 ++ for start, stop, row in normalized: ++ if start != cursor or stop < start or stop > tokens or row < 0: ++ raise ValueError("segments must be ordered, contiguous, and in range") ++ host[start:stop] = row ++ cursor = stop ++ if cursor != tokens: ++ raise ValueError("segments must cover every packed token") ++ _segment_cache_key = key ++ _segment_cache_value = host.to(device=device) ++ return _segment_cache_value ++ ++ ++def _validate_activation(tensor: torch.Tensor, name: str) -> None: ++ if tensor.ndim != 2: ++ raise ValueError(f"{name} must have shape [tokens, hidden]") ++ if tensor.device.type != "cuda" or tensor.dtype != torch.bfloat16: ++ raise TypeError(f"{name} must be a CUDA BF16 tensor") ++ if tensor.stride(1) != 1: ++ raise ValueError(f"{name}'s hidden dimension must be contiguous") ++ ++ ++def _validate_table(table: torch.Tensor, activation: torch.Tensor, name: str) -> None: ++ if table.ndim != 2 or table.shape[1] != activation.shape[1]: ++ raise ValueError(f"{name} must have shape [rows, {activation.shape[1]}]") ++ if table.device != activation.device or table.dtype not in {torch.bfloat16, torch.float32}: ++ raise TypeError(f"{name} must be CUDA BF16/FP32 on the activation device") ++ if table.stride(1) != 1: ++ raise ValueError(f"{name}'s hidden dimension must be contiguous") ++ ++ ++def fused_modulate_( ++ activation: torch.Tensor, ++ shift: torch.Tensor, ++ scale: torch.Tensor, ++ row_index: torch.Tensor, ++) -> torch.Tensor: ++ """Apply segmented scale/shift in place with eager-equivalent BF16 rounding.""" ++ _validate_activation(activation, "activation") ++ _validate_table(shift, activation, "shift") ++ _validate_table(scale, activation, "scale") ++ if row_index.shape != (activation.shape[0],) or row_index.dtype != torch.int32: ++ raise TypeError("row_index must be int32 with one entry per token") ++ if row_index.device != activation.device or not row_index.is_contiguous(): ++ raise TypeError("row_index must be contiguous on the activation device") ++ grid = (activation.shape[0], triton.cdiv(activation.shape[1], _TILE)) ++ _modulate_kernel[grid]( ++ activation, ++ scale, ++ shift, ++ row_index, ++ activation.shape[0], ++ activation.shape[1], ++ activation.stride(0), ++ activation.stride(1), ++ scale.stride(0), ++ scale.stride(1), ++ shift.stride(0), ++ shift.stride(1), ++ BLOCK=_TILE, ++ num_warps=4, ++ ) ++ return activation ++ ++ ++def fused_gate_add_( ++ residual: torch.Tensor, ++ update: torch.Tensor, ++ gate: torch.Tensor, ++ row_index: torch.Tensor, ++) -> torch.Tensor: ++ """Apply the segmented residual gate in place with addcmul-equivalent math.""" ++ _validate_activation(residual, "residual") ++ _validate_activation(update, "update") ++ if residual.shape != update.shape or residual.device != update.device: ++ raise ValueError("residual and update must share shape and device") ++ _validate_table(gate, residual, "gate") ++ if row_index.shape != (residual.shape[0],) or row_index.dtype != torch.int32: ++ raise TypeError("row_index must be int32 with one entry per token") ++ if row_index.device != residual.device or not row_index.is_contiguous(): ++ raise TypeError("row_index must be contiguous on the residual device") ++ grid = (residual.shape[0], triton.cdiv(residual.shape[1], _TILE)) ++ _gate_add_kernel[grid]( ++ residual, ++ gate, ++ update, ++ row_index, ++ residual.shape[0], ++ residual.shape[1], ++ residual.stride(0), ++ residual.stride(1), ++ gate.stride(0), ++ gate.stride(1), ++ update.stride(0), ++ update.stride(1), ++ BLOCK=_TILE, ++ num_warps=4, ++ ) ++ return residual diff --git a/research/nvfp4_lora_materialization/README.md b/research/nvfp4_lora_materialization/README.md new file mode 100644 index 0000000..f419fbd --- /dev/null +++ b/research/nvfp4_lora_materialization/README.md @@ -0,0 +1,5 @@ +# NVFP4 LoRA Materialization Variant + +The worktree records a rejected variant that materialized optional BF16 LoRA +input while fused producers packed NVFP4. The exact implementation source is not +present. See `UNRECOVERABLE.md`. diff --git a/research/nvfp4_lora_materialization/UNRECOVERABLE.md b/research/nvfp4_lora_materialization/UNRECOVERABLE.md new file mode 100644 index 0000000..dc0dd6f --- /dev/null +++ b/research/nvfp4_lora_materialization/UNRECOVERABLE.md @@ -0,0 +1,9 @@ +# Unrecoverable LoRA Variant Source + +The exact LoRA fused-producer/materialization implementation is absent from the +current source diff, untracked files, reachable history, reflogs, and dangling +Git objects. Existing production fallback checks reject active LoRA in fused +producer methods, but they are not the removed materialization candidate. + +Only benchmark and prose evidence remains. Reconstructing source from that +evidence would not be exact recovery. diff --git a/research/qkv_layout_variants/README.md b/research/qkv_layout_variants/README.md new file mode 100644 index 0000000..8a8a06c --- /dev/null +++ b/research/qkv_layout_variants/README.md @@ -0,0 +1,13 @@ +# QKV Layout, Swizzle, and Wide-Store Recovery + +The recoverable source is split between the shared CUDA patch and a focused +Python dispatch patch: + +- `patches/0001-strided-nhd-dispatch.patch`: current `attention.py` changes for + projection-strided NHD Sage2 and contiguous token-major output. +- `../shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch`: + QKV split/layout kernels, vectorized 16-byte stores, generalized NVFP4 scale + swizzle, and associated extension binding. This patch also contains other + requested CUDA experiments and is not isolated. + +No experimental dispatch or build default is added by this capsule. diff --git a/research/qkv_layout_variants/patches/0001-strided-nhd-dispatch.patch b/research/qkv_layout_variants/patches/0001-strided-nhd-dispatch.patch new file mode 100644 index 0000000..a3af75c --- /dev/null +++ b/research/qkv_layout_variants/patches/0001-strided-nhd-dispatch.patch @@ -0,0 +1,61 @@ +diff --git a/src/h3_blackwell_runtime/attention.py b/src/h3_blackwell_runtime/attention.py +index 7b7560d..a5372f8 100644 +--- a/src/h3_blackwell_runtime/attention.py ++++ b/src/h3_blackwell_runtime/attention.py +@@ -169,6 +169,13 @@ def run_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, backend: + raise ValueError(f"Unsupported H3 attention backend: {backend}") + + ++def run_sage_attention_nhd(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor: ++ """Run Sage2 directly on projection-strided NHD Q/K/V views.""" ++ from sageattention import sageattn ++ ++ return sageattn(q, k, v, is_causal=False, tensor_layout="NHD", smooth_k=False) ++ ++ + def apply_split_half_rope(x: torch.Tensor, rotation: torch.Tensor) -> torch.Tensor: + """Apply H3's split-half rotary table to `[batch, sequence, heads, dim]`.""" + rotated_width = rotation.shape[-3] * 2 +@@ -231,8 +238,8 @@ class H3SageAttention(nn.Module): + @classmethod + def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16, backend: str = DEFAULT_ATTENTION_BACKEND): + return cls( +- checkpoint.nvfp4_linear(f"{prefix}.qkv_proj", output_dtype=output_dtype), +- checkpoint.nvfp4_linear(f"{prefix}.out_proj", output_dtype=output_dtype), ++ checkpoint.nvfp4_linear(f"{prefix}.qkv_proj", output_dtype=output_dtype, role="h3_attn_qkv"), ++ checkpoint.nvfp4_linear(f"{prefix}.out_proj", output_dtype=output_dtype, role="h3_attn_out"), + checkpoint.tensor(f"{prefix}.q_norm.weight", dtype=output_dtype), + checkpoint.tensor(f"{prefix}.k_norm.weight", dtype=output_dtype), + backend=backend, +@@ -244,6 +251,7 @@ class H3SageAttention(nn.Module): + rope_rotation: torch.Tensor, + sequence_parallel: "SequenceParallelContext | None" = None, + tensor_parallel: "SequenceParallelContext | None" = None, ++ modulation: tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None = None, + ) -> torch.Tensor: + if x.ndim != 2: + raise ValueError("H3 attention expects `[sequence, hidden]` input.") +@@ -253,13 +261,22 @@ class H3SageAttention(nn.Module): + return self._forward_tensor_parallel(x, rope_rotation, tensor_parallel) + sequence = x.shape[0] + inner = self.heads * self.head_dim +- qkv = self.qkv_proj(x) ++ qkv = self.qkv_proj.forward_modulated(x, *modulation) if modulation is not None else self.qkv_proj(x) + q, k, v = qkv.split(inner, dim=-1) + q = q.view(1, sequence, self.heads, self.head_dim) + k = k.view(1, sequence, self.heads, self.head_dim) + v = v.view(1, sequence, self.heads, self.head_dim) + + q, k = rms_rope_split_half_(q, k, rope_rotation, self.q_norm_weight, self.k_norm_weight, self.eps) ++ if ( ++ sequence_parallel is None ++ and self.backend == "sage2" ++ and os.getenv("H3_SAGE_QKV_LAYOUT", "hnd").lower() == "strided_nhd" ++ ): ++ out = run_sage_attention_nhd(q, k, v) ++ if not out.is_contiguous(): ++ raise RuntimeError("Sage2 NHD output must be contiguous for zero-copy output projection") ++ return self.out_proj(out.reshape(sequence, inner)) + if sequence_parallel is not None: + q, k, v = sequence_parallel.seq_to_heads(q, k, v) + if self.backend == "sol_attn": diff --git a/research/sage2_direct_vprep/README.md b/research/sage2_direct_vprep/README.md new file mode 100644 index 0000000..1d64034 --- /dev/null +++ b/research/sage2_direct_vprep/README.md @@ -0,0 +1,15 @@ +# Sage2 P2 Direct V Preparation Recovery + +Recovered implementation sources are preserved without production integration. + +- `patches/0001-vprep-validation.patch`: standalone untracked boundary-length + validator. +- `patches/0002-scheduler-decomposition.patch`: standalone untracked Sage2 + decomposition harness that exercises direct V preparation. +- `../shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch`: + three-stage CUDA implementation and binding. The shared patch is intentionally + mixed and must not be treated as an isolated P2 patch. + +The candidate emitted Sage2's padded/permuted E4M3 V and FP32 channel scales +directly from projection-strided NHD BF16 V. It was rejected at its isolated +absolute-time gate. diff --git a/research/sage2_direct_vprep/patches/0001-vprep-validation.patch b/research/sage2_direct_vprep/patches/0001-vprep-validation.patch new file mode 100644 index 0000000..adee3dd --- /dev/null +++ b/research/sage2_direct_vprep/patches/0001-vprep-validation.patch @@ -0,0 +1,124 @@ +diff --git a/tools/validate_sage2_vprep.py b/tools/validate_sage2_vprep.py +new file mode 100644 +index 0000000..e644d63 +--- /dev/null ++++ b/tools/validate_sage2_vprep.py +@@ -0,0 +1,118 @@ ++"""Validate Vortex direct Sage2 V preparation against SageAttention 2.2.0.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.sage2_entry import prepare_v ++from profile_attention_path import summarize ++ ++ ++def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: ++ delta = actual.float() - expected.float() ++ return { ++ "equal": torch.equal(actual, expected), ++ "different_elements": int(torch.count_nonzero(actual != expected).item()), ++ "max_abs": delta.abs().max().item() if delta.numel() else 0.0, ++ "mean_abs": delta.abs().mean().item() if delta.numel() else 0.0, ++ } ++ ++ ++def measure(fn, *, warmup: int, iterations: int) -> dict: ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ samples = [] ++ for _ in range(iterations): ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ fn() ++ finished.record() ++ finished.synchronize() ++ samples.append(started.elapsed_time(finished) / 1000.0) ++ return summarize(samples) ++ ++ ++def baseline(v: torch.Tensor): ++ import sageattention.core as sage_core ++ ++ return sage_core.per_channel_fp8( ++ v, tensor_layout="NHD", scale_max=2.25, smooth_v=False, ++ ) ++ ++ ++def run_case(sequence: int, heads: int, seed: int, warmup: int, iterations: int) -> dict: ++ generator = torch.Generator(device="cuda").manual_seed(seed) ++ storage = torch.randn( ++ (sequence, heads * 128 * 3), ++ generator=generator, ++ device="cuda", ++ dtype=torch.bfloat16, ++ ) ++ v = storage[:, heads * 128 * 2 :].view(1, sequence, heads, 128) ++ reference_fp8, reference_scale, _ = baseline(v) ++ candidate_fp8, candidate_scale = prepare_v(v) ++ torch.cuda.synchronize() ++ result = { ++ "sequence": sequence, ++ "heads": heads, ++ "stride": list(v.stride()), ++ "fp8": difference(candidate_fp8, reference_fp8), ++ "scale": difference(candidate_scale, reference_scale), ++ } ++ if sequence >= 1024: ++ result["baseline_timing"] = measure( ++ lambda: baseline(v), warmup=warmup, iterations=iterations, ++ ) ++ result["candidate_timing"] = measure( ++ lambda: prepare_v(v), warmup=warmup, iterations=iterations, ++ ) ++ return result ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument( ++ "--lengths", ++ nargs="+", ++ type=int, ++ default=(1, 31, 32, 33, 63, 64, 65, 127, 128, 129, 37760, 37761, 37810), ++ ) ++ parser.add_argument("--heads", type=int, default=56) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ cases = [ ++ run_case(length, args.heads, args.seed + index, args.warmup, args.iterations) ++ for index, length in enumerate(args.lengths) ++ ] ++ report = { ++ "status": "pass" if all(case["fp8"]["equal"] and case["scale"]["equal"] for case in cases) else "fail", ++ "contract": { ++ "tensor_layout": "NHD", ++ "head_dim": 128, ++ "scale_max": 2.25, ++ "output": "Sage2 padded/permuted E4M3 V and FP32 per-channel scale", ++ }, ++ "cases": cases, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if report["status"] != "pass": ++ raise RuntimeError("Vortex Sage2 V preparation did not match SageAttention 2.2.0") ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/sage2_direct_vprep/patches/0002-scheduler-decomposition.patch b/research/sage2_direct_vprep/patches/0002-scheduler-decomposition.patch new file mode 100644 index 0000000..311556c --- /dev/null +++ b/research/sage2_direct_vprep/patches/0002-scheduler-decomposition.patch @@ -0,0 +1,413 @@ +diff --git a/tools/profile_sage2_scheduler.py b/tools/profile_sage2_scheduler.py +new file mode 100644 +index 0000000..03130e1 +--- /dev/null ++++ b/tools/profile_sage2_scheduler.py +@@ -0,0 +1,407 @@ ++"""Profile exact SageAttention2 preparation, mainloop, and tail scheduling on real H3 tensors.""" ++ ++from __future__ import annotations ++ ++import argparse ++import hashlib ++import json ++import math ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.sage2_entry import prepare_v ++from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize ++ ++ ++CTA_Q = 128 ++CTA_K = 64 ++WARP_Q = 32 ++WARP_K = 64 ++V_SCALE_MAX = 2.25 ++ ++ ++def tensor_sha256(value: torch.Tensor) -> str: ++ host_bytes = value.detach().contiguous().view(torch.uint8).cpu().numpy() ++ return hashlib.sha256(memoryview(host_bytes)).hexdigest() ++ ++ ++def event_measure(fn, *, warmup: int, iterations: int): ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ values = [] ++ result = None ++ for _ in range(iterations): ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ result = fn() ++ finished.record() ++ finished.synchronize() ++ values.append(started.elapsed_time(finished) / 1000.0) ++ return summarize(values), result ++ ++ ++def quantize_qk(q: torch.Tensor, k: torch.Tensor, km: torch.Tensor): ++ import sageattention.core as sage_core ++ ++ return sage_core.per_warp_int8_cuda( ++ q, ++ k, ++ km, ++ BLKQ=CTA_Q, ++ WARPQ=WARP_Q, ++ BLKK=CTA_K, ++ tensor_layout="NHD", ++ ) ++ ++ ++def quantize_v(v: torch.Tensor): ++ import sageattention.core as sage_core ++ ++ return sage_core.per_channel_fp8( ++ v, ++ tensor_layout="NHD", ++ scale_max=V_SCALE_MAX, ++ smooth_v=False, ++ ) ++ ++ ++def run_mainloop( ++ q_int8: torch.Tensor, ++ k_int8: torch.Tensor, ++ v_fp8: torch.Tensor, ++ q_scale: torch.Tensor, ++ k_scale: torch.Tensor, ++ v_scale: torch.Tensor, ++ output: torch.Tensor, ++) -> torch.Tensor: ++ import sageattention.core as sage_core ++ ++ sage_core.sm89_compile.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( ++ q_int8, ++ k_int8, ++ v_fp8, ++ output, ++ q_scale, ++ k_scale, ++ v_scale, ++ 0, ++ 0, ++ 2, ++ output.shape[-1] ** -0.5, ++ 0, ++ ) ++ return output ++ ++ ++def prepare_quantized(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor): ++ km = k.mean(dim=1, keepdim=True) ++ q_int8, q_scale, k_int8, k_scale = quantize_qk(q, k, km) ++ v_fp8, v_scale, _ = quantize_v(v) ++ output = torch.empty(q.shape, dtype=q.dtype, device=q.device) ++ return km, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output ++ ++ ++def tail_row( ++ q: torch.Tensor, ++ k: torch.Tensor, ++ v: torch.Tensor, ++ *, ++ warmup: int, ++ iterations: int, ++) -> dict: ++ quantized = prepare_quantized(q, k, v) ++ _, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output = quantized ++ timing, result = event_measure( ++ lambda: run_mainloop( ++ q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, ++ ), ++ warmup=warmup, ++ iterations=iterations, ++ ) ++ q_len = q.shape[1] ++ kv_len = k.shape[1] ++ q_ctas = math.ceil(q_len / CTA_Q) ++ k_iterations = math.ceil(kv_len / CTA_K) ++ return { ++ "q_len": q_len, ++ "kv_len": kv_len, ++ "q_ctas_per_head": q_ctas, ++ "k_iterations_per_cta": k_iterations, ++ "q_tail_rows": q_len % CTA_Q, ++ "k_tail_rows": kv_len % CTA_K, ++ "scheduled_q_rows": q_ctas * CTA_Q, ++ "q_row_efficiency": q_len / (q_ctas * CTA_Q), ++ "mainloop": timing, ++ "checksum": result.float().sum().item(), ++ } ++ ++ ++def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: ++ delta = actual.float() - expected.float() ++ return { ++ "equal": torch.equal(actual, expected), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ } ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--attention", default="sage2", choices=("sage2",)) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--tail-iterations", type=int, default=5) ++ parser.add_argument("--skip-tail-study", action="store_true") ++ parser.add_argument("--cuda-profiler-capture", action="store_true") ++ parser.add_argument("--expected-output-sha256") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ block, hidden, rotation, _segments, metadata = representative_attention_inputs(args) ++ with torch.inference_mode(): ++ q, k, v, _ = prepare_qkv(block, hidden, rotation, None) ++ km_timing, km = event_measure( ++ lambda: k.mean(dim=1, keepdim=True), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ qk_timing, qk = event_measure( ++ lambda: quantize_qk(q, k, km), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ q_int8, q_scale, k_int8, k_scale = qk ++ import sageattention.core as sage_core ++ import sageattention.quant as sage_quant ++ ++ q_int8_probe = torch.empty(q.shape, dtype=torch.int8, device=q.device) ++ q_scale_probe = torch.empty_like(q_scale) ++ q_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.quant_per_warp_int8_cuda( ++ q, q_int8_probe, q_scale_probe, CTA_Q, WARP_Q, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ k_int8_probe = torch.empty(k.shape, dtype=torch.int8, device=k.device) ++ k_scale_probe = torch.empty_like(k_scale) ++ k_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( ++ k, km.squeeze(1), k_int8_probe, k_scale_probe, CTA_K, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ padded_k = math.ceil(v.shape[1] / CTA_K) * CTA_K ++ v_transposed = torch.empty( ++ (v.shape[0], v.shape[3], v.shape[2], padded_k), ++ dtype=v.dtype, ++ device=v.device, ++ ) ++ v_transpose_timing, _ = event_measure( ++ lambda: sage_quant._fused.transpose_pad_permute_cuda(v, v_transposed, 0), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_fp8_probe = torch.empty_like(v_transposed, dtype=torch.float8_e4m3fn) ++ v_scale_probe = torch.empty( ++ (v.shape[0], v.shape[2], v.shape[3]), ++ dtype=torch.float32, ++ device=v.device, ++ ) ++ v_scale_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.scale_fuse_quant_cuda( ++ v_transposed, v_fp8_probe, v_scale_probe, v.shape[1], V_SCALE_MAX, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_timing, vq = event_measure( ++ lambda: quantize_v(v), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_fp8, v_scale, _ = vq ++ candidate_v_timing, candidate_vq = event_measure( ++ lambda: prepare_v(v, scale_max=V_SCALE_MAX), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ candidate_v_fp8, candidate_v_scale = candidate_vq ++ output = torch.empty(q.shape, dtype=q.dtype, device=q.device) ++ mainloop_fn = lambda: run_mainloop( ++ q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, ++ ) ++ ++ if args.cuda_profiler_capture: ++ for _ in range(args.warmup): ++ mainloop_fn() ++ torch.cuda.synchronize() ++ torch.cuda.cudart().cudaProfilerStart() ++ captured = mainloop_fn() ++ torch.cuda.synchronize() ++ torch.cuda.cudart().cudaProfilerStop() ++ report = { ++ "metadata": metadata, ++ "capture": "one unchanged prequantized Sage2 mainloop", ++ "q_shape": list(q.shape), ++ "k_shape": list(k.shape), ++ "v_shape": list(v.shape), ++ "checksum": captured.float().sum().item(), ++ "scheduler": { ++ "cta_q": CTA_Q, ++ "cta_k": CTA_K, ++ "warp_q": WARP_Q, ++ "warp_k": WARP_K, ++ "warps_per_cta": 4, ++ "threads_per_cta": 128, ++ "dynamic_shared_memory_bytes": 32768, ++ "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), ++ "heads": q.shape[2], ++ "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], ++ "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), ++ "explicit_pipeline_stages": 2, ++ }, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ return ++ ++ mainloop_timing, manual_output = event_measure( ++ mainloop_fn, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ candidate_output = torch.empty_like(output) ++ run_mainloop( ++ q_int8, ++ k_int8, ++ candidate_v_fp8, ++ q_scale, ++ k_scale, ++ candidate_v_scale, ++ candidate_output, ++ ) ++ torch.cuda.synchronize() ++ reference = __import__("sageattention").sageattn( ++ q, k, v, tensor_layout="NHD", is_causal=False, smooth_k=False, ++ ) ++ torch.cuda.synchronize() ++ output_sha256 = tensor_sha256(manual_output) ++ expected_output_matches = ( ++ args.expected_output_sha256 is None ++ or output_sha256 == args.expected_output_sha256 ++ ) ++ ++ tail_study = [] ++ if not args.skip_tail_study: ++ q_lengths = sorted({ ++ (q.shape[1] // CTA_Q) * CTA_Q, ++ (q.shape[1] // CTA_Q) * CTA_Q + 1, ++ q.shape[1], ++ }) ++ kv_lengths = sorted({ ++ (k.shape[1] // CTA_K) * CTA_K, ++ (k.shape[1] // CTA_K) * CTA_K + 1, ++ k.shape[1], ++ }) ++ for q_len in q_lengths: ++ tail_study.append({ ++ "sweep": "q_tail_fixed_kv", ++ **tail_row( ++ q[:, :q_len], k, v, ++ warmup=args.warmup, ++ iterations=args.tail_iterations, ++ ), ++ }) ++ for kv_len in kv_lengths: ++ tail_study.append({ ++ "sweep": "kv_tail_fixed_q", ++ **tail_row( ++ q, k[:, :kv_len], v[:, :kv_len], ++ warmup=args.warmup, ++ iterations=args.tail_iterations, ++ ), ++ }) ++ ++ report = { ++ "metadata": metadata, ++ "q_shape": list(q.shape), ++ "k_shape": list(k.shape), ++ "v_shape": list(v.shape), ++ "warmup": args.warmup, ++ "iterations": args.iterations, ++ "phase_timings": { ++ "k_mean_and_smoothing_preparation": km_timing, ++ "qk_int8_quantization": qk_timing, ++ "q_int8_quantization": q_quant_timing, ++ "k_int8_subtract_mean_quantization": k_quant_timing, ++ "v_fp8_transpose_scale_quantization": v_timing, ++ "v_transpose_pad_permute": v_transpose_timing, ++ "v_scale_fp8_quantization": v_scale_quant_timing, ++ "vortex_direct_v_fp8_preparation": candidate_v_timing, ++ "fused_mainloop": mainloop_timing, ++ }, ++ "fused_mainloop_phases": { ++ "int8_qk": "fused inside qk_int_sv_f8_attn_kernel", ++ "scale_application": "fused inside qk_int_sv_f8_attn_kernel", ++ "online_softmax": "fused inside qk_int_sv_f8_attn_kernel", ++ "pv_accumulation": "fused inside qk_int_sv_f8_attn_kernel", ++ "final_normalization_and_output": "fused inside qk_int_sv_f8_attn_kernel", ++ "timing_policy": "Do not assign independent wall time without changing the exact kernel schedule; use source-correlated hardware counters.", ++ }, ++ "scheduler": { ++ "cta_q": CTA_Q, ++ "cta_k": CTA_K, ++ "warp_q": WARP_Q, ++ "warp_k": WARP_K, ++ "warps_per_cta": 4, ++ "threads_per_cta": 128, ++ "dynamic_shared_memory_bytes": 32768, ++ "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), ++ "heads": q.shape[2], ++ "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], ++ "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), ++ "q_tail_rows": q.shape[1] % CTA_Q, ++ "k_tail_rows": k.shape[1] % CTA_K, ++ "explicit_pipeline_stages": 2, ++ }, ++ "manual_decomposition_vs_public_sage2": difference(manual_output, reference), ++ "vortex_v_fp8_vs_sage2": difference(candidate_v_fp8, v_fp8), ++ "vortex_v_scale_vs_sage2": difference(candidate_v_scale, v_scale), ++ "vortex_v_mainloop_vs_sage2": difference(candidate_output, manual_output), ++ "manual_checksum": manual_output.float().sum().item(), ++ "reference_checksum": reference.float().sum().item(), ++ "output_sha256": output_sha256, ++ "expected_output_sha256": args.expected_output_sha256, ++ "expected_output_matches": expected_output_matches, ++ "tail_study": tail_study, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if not expected_output_matches: ++ raise RuntimeError( ++ f"output SHA256 mismatch: expected {args.expected_output_sha256}, got {output_sha256}" ++ ) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/sage2_entry_fusion/README.md b/research/sage2_entry_fusion/README.md new file mode 100644 index 0000000..251d3b7 --- /dev/null +++ b/research/sage2_entry_fusion/README.md @@ -0,0 +1,16 @@ +# Sage2 P1 Entry Fusion Recovery + +Recovered implementation sources are preserved as generated patches and are not +connected to production dispatch. + +- `patches/0001-sage2-entry-wrapper.patch`: standalone untracked wrapper from + `src/h3_blackwell_runtime/sage2_entry.py`. +- `patches/0002-entry-validation.patch`: standalone untracked randomized and + real-block validation harness. +- `../shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch`: + exact CUDA implementation and extension binding. This shared patch is mixed + with other requested experiments and is not isolated. + +The worktree documentation records exact randomized and block 0/24/49 parity, +but rejection at the complete-block performance gate. No runtime branch is +reconnected here. diff --git a/research/sage2_entry_fusion/patches/0001-sage2-entry-wrapper.patch b/research/sage2_entry_fusion/patches/0001-sage2-entry-wrapper.patch new file mode 100644 index 0000000..1cb962c --- /dev/null +++ b/research/sage2_entry_fusion/patches/0001-sage2-entry-wrapper.patch @@ -0,0 +1,87 @@ +diff --git a/src/h3_blackwell_runtime/sage2_entry.py b/src/h3_blackwell_runtime/sage2_entry.py +new file mode 100644 +index 0000000..215d4d1 +--- /dev/null ++++ b/src/h3_blackwell_runtime/sage2_entry.py +@@ -0,0 +1,81 @@ ++"""Exact fused entry preparation for the pinned SageAttention 2.2.0 path.""" ++ ++from __future__ import annotations ++ ++import math ++ ++import torch ++ ++ ++def prepare_qk( ++ q: torch.Tensor, ++ k: torch.Tensor, ++ rotation: torch.Tensor, ++ q_weight: torch.Tensor, ++ k_weight: torch.Tensor, ++ epsilon: float, ++ *, ++ materialize_q: bool = False, ++) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: ++ """Prepare Q/K in place and emit Sage2's exact per-warp Q representation.""" ++ from .nvfp4_quant import _vortex_scale_extension ++ ++ return tuple( ++ _vortex_scale_extension().sage2_prepare_qk( ++ q, k, rotation, q_weight, k_weight, epsilon, materialize_q, ++ ) ++ ) ++ ++ ++def prepare_v(v: torch.Tensor, *, scale_max: float = 2.25) -> tuple[torch.Tensor, torch.Tensor]: ++ """Emit Sage2's FP8 V tensor and per-channel scale without a BF16 transpose tensor.""" ++ from .nvfp4_quant import _vortex_scale_extension ++ ++ return tuple(_vortex_scale_extension().sage2_prepare_v(v, scale_max)) ++ ++ ++def attention_nhd( ++ q: torch.Tensor, ++ k: torch.Tensor, ++ v: torch.Tensor, ++ rotation: torch.Tensor, ++ q_weight: torch.Tensor, ++ k_weight: torch.Tensor, ++ epsilon: float, ++) -> torch.Tensor: ++ """Run the unchanged Sage2 mainloop after fused strided-NHD entry preparation.""" ++ import sageattention.core as sage_core ++ import sageattention.quant as sage_quant ++ ++ q_int8, q_scale, _ = prepare_qk( ++ q, k, rotation, q_weight, k_weight, epsilon, materialize_q=False, ++ ) ++ k_mean = k.mean(dim=1, keepdim=True) ++ k_int8 = torch.empty(k.shape, dtype=torch.int8, device=k.device) ++ k_scale = torch.empty( ++ (k.shape[0], k.shape[2], math.ceil(k.shape[1] / 64)), ++ dtype=torch.float32, ++ device=k.device, ++ ) ++ sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( ++ k, k_mean.squeeze(1), k_int8, k_scale, 64, 0, ++ ) ++ v_fp8, v_scale, _ = sage_core.per_channel_fp8( ++ v, tensor_layout="NHD", scale_max=2.25, smooth_v=False, ++ ) ++ output = torch.empty(q.shape, dtype=q.dtype, device=q.device) ++ sage_core.sm89_compile.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( ++ q_int8, ++ k_int8, ++ v_fp8, ++ output, ++ q_scale, ++ k_scale, ++ v_scale, ++ 0, ++ 0, ++ 2, ++ output.shape[-1] ** -0.5, ++ 0, ++ ) ++ return output diff --git a/research/sage2_entry_fusion/patches/0002-entry-validation.patch b/research/sage2_entry_fusion/patches/0002-entry-validation.patch new file mode 100644 index 0000000..4ff2a5e --- /dev/null +++ b/research/sage2_entry_fusion/patches/0002-entry-validation.patch @@ -0,0 +1,341 @@ +diff --git a/tools/validate_sage2_entry_fusion.py b/tools/validate_sage2_entry_fusion.py +new file mode 100644 +index 0000000..b19ae7a +--- /dev/null ++++ b/tools/validate_sage2_entry_fusion.py +@@ -0,0 +1,335 @@ ++"""Validate exact fused Sage2 entry preparation on randomized and real H3 tensors.""" ++ ++from __future__ import annotations ++ ++import argparse ++import json ++import math ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.adaln import H3CurveAdaLN ++from h3_blackwell_runtime.attention import rms_norm, rms_rope_split_half_ ++from h3_blackwell_runtime.block import H3DiTBlock, modulate_segments ++from h3_blackwell_runtime.checkpoint import H3Checkpoint ++from h3_blackwell_runtime.packing import H3PromptPacker ++from h3_blackwell_runtime.rope import h3_rope_rotation ++from h3_blackwell_runtime.sage2_entry import prepare_qk ++from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas ++from h3_blackwell_runtime.t2v import random_av_latents ++from profile_attention_path import summarize ++from profile_sage2_scheduler import run_mainloop ++ ++ ++def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: ++ delta = actual.float() - expected.float() ++ result = { ++ "equal": torch.equal(actual, expected), ++ "different_elements": int(torch.count_nonzero(actual != expected).item()), ++ "max_abs": delta.abs().max().item() if delta.numel() else 0.0, ++ "mean_abs": delta.abs().mean().item() if delta.numel() else 0.0, ++ } ++ if not result["equal"] and result["different_elements"] <= 8: ++ indices = torch.nonzero(actual != expected, as_tuple=False) ++ result["mismatches"] = [ ++ { ++ "index": index.tolist(), ++ "actual": actual[tuple(index)].item(), ++ "expected": expected[tuple(index)].item(), ++ } ++ for index in indices ++ ] ++ return result ++ ++ ++def measure(fn, warmup: int, iterations: int) -> dict: ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ before_allocated = torch.cuda.memory_allocated() ++ before_reserved = torch.cuda.memory_reserved() ++ torch.cuda.reset_peak_memory_stats() ++ samples = [] ++ for _ in range(iterations): ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ fn() ++ finished.record() ++ finished.synchronize() ++ samples.append(started.elapsed_time(finished) / 1000.0) ++ return { ++ "timing": summarize(samples), ++ "allocated_bytes_before": before_allocated, ++ "reserved_bytes_before": before_reserved, ++ "peak_allocated_bytes": torch.cuda.max_memory_allocated(), ++ "peak_reserved_bytes": torch.cuda.max_memory_reserved(), ++ "peak_allocated_delta_bytes": torch.cuda.max_memory_allocated() - before_allocated, ++ "peak_reserved_delta_bytes": torch.cuda.max_memory_reserved() - before_reserved, ++ } ++ ++ ++def views(qkv: torch.Tensor, heads: int) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: ++ sequence = qkv.shape[0] ++ inner = heads * 128 ++ return tuple(part.view(1, sequence, heads, 128) for part in qkv.split(inner, dim=-1)) ++ ++ ++def q_quant(q: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: ++ import sageattention.quant as sage_quant ++ ++ groups = math.ceil(q.shape[1] / 128) * 4 ++ output = torch.empty(q.shape, dtype=torch.int8, device=q.device) ++ scale = torch.empty((q.shape[0], q.shape[2], groups), dtype=torch.float32, device=q.device) ++ sage_quant._fused.quant_per_warp_int8_cuda(q, output, scale, 128, 32, 0) ++ return output, scale ++ ++ ++def k_quant(k: torch.Tensor, mean: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: ++ import sageattention.quant as sage_quant ++ ++ output = torch.empty(k.shape, dtype=torch.int8, device=k.device) ++ scale = torch.empty( ++ (k.shape[0], k.shape[2], math.ceil(k.shape[1] / 64)), ++ dtype=torch.float32, ++ device=k.device, ++ ) ++ sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( ++ k, mean.squeeze(1), output, scale, 64, 0, ++ ) ++ return output, scale ++ ++ ++def randomized_case(sequence: int, heads: int, seed: int) -> dict: ++ generator = torch.Generator(device="cuda").manual_seed(seed) ++ qkv = torch.randn( ++ (sequence, heads * 128 * 3), generator=generator, device="cuda", dtype=torch.bfloat16, ++ ) ++ rotation = torch.randn( ++ (1, sequence, 1, 48, 2, 2), generator=generator, device="cuda", dtype=torch.bfloat16, ++ ) ++ q_weight = torch.randn((128,), generator=generator, device="cuda", dtype=torch.bfloat16) ++ k_weight = torch.randn((128,), generator=generator, device="cuda", dtype=torch.bfloat16) ++ ++ reference_storage = qkv.clone() ++ q_reference, k_reference, _ = views(reference_storage, heads) ++ rms_rope_split_half_(q_reference, k_reference, rotation, q_weight, k_weight, 1e-5) ++ q_int8_reference, q_scale_reference = q_quant(q_reference) ++ ++ candidate_storage = qkv.clone() ++ q_candidate, k_candidate, _ = views(candidate_storage, heads) ++ q_int8_candidate, q_scale_candidate, q_prepared_candidate = prepare_qk( ++ q_candidate, ++ k_candidate, ++ rotation, ++ q_weight, ++ k_weight, ++ 1e-5, ++ materialize_q=True, ++ ) ++ return { ++ "sequence": sequence, ++ "heads": heads, ++ "q_prepared": difference(q_prepared_candidate, q_reference), ++ "k_prepared": difference(k_candidate, k_reference), ++ "q_int8": difference(q_int8_candidate, q_int8_reference), ++ "q_scale": difference(q_scale_candidate, q_scale_reference), ++ } ++ ++ ++def real_case(block: H3DiTBlock, hidden: torch.Tensor, rotation: torch.Tensor, modulation, segments, warmup: int, iterations: int) -> dict: ++ shift_msa, scale_msa, *_ = modulation ++ h_msa = modulate_segments( ++ rms_norm(hidden, block.norm1_weight, block.norm_eps), shift_msa, scale_msa, segments, ++ ) ++ raw_qkv = block.attention.qkv_proj(h_msa) ++ heads = block.attention.heads ++ ++ reference_storage = raw_qkv.clone() ++ q_reference, k_reference, v_reference = views(reference_storage, heads) ++ rms_rope_split_half_( ++ q_reference, ++ k_reference, ++ rotation, ++ block.attention.q_norm_weight, ++ block.attention.k_norm_weight, ++ block.attention.eps, ++ ) ++ q_int8_reference, q_scale_reference = q_quant(q_reference) ++ k_mean_reference = k_reference.mean(dim=1, keepdim=True) ++ k_int8_reference, k_scale_reference = k_quant(k_reference, k_mean_reference) ++ ++ candidate_storage = raw_qkv.clone() ++ q_candidate, k_candidate, v_candidate = views(candidate_storage, heads) ++ q_int8_candidate, q_scale_candidate, q_prepared_candidate = prepare_qk( ++ q_candidate, ++ k_candidate, ++ rotation, ++ block.attention.q_norm_weight, ++ block.attention.k_norm_weight, ++ block.attention.eps, ++ materialize_q=True, ++ ) ++ k_mean_candidate = k_candidate.mean(dim=1, keepdim=True) ++ k_int8_candidate, k_scale_candidate = k_quant(k_candidate, k_mean_candidate) ++ ++ import sageattention.core as sage_core ++ ++ v_fp8, v_scale, _ = sage_core.per_channel_fp8( ++ v_reference, tensor_layout="NHD", scale_max=2.25, smooth_v=False, ++ ) ++ reference_output = torch.empty_like(q_reference) ++ candidate_output = torch.empty_like(q_reference) ++ run_mainloop( ++ q_int8_reference, ++ k_int8_reference, ++ v_fp8, ++ q_scale_reference, ++ k_scale_reference, ++ v_scale, ++ reference_output, ++ ) ++ run_mainloop( ++ q_int8_candidate, ++ k_int8_candidate, ++ v_fp8, ++ q_scale_candidate, ++ k_scale_candidate, ++ v_scale, ++ candidate_output, ++ ) ++ torch.cuda.synchronize() ++ ++ baseline_timing_storage = raw_qkv.clone() ++ baseline_q, baseline_k, _ = views(baseline_timing_storage, heads) ++ ++ def baseline_entry(): ++ rms_rope_split_half_( ++ baseline_q, ++ baseline_k, ++ rotation, ++ block.attention.q_norm_weight, ++ block.attention.k_norm_weight, ++ block.attention.eps, ++ ) ++ return q_quant(baseline_q) ++ ++ candidate_timing_storage = raw_qkv.clone() ++ candidate_q, candidate_k, _ = views(candidate_timing_storage, heads) ++ ++ def candidate_entry(): ++ return prepare_qk( ++ candidate_q, ++ candidate_k, ++ rotation, ++ block.attention.q_norm_weight, ++ block.attention.k_norm_weight, ++ block.attention.eps, ++ materialize_q=False, ++ ) ++ ++ return { ++ "block": block.block_index if hasattr(block, "block_index") else None, ++ "q_shape": list(q_reference.shape), ++ "q_stride": list(q_reference.stride()), ++ "q_prepared": difference(q_prepared_candidate, q_reference), ++ "k_prepared": difference(k_candidate, k_reference), ++ "q_int8": difference(q_int8_candidate, q_int8_reference), ++ "q_scale": difference(q_scale_candidate, q_scale_reference), ++ "k_mean": difference(k_mean_candidate, k_mean_reference), ++ "k_int8": difference(k_int8_candidate, k_int8_reference), ++ "k_scale": difference(k_scale_candidate, k_scale_reference), ++ "attention_output": difference(candidate_output, reference_output), ++ "baseline_entry": measure(baseline_entry, warmup, iterations), ++ "candidate_entry": measure(candidate_entry, warmup, iterations), ++ } ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--blocks", nargs="+", type=int, default=(0, 24, 49)) ++ parser.add_argument("--randomized-only", action="store_true") ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ randomized = [ ++ randomized_case(sequence, 2, args.seed + sequence) ++ for sequence in (1, 31, 32, 33, 127, 128, 129) ++ ] ++ real = [] ++ metadata = {} ++ if not args.randomized_only: ++ torch.manual_seed(args.seed) ++ checkpoint = H3Checkpoint(args.model_path, device="cuda") ++ packer = H3PromptPacker(checkpoint) ++ video, audio, aligned_frames = random_av_latents( ++ args.width, args.height, args.frames, args.seed, device="cuda", ++ ) ++ sigma = beta_sigmas(args.steps, device="cuda")[args.sampler_step - 1] ++ native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) ++ text = torch.randn(1, args.text_tokens, 5376, device="cuda", dtype=torch.bfloat16) ++ hidden, timesteps, segments, positions, _, _ = packer( ++ text, video, native_audio, _model_sigma(sigma), ++ ) ++ rotation = h3_rope_rotation( ++ positions.to("cuda"), ++ checkpoint.tensor("rope.inv_freq", dtype=torch.float32), ++ hidden.dtype, ++ ) ++ for block_index in args.blocks: ++ block = H3DiTBlock.from_checkpoint(checkpoint, block_index, attention_backend="sage2").eval() ++ block.block_index = block_index ++ adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{block_index}.adaln_proj").eval() ++ with torch.inference_mode(): ++ real.append(real_case( ++ block, ++ hidden, ++ rotation, ++ tuple(value.detach() for value in adaln(timesteps)), ++ segments, ++ args.warmup, ++ args.iterations, ++ )) ++ metadata = { ++ "resolution": [args.width, args.height], ++ "frames": aligned_frames, ++ "tokens": hidden.shape[0], ++ "seed": args.seed, ++ "blocks": args.blocks, ++ } ++ report = { ++ "device": torch.cuda.get_device_name(), ++ "torch": torch.__version__, ++ "metadata": metadata, ++ "randomized": randomized, ++ "real": real, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ parity_items = randomized + real ++ if any( ++ not value["equal"] ++ for item in parity_items ++ for key, value in item.items() ++ if isinstance(value, dict) and "equal" in value ++ ): ++ raise RuntimeError("Sage2 entry fusion parity failed") ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/sage2_scheduler_register_variants/README.md b/research/sage2_scheduler_register_variants/README.md new file mode 100644 index 0000000..2b22ea2 --- /dev/null +++ b/research/sage2_scheduler_register_variants/README.md @@ -0,0 +1,11 @@ +# Sage2 Scheduler and Register Variant Recovery + +- `patches/0001-register-variant-harness.patch` preserves the standalone + extension comparison harness. +- `patches/0002-scheduler-decomposition.patch` preserves the standalone phase, + tail, and mainloop profiling harness. + +The exact patched SageAttention sources for register caps, narrowed scopes, +in-place scores, early-K prefetch, softmax-chain interleaving, and padded V shared +memory are absent. See `UNRECOVERABLE.md`. The separate temporal-pair source is +already preserved under `research/sage2_temporal_pair/`. diff --git a/research/sage2_scheduler_register_variants/UNRECOVERABLE.md b/research/sage2_scheduler_register_variants/UNRECOVERABLE.md new file mode 100644 index 0000000..591b05a --- /dev/null +++ b/research/sage2_scheduler_register_variants/UNRECOVERABLE.md @@ -0,0 +1,7 @@ +# Unrecoverable Variant Sources + +Exact source patches for the register-cap, narrowed-scope, in-place-score, +early-K-prefetch, independent-softmax-chain, and padded-V-shared-memory variants +are absent from the worktree, reachable history, reflogs, and dangling objects. +Only harnesses, reports, and prose descriptions remain, so no implementation is +reconstructed here. diff --git a/research/sage2_scheduler_register_variants/patches/0001-register-variant-harness.patch b/research/sage2_scheduler_register_variants/patches/0001-register-variant-harness.patch new file mode 100644 index 0000000..b7e270c --- /dev/null +++ b/research/sage2_scheduler_register_variants/patches/0001-register-variant-harness.patch @@ -0,0 +1,199 @@ +diff --git a/tools/profile_sage2_register_variants.py b/tools/profile_sage2_register_variants.py +new file mode 100644 +index 0000000..23419c6 +--- /dev/null ++++ b/tools/profile_sage2_register_variants.py +@@ -0,0 +1,193 @@ ++"""Benchmark isolated Sage2 SM89 extension variants on one real block-24 input.""" ++ ++from __future__ import annotations ++ ++import argparse ++import hashlib ++import importlib.util ++import json ++import math ++from pathlib import Path ++ ++import torch ++ ++from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize ++from profile_sage2_scheduler import prepare_quantized, run_mainloop ++ ++ ++EXPECTED_OUTPUT_SHA256 = "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574" ++ ++ ++def parse_variant(value: str) -> tuple[str, Path]: ++ try: ++ name, path = value.split("=", 1) ++ except ValueError as exc: ++ raise argparse.ArgumentTypeError("variant must be NAME=PATH") from exc ++ return name, Path(path) ++ ++ ++def tensor_sha256(value: torch.Tensor) -> str: ++ host_bytes = value.detach().contiguous().view(torch.uint8).cpu().numpy() ++ return hashlib.sha256(memoryview(host_bytes)).hexdigest() ++ ++ ++def load_variant(path: Path): ++ spec = importlib.util.spec_from_file_location("sageattention._qattn_sm89", path) ++ if spec is None or spec.loader is None: ++ raise RuntimeError(f"cannot load extension from {path}") ++ module = importlib.util.module_from_spec(spec) ++ spec.loader.exec_module(module) ++ return module ++ ++ ++def measure(fn, *, warmup: int, iterations: int) -> list[float]: ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ values = [] ++ for _ in range(iterations): ++ start = torch.cuda.Event(enable_timing=True) ++ end = torch.cuda.Event(enable_timing=True) ++ start.record() ++ fn() ++ end.record() ++ end.synchronize() ++ values.append(start.elapsed_time(end) / 1000.0) ++ return values ++ ++ ++def main() -> None: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--variant", type=parse_variant, action="append", required=True) ++ parser.add_argument("--rounds", type=int, default=3) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--attention", default="sage2", choices=("sage2",)) ++ parser.add_argument("--device", default="cuda") ++ args = parser.parse_args() ++ ++ block, hidden, rotation, _segments, metadata = representative_attention_inputs(args) ++ with torch.inference_mode(): ++ q, k, v, _ = prepare_qkv(block, hidden, rotation, None) ++ quantized = prepare_quantized(q, k, v) ++ _, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, reference_output = quantized ++ run_mainloop( ++ q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, reference_output, ++ ) ++ torch.cuda.synchronize() ++ reference_sha256 = tensor_sha256(reference_output) ++ if reference_sha256 != EXPECTED_OUTPUT_SHA256: ++ raise RuntimeError( ++ f"deployed reference mismatch: expected {EXPECTED_OUTPUT_SHA256}, got {reference_sha256}" ++ ) ++ ++ variants = [] ++ for name, path in args.variant: ++ extension = load_variant(path) ++ output = torch.empty_like(reference_output) ++ ++ def launch(extension=extension, output=output): ++ extension.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( ++ q_int8, ++ k_int8, ++ v_fp8, ++ output, ++ q_scale, ++ k_scale, ++ v_scale, ++ 0, ++ 0, ++ 2, ++ output.shape[-1] ** -0.5, ++ 0, ++ ) ++ return output ++ ++ variants.append({ ++ "name": name, ++ "path": str(path), ++ "output": output, ++ "launch": launch, ++ "samples": [], ++ "rounds": [], ++ }) ++ ++ for round_index in range(args.rounds): ++ round_samples = {variant["name"]: [] for variant in variants} ++ for variant in variants: ++ for _ in range(args.warmup): ++ variant["launch"]() ++ torch.cuda.synchronize() ++ for iteration in range(args.iterations): ++ offset = (round_index + iteration) % len(variants) ++ ordered = variants[offset:] + variants[:offset] ++ for variant in ordered: ++ samples = measure(variant["launch"], warmup=0, iterations=1) ++ variant["samples"].extend(samples) ++ round_samples[variant["name"]].extend(samples) ++ for variant in variants: ++ variant["rounds"].append(summarize(round_samples[variant["name"]])) ++ ++ results = [] ++ baseline_p50 = None ++ for variant in variants: ++ variant["launch"]() ++ torch.cuda.synchronize() ++ output_sha256 = tensor_sha256(variant["output"]) ++ timing = summarize(variant["samples"]) ++ if variant["name"] == "baseline": ++ baseline_p50 = timing["p50_s"] ++ results.append({ ++ "name": variant["name"], ++ "path": variant["path"], ++ "output_sha256": output_sha256, ++ "byte_exact": output_sha256 == reference_sha256, ++ "timing": timing, ++ "rounds": variant["rounds"], ++ }) ++ ++ if baseline_p50 is None: ++ raise RuntimeError("a variant named 'baseline' is required") ++ for result in results: ++ result["speedup_vs_baseline_percent"] = ( ++ baseline_p50 / result["timing"]["p50_s"] - 1.0 ++ ) * 100.0 ++ ++ report = { ++ "metadata": metadata, ++ "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", ++ "reference_output_sha256": reference_sha256, ++ "q_shape": list(q.shape), ++ "scheduler": { ++ "cta_q": 128, ++ "cta_k": 64, ++ "warp_q": 32, ++ "warp_k": 64, ++ "threads_per_cta": 128, ++ "dynamic_shared_memory_bytes": 32768, ++ "grid_ctas": math.ceil(q.shape[1] / 128) * q.shape[2], ++ }, ++ "rounds": args.rounds, ++ "warmup": args.warmup, ++ "iterations_per_round": args.iterations, ++ "variants": results, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if not all(result["byte_exact"] for result in results): ++ raise RuntimeError("one or more variants failed byte-exact output parity") ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/sage2_scheduler_register_variants/patches/0002-scheduler-decomposition.patch b/research/sage2_scheduler_register_variants/patches/0002-scheduler-decomposition.patch new file mode 100644 index 0000000..311556c --- /dev/null +++ b/research/sage2_scheduler_register_variants/patches/0002-scheduler-decomposition.patch @@ -0,0 +1,413 @@ +diff --git a/tools/profile_sage2_scheduler.py b/tools/profile_sage2_scheduler.py +new file mode 100644 +index 0000000..03130e1 +--- /dev/null ++++ b/tools/profile_sage2_scheduler.py +@@ -0,0 +1,407 @@ ++"""Profile exact SageAttention2 preparation, mainloop, and tail scheduling on real H3 tensors.""" ++ ++from __future__ import annotations ++ ++import argparse ++import hashlib ++import json ++import math ++from pathlib import Path ++ ++import torch ++ ++from h3_blackwell_runtime.sage2_entry import prepare_v ++from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize ++ ++ ++CTA_Q = 128 ++CTA_K = 64 ++WARP_Q = 32 ++WARP_K = 64 ++V_SCALE_MAX = 2.25 ++ ++ ++def tensor_sha256(value: torch.Tensor) -> str: ++ host_bytes = value.detach().contiguous().view(torch.uint8).cpu().numpy() ++ return hashlib.sha256(memoryview(host_bytes)).hexdigest() ++ ++ ++def event_measure(fn, *, warmup: int, iterations: int): ++ for _ in range(warmup): ++ fn() ++ torch.cuda.synchronize() ++ values = [] ++ result = None ++ for _ in range(iterations): ++ started = torch.cuda.Event(enable_timing=True) ++ finished = torch.cuda.Event(enable_timing=True) ++ started.record() ++ result = fn() ++ finished.record() ++ finished.synchronize() ++ values.append(started.elapsed_time(finished) / 1000.0) ++ return summarize(values), result ++ ++ ++def quantize_qk(q: torch.Tensor, k: torch.Tensor, km: torch.Tensor): ++ import sageattention.core as sage_core ++ ++ return sage_core.per_warp_int8_cuda( ++ q, ++ k, ++ km, ++ BLKQ=CTA_Q, ++ WARPQ=WARP_Q, ++ BLKK=CTA_K, ++ tensor_layout="NHD", ++ ) ++ ++ ++def quantize_v(v: torch.Tensor): ++ import sageattention.core as sage_core ++ ++ return sage_core.per_channel_fp8( ++ v, ++ tensor_layout="NHD", ++ scale_max=V_SCALE_MAX, ++ smooth_v=False, ++ ) ++ ++ ++def run_mainloop( ++ q_int8: torch.Tensor, ++ k_int8: torch.Tensor, ++ v_fp8: torch.Tensor, ++ q_scale: torch.Tensor, ++ k_scale: torch.Tensor, ++ v_scale: torch.Tensor, ++ output: torch.Tensor, ++) -> torch.Tensor: ++ import sageattention.core as sage_core ++ ++ sage_core.sm89_compile.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( ++ q_int8, ++ k_int8, ++ v_fp8, ++ output, ++ q_scale, ++ k_scale, ++ v_scale, ++ 0, ++ 0, ++ 2, ++ output.shape[-1] ** -0.5, ++ 0, ++ ) ++ return output ++ ++ ++def prepare_quantized(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor): ++ km = k.mean(dim=1, keepdim=True) ++ q_int8, q_scale, k_int8, k_scale = quantize_qk(q, k, km) ++ v_fp8, v_scale, _ = quantize_v(v) ++ output = torch.empty(q.shape, dtype=q.dtype, device=q.device) ++ return km, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output ++ ++ ++def tail_row( ++ q: torch.Tensor, ++ k: torch.Tensor, ++ v: torch.Tensor, ++ *, ++ warmup: int, ++ iterations: int, ++) -> dict: ++ quantized = prepare_quantized(q, k, v) ++ _, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output = quantized ++ timing, result = event_measure( ++ lambda: run_mainloop( ++ q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, ++ ), ++ warmup=warmup, ++ iterations=iterations, ++ ) ++ q_len = q.shape[1] ++ kv_len = k.shape[1] ++ q_ctas = math.ceil(q_len / CTA_Q) ++ k_iterations = math.ceil(kv_len / CTA_K) ++ return { ++ "q_len": q_len, ++ "kv_len": kv_len, ++ "q_ctas_per_head": q_ctas, ++ "k_iterations_per_cta": k_iterations, ++ "q_tail_rows": q_len % CTA_Q, ++ "k_tail_rows": kv_len % CTA_K, ++ "scheduled_q_rows": q_ctas * CTA_Q, ++ "q_row_efficiency": q_len / (q_ctas * CTA_Q), ++ "mainloop": timing, ++ "checksum": result.float().sum().item(), ++ } ++ ++ ++def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: ++ delta = actual.float() - expected.float() ++ return { ++ "equal": torch.equal(actual, expected), ++ "max_abs": delta.abs().max().item(), ++ "mean_abs": delta.abs().mean().item(), ++ } ++ ++ ++def parse_args() -> argparse.Namespace: ++ parser = argparse.ArgumentParser(description=__doc__) ++ parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") ++ parser.add_argument("--output", type=Path, required=True) ++ parser.add_argument("--width", type=int, default=1344) ++ parser.add_argument("--height", type=int, default=768) ++ parser.add_argument("--frames", type=int, default=124) ++ parser.add_argument("--steps", type=int, default=12) ++ parser.add_argument("--sampler-step", type=int, default=1) ++ parser.add_argument("--seed", type=int, default=440420) ++ parser.add_argument("--text-tokens", type=int, default=100) ++ parser.add_argument("--block-index", type=int, default=24) ++ parser.add_argument("--attention", default="sage2", choices=("sage2",)) ++ parser.add_argument("--warmup", type=int, default=3) ++ parser.add_argument("--iterations", type=int, default=10) ++ parser.add_argument("--tail-iterations", type=int, default=5) ++ parser.add_argument("--skip-tail-study", action="store_true") ++ parser.add_argument("--cuda-profiler-capture", action="store_true") ++ parser.add_argument("--expected-output-sha256") ++ parser.add_argument("--device", default="cuda") ++ return parser.parse_args() ++ ++ ++def main() -> None: ++ args = parse_args() ++ block, hidden, rotation, _segments, metadata = representative_attention_inputs(args) ++ with torch.inference_mode(): ++ q, k, v, _ = prepare_qkv(block, hidden, rotation, None) ++ km_timing, km = event_measure( ++ lambda: k.mean(dim=1, keepdim=True), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ qk_timing, qk = event_measure( ++ lambda: quantize_qk(q, k, km), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ q_int8, q_scale, k_int8, k_scale = qk ++ import sageattention.core as sage_core ++ import sageattention.quant as sage_quant ++ ++ q_int8_probe = torch.empty(q.shape, dtype=torch.int8, device=q.device) ++ q_scale_probe = torch.empty_like(q_scale) ++ q_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.quant_per_warp_int8_cuda( ++ q, q_int8_probe, q_scale_probe, CTA_Q, WARP_Q, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ k_int8_probe = torch.empty(k.shape, dtype=torch.int8, device=k.device) ++ k_scale_probe = torch.empty_like(k_scale) ++ k_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( ++ k, km.squeeze(1), k_int8_probe, k_scale_probe, CTA_K, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ padded_k = math.ceil(v.shape[1] / CTA_K) * CTA_K ++ v_transposed = torch.empty( ++ (v.shape[0], v.shape[3], v.shape[2], padded_k), ++ dtype=v.dtype, ++ device=v.device, ++ ) ++ v_transpose_timing, _ = event_measure( ++ lambda: sage_quant._fused.transpose_pad_permute_cuda(v, v_transposed, 0), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_fp8_probe = torch.empty_like(v_transposed, dtype=torch.float8_e4m3fn) ++ v_scale_probe = torch.empty( ++ (v.shape[0], v.shape[2], v.shape[3]), ++ dtype=torch.float32, ++ device=v.device, ++ ) ++ v_scale_quant_timing, _ = event_measure( ++ lambda: sage_quant._fused.scale_fuse_quant_cuda( ++ v_transposed, v_fp8_probe, v_scale_probe, v.shape[1], V_SCALE_MAX, 0, ++ ), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_timing, vq = event_measure( ++ lambda: quantize_v(v), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ v_fp8, v_scale, _ = vq ++ candidate_v_timing, candidate_vq = event_measure( ++ lambda: prepare_v(v, scale_max=V_SCALE_MAX), ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ candidate_v_fp8, candidate_v_scale = candidate_vq ++ output = torch.empty(q.shape, dtype=q.dtype, device=q.device) ++ mainloop_fn = lambda: run_mainloop( ++ q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, ++ ) ++ ++ if args.cuda_profiler_capture: ++ for _ in range(args.warmup): ++ mainloop_fn() ++ torch.cuda.synchronize() ++ torch.cuda.cudart().cudaProfilerStart() ++ captured = mainloop_fn() ++ torch.cuda.synchronize() ++ torch.cuda.cudart().cudaProfilerStop() ++ report = { ++ "metadata": metadata, ++ "capture": "one unchanged prequantized Sage2 mainloop", ++ "q_shape": list(q.shape), ++ "k_shape": list(k.shape), ++ "v_shape": list(v.shape), ++ "checksum": captured.float().sum().item(), ++ "scheduler": { ++ "cta_q": CTA_Q, ++ "cta_k": CTA_K, ++ "warp_q": WARP_Q, ++ "warp_k": WARP_K, ++ "warps_per_cta": 4, ++ "threads_per_cta": 128, ++ "dynamic_shared_memory_bytes": 32768, ++ "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), ++ "heads": q.shape[2], ++ "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], ++ "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), ++ "explicit_pipeline_stages": 2, ++ }, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ return ++ ++ mainloop_timing, manual_output = event_measure( ++ mainloop_fn, ++ warmup=args.warmup, ++ iterations=args.iterations, ++ ) ++ candidate_output = torch.empty_like(output) ++ run_mainloop( ++ q_int8, ++ k_int8, ++ candidate_v_fp8, ++ q_scale, ++ k_scale, ++ candidate_v_scale, ++ candidate_output, ++ ) ++ torch.cuda.synchronize() ++ reference = __import__("sageattention").sageattn( ++ q, k, v, tensor_layout="NHD", is_causal=False, smooth_k=False, ++ ) ++ torch.cuda.synchronize() ++ output_sha256 = tensor_sha256(manual_output) ++ expected_output_matches = ( ++ args.expected_output_sha256 is None ++ or output_sha256 == args.expected_output_sha256 ++ ) ++ ++ tail_study = [] ++ if not args.skip_tail_study: ++ q_lengths = sorted({ ++ (q.shape[1] // CTA_Q) * CTA_Q, ++ (q.shape[1] // CTA_Q) * CTA_Q + 1, ++ q.shape[1], ++ }) ++ kv_lengths = sorted({ ++ (k.shape[1] // CTA_K) * CTA_K, ++ (k.shape[1] // CTA_K) * CTA_K + 1, ++ k.shape[1], ++ }) ++ for q_len in q_lengths: ++ tail_study.append({ ++ "sweep": "q_tail_fixed_kv", ++ **tail_row( ++ q[:, :q_len], k, v, ++ warmup=args.warmup, ++ iterations=args.tail_iterations, ++ ), ++ }) ++ for kv_len in kv_lengths: ++ tail_study.append({ ++ "sweep": "kv_tail_fixed_q", ++ **tail_row( ++ q, k[:, :kv_len], v[:, :kv_len], ++ warmup=args.warmup, ++ iterations=args.tail_iterations, ++ ), ++ }) ++ ++ report = { ++ "metadata": metadata, ++ "q_shape": list(q.shape), ++ "k_shape": list(k.shape), ++ "v_shape": list(v.shape), ++ "warmup": args.warmup, ++ "iterations": args.iterations, ++ "phase_timings": { ++ "k_mean_and_smoothing_preparation": km_timing, ++ "qk_int8_quantization": qk_timing, ++ "q_int8_quantization": q_quant_timing, ++ "k_int8_subtract_mean_quantization": k_quant_timing, ++ "v_fp8_transpose_scale_quantization": v_timing, ++ "v_transpose_pad_permute": v_transpose_timing, ++ "v_scale_fp8_quantization": v_scale_quant_timing, ++ "vortex_direct_v_fp8_preparation": candidate_v_timing, ++ "fused_mainloop": mainloop_timing, ++ }, ++ "fused_mainloop_phases": { ++ "int8_qk": "fused inside qk_int_sv_f8_attn_kernel", ++ "scale_application": "fused inside qk_int_sv_f8_attn_kernel", ++ "online_softmax": "fused inside qk_int_sv_f8_attn_kernel", ++ "pv_accumulation": "fused inside qk_int_sv_f8_attn_kernel", ++ "final_normalization_and_output": "fused inside qk_int_sv_f8_attn_kernel", ++ "timing_policy": "Do not assign independent wall time without changing the exact kernel schedule; use source-correlated hardware counters.", ++ }, ++ "scheduler": { ++ "cta_q": CTA_Q, ++ "cta_k": CTA_K, ++ "warp_q": WARP_Q, ++ "warp_k": WARP_K, ++ "warps_per_cta": 4, ++ "threads_per_cta": 128, ++ "dynamic_shared_memory_bytes": 32768, ++ "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), ++ "heads": q.shape[2], ++ "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], ++ "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), ++ "q_tail_rows": q.shape[1] % CTA_Q, ++ "k_tail_rows": k.shape[1] % CTA_K, ++ "explicit_pipeline_stages": 2, ++ }, ++ "manual_decomposition_vs_public_sage2": difference(manual_output, reference), ++ "vortex_v_fp8_vs_sage2": difference(candidate_v_fp8, v_fp8), ++ "vortex_v_scale_vs_sage2": difference(candidate_v_scale, v_scale), ++ "vortex_v_mainloop_vs_sage2": difference(candidate_output, manual_output), ++ "manual_checksum": manual_output.float().sum().item(), ++ "reference_checksum": reference.float().sum().item(), ++ "output_sha256": output_sha256, ++ "expected_output_sha256": args.expected_output_sha256, ++ "expected_output_matches": expected_output_matches, ++ "tail_study": tail_study, ++ } ++ args.output.parent.mkdir(parents=True, exist_ok=True) ++ args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") ++ print(json.dumps(report, indent=2), flush=True) ++ if not expected_output_matches: ++ raise RuntimeError( ++ f"output SHA256 mismatch: expected {args.expected_output_sha256}, got {output_sha256}" ++ ) ++ ++ ++if __name__ == "__main__": ++ main() diff --git a/research/sage2_temporal_pair/README.md b/research/sage2_temporal_pair/README.md new file mode 100644 index 0000000..626afba --- /dev/null +++ b/research/sage2_temporal_pair/README.md @@ -0,0 +1,69 @@ +# Sage2 SM89 Temporal Pair Experiment + +This is an isolated experiment for SageAttention commit +`d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5`. It does not alter H3 runtime +dispatch or install an extension into the runtime image. + +## Candidate + +The patch adds a default-disabled kernel template option and enables it only in +`sm89_qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf.cu` in the candidate +checkout. The four existing warps retain their original query rows and private +`RS`, `RS_f8`, `RO`, `m`, and `d` state. + +For each KV tile, warp pair A and pair B alternate these phases: + +```text +A: QK + scale + online softmax + score conversion | B: previous-tile PV +B: QK + scale + online softmax + score conversion | A: current-tile PV +``` + +K and V remain single-buffered in their existing distinct shared-memory +regions. All warps cooperatively stage each next tile. Full-CTA barriers occur +before either buffer is overwritten and after each `cp.async.wait_group<0>()`. +There is no score or accumulator transfer. Pair B carries one `RS_f8` fragment +between loop iterations; the prologue fills and the epilogue drains that one +fragment. + +This differs from early-K prefetch because the intended overlap is INT QK from +one pair with FP8 PV from the other pair. It differs from independent softmax +chain interleaving because each warp's exact per-tile numerical sequence remains +`QK -> update_mdo -> accumulate_d -> RS_32_to_8 -> PV`. + +## Run On Spark + +From the project checkout on Spark: + +```bash +bash research/sage2_temporal_pair/run_spark_experiment.sh +``` + +The script creates detached baseline and candidate worktrees, checks and applies +the patch, builds separate `_qattn_sm89` extensions for `sm_121a`, and runs: + +1. A compute-sanitizer quick check. +2. Byte parity over one-, two-, three-, and four-tile boundary shapes. +3. The real block-24 SHA and rotating timing harness. +4. The absolute `<220 ms` candidate gate. + +Useful controls: + +```bash +RUN_SANITIZER=0 bash research/sage2_temporal_pair/run_spark_experiment.sh +RUN_NCU=1 bash research/sage2_temporal_pair/run_spark_experiment.sh +SAGE2_TEMPORAL_RUN_ID=manual-01 bash research/sage2_temporal_pair/run_spark_experiment.sh +MAX_JOBS=4 bash research/sage2_temporal_pair/run_spark_experiment.sh +``` + +The default working root is +`/home/daniel/aeon-spark-test/h3/sage2-temporal-pair`. JSON and optional NCU +reports are written under +`/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks`. +Build logs, including `ptxas -v` register and spill reports, remain under the +run-specific `builds` directory. + +Stop if the build fails, sanitizer reports an error, any short shape differs, +the real output SHA differs from `4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574`, +candidate p50 is at least 220 ms, shared memory exceeds the baseline 32768 +bytes, or candidate register/spill growth erases scheduler eligibility. Do not +integrate or replace the deployed extension from this experiment. diff --git a/research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch b/research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch new file mode 100644 index 0000000..a9c3c0a --- /dev/null +++ b/research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch @@ -0,0 +1,270 @@ +diff --git a/csrc/qattn/qk_int_sv_f8_cuda_sm89.cuh b/csrc/qattn/qk_int_sv_f8_cuda_sm89.cuh +index f39719f..2aa334a 100644 +--- a/csrc/qattn/qk_int_sv_f8_cuda_sm89.cuh ++++ b/csrc/qattn/qk_int_sv_f8_cuda_sm89.cuh +@@ -41,8 +41,111 @@ + #define MMA_SV_N 16 + #define MMA_SV_K 32 + ++template ++__device__ __forceinline__ void temporal_pair_produce_score( ++ const smem_t &smem_Q, ++ const smem_t &smem_K, ++ int32_t RS[][num_tiles_k][8], uint32_t RQ[][4], ++ const uint32_t Q_smem_offset_mma, const uint32_t K_smem_offset_mma, ++ DTypeSVAccum RO[][num_tiles_v][8], float m[][2], float d[][2], ++ uint32_t RS_f8[][num_tiles_k / 2][4], ++ const uint32_t tile, const uint32_t num_iterations, ++ const uint32_t Q_idx_lane_base, uint32_t &K_idx_lane_base, const uint32_t kv_len, ++ const float original_sm_scale, const float dequant_scale) ++{ ++ if constexpr (num_tiles_qk_inner == 1) ++ { ++ compute_int_qk(smem_K, RS, RQ, K_smem_offset_mma); ++ } ++ else ++ { ++ uint32_t q_offset = Q_smem_offset_mma; ++ uint32_t k_offset = K_smem_offset_mma; ++ compute_int_qk(smem_Q, smem_K, RS, q_offset, k_offset); ++ } ++ ++ float RS_f32[num_tiles_q][num_tiles_k][8]; ++ const bool final_pair = tile + 2 >= num_iterations; ++#pragma unroll ++ for (uint32_t fq = 0; fq < num_tiles_q; fq++) ++ { ++#pragma unroll ++ for (uint32_t fk = 0; fk < num_tiles_k; fk++) ++ { ++#pragma unroll ++ for (uint32_t k = 0; k < 8; k++) ++ { ++ float score = __int2float_rz(RS[fq][fk][k]); ++ RS_f32[fq][fk][k] = final_pair ? score * dequant_scale : score; ++ } ++ } ++ } ++ ++ if (final_pair) ++ { ++ if constexpr (mask_mode == MaskMode::kCausal) ++ { ++ apply_causal_mask(Q_idx_lane_base, K_idx_lane_base, RS_f32); ++ } ++ if (tile + 1 == num_iterations) ++ { ++ apply_out_of_bound_mask(K_idx_lane_base, RS_f32, kv_len); ++ } ++ } ++ K_idx_lane_base += CTA_K; ++ ++ const float tile_sm_scale = final_pair ? original_sm_scale : original_sm_scale * dequant_scale; ++ if constexpr (std::is_same::value) ++ { ++ update_mdo(RS_f32, RO, m, d, tile_sm_scale); ++ } ++ else ++ { ++ update_mdo(RS_f32, RO, m, d, tile_sm_scale); ++ } ++ if constexpr (DenominatorAccumUnit == ComputeUnit::kCudaCore) ++ { ++ accumulate_d(RS_f32, d); ++ } ++ RS_32_to_8(RS_f32, RS_f8); ++ if constexpr (DenominatorAccumUnit == ComputeUnit::kTensorCore) ++ { ++ accumulate_d_f8(RS_f8, d); ++ } ++} ++ ++template ++__device__ __forceinline__ void temporal_pair_consume_value( ++ const smem_t &smem_V, ++ uint32_t RS_f8[][num_tiles_k / 2][4], DTypeSVAccum RO[][num_tiles_v][8], float d[][2]) ++{ ++ if constexpr (!use_inst_buffer) ++ { ++ compute_fp8_sv(smem_V, RS_f8, RO, d); ++ } ++ else if constexpr (!use_pv_fp16_accu) ++ { ++ compute_fp8_sv_inst_buf(smem_V, RS_f8, RO, d); ++ } ++ else ++ { ++ compute_fp8_sv_inst_buf_fp16_accu(smem_V, RS_f8, RO, d); ++ } ++} ++ + template ++ typename DTypeSVAccum = float, bool use_inst_buffer = false, typename DTypeOut = half, ComputeUnit DenominatorAccumUnit, MaskMode mask_mode = MaskMode::kNone, bool return_lse = false, bool fuse_v_scale=false, bool fuse_v_mean=false, bool use_pv_fp16_accu=false, bool temporal_pair_pipeline=false> + __global__ void qk_int_sv_f8_attn_kernel(int8_t *__restrict__ Q, int8_t *__restrict__ K, int8_t *__restrict__ V, DTypeOut *__restrict__ O, float *__restrict__ Lse, + float *__restrict__ Q_scale, float *__restrict__ K_scale, float *__restrict__ V_scale, float *__restrict__ V_mean, + const uint32_t qo_len, const uint32_t kv_len, const uint32_t num_kv_groups, +@@ -266,6 +369,124 @@ __global__ void qk_int_sv_f8_attn_kernel(int8_t *__restrict__ Q, int8_t *__restr + + K_load_idx_lane_base += CTA_K; + ++ if constexpr (temporal_pair_pipeline) ++ { ++ // K0 and V0 were issued by every warp above. The candidate keeps one score ++ // fragment per warp and temporally staggers the two warp pairs; no score, ++ // softmax state, or output accumulator crosses a warp boundary. ++ cp_async::wait_group<0>(); ++ __syncthreads(); ++ ++ uint32_t RS_f8[num_tiles_q][num_tiles_k / 2][4]; ++ const bool pair_a = warp_id < (num_warps / 2); ++ ++ if (pair_a) ++ { ++ temporal_pair_produce_score( ++ smem_Q, smem_K, RS, RQ, Q_smem_offset_mma, K_smem_offset_mma, ++ RO, m, d, RS_f8, 0, num_iterations, Q_idx_lane_base, K_idx_lane_base, ++ kv_len, original_sm_scale, q_scale * K_scale[k_scale_idx]); ++ } ++ __syncthreads(); ++ ++ if (pair_a) ++ { ++ temporal_pair_consume_value(smem_V, RS_f8, RO, d); ++ } ++ else ++ { ++ temporal_pair_produce_score( ++ smem_Q, smem_K, RS, RQ, Q_smem_offset_mma, K_smem_offset_mma, ++ RO, m, d, RS_f8, 0, num_iterations, Q_idx_lane_base, K_idx_lane_base, ++ kv_len, original_sm_scale, q_scale * K_scale[k_scale_idx]); ++ } ++ __syncthreads(); ++ ++ for (uint32_t tile = 1; tile < num_iterations; tile++) ++ { ++ if (tile + 1 == num_iterations) ++ { ++ load_global_to_share( ++ &K_lane_base_ptr, K_smem_offset_load, stride_seq_k, smem_K, ++ K_load_idx_lane_base, kv_len); ++ } ++ else ++ { ++ load_global_to_share( ++ &K_lane_base_ptr, K_smem_offset_load, stride_seq_k, smem_K); ++ } ++ cp_async::commit_group(); ++ cp_async::wait_group<0>(); ++ __syncthreads(); ++ ++ if (pair_a) ++ { ++ temporal_pair_produce_score( ++ smem_Q, smem_K, RS, RQ, Q_smem_offset_mma, K_smem_offset_mma, ++ RO, m, d, RS_f8, tile, num_iterations, Q_idx_lane_base, K_idx_lane_base, ++ kv_len, original_sm_scale, ++ q_scale * K_scale[k_scale_idx + tile * k_scale_advance_offset]); ++ } ++ else ++ { ++ temporal_pair_consume_value(smem_V, RS_f8, RO, d); ++ } ++ __syncthreads(); ++ ++ load_fp8_V_global_to_share( ++ &V_lane_base_ptr, V_smem_offset_load, stride_d_v, smem_V); ++ cp_async::commit_group(); ++ cp_async::wait_group<0>(); ++ __syncthreads(); ++ ++ if (pair_a) ++ { ++ temporal_pair_consume_value(smem_V, RS_f8, RO, d); ++ } ++ else ++ { ++ temporal_pair_produce_score( ++ smem_Q, smem_K, RS, RQ, Q_smem_offset_mma, K_smem_offset_mma, ++ RO, m, d, RS_f8, tile, num_iterations, Q_idx_lane_base, K_idx_lane_base, ++ kv_len, original_sm_scale, ++ q_scale * K_scale[k_scale_idx + tile * k_scale_advance_offset]); ++ } ++ __syncthreads(); ++ K_load_idx_lane_base += CTA_K; ++ } ++ ++ // Pair B deliberately carries exactly one score fragment out of the last ++ // phase. Drain it before the unchanged normalization and output epilogue. ++ if (!pair_a) ++ { ++ temporal_pair_consume_value(smem_V, RS_f8, RO, d); ++ } ++ __syncthreads(); ++ } ++ else ++ { + #pragma unroll + for (uint32_t iter = 1; iter < num_iterations - 1; iter++) + { +@@ -566,6 +787,7 @@ __global__ void qk_int_sv_f8_attn_kernel(int8_t *__restrict__ Q, int8_t *__restr + __syncthreads(); + + } ++ } + + // TODO: thread block sync mdo state for num_warps_k > 0. Then only one thread block needs to do the final saving. + +diff --git a/csrc/qattn/sm89_qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf.cu b/csrc/qattn/sm89_qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf.cu +index 727030f..3a72af1 100644 +--- a/csrc/qattn/sm89_qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf.cu ++++ b/csrc/qattn/sm89_qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf.cu +@@ -152,7 +152,7 @@ torch::Tensor qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf(torch::Tensor q + size_t smem_max = std::max(CTA_Q * HEAD_DIM * sizeof(int8_t) + CTA_K * HEAD_DIM * sizeof(int8_t) + CTA_K * HEAD_DIM * sizeof(int8_t), CTA_Q * HEAD_DIM * sizeof(half)); + + auto kernel_func = qk_int_sv_f8_attn_kernel(QK_QUANT_GRAN), static_cast(QK_QUANT_GRAN), +- float, true, DTypeOut, ComputeUnit::kCudaCore, mask_mode, RETURN_LSE, true, false, true>; ++ float, true, DTypeOut, ComputeUnit::kCudaCore, mask_mode, RETURN_LSE, true, false, true, true>; + + cudaFuncSetAttribute(kernel_func, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_max); + +@@ -184,4 +184,4 @@ torch::Tensor qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf(torch::Tensor q + }); + + return lse; +-} +\ No newline at end of file ++} diff --git a/research/sage2_temporal_pair/run_spark_experiment.sh b/research/sage2_temporal_pair/run_spark_experiment.sh new file mode 100644 index 0000000..7a3fec9 --- /dev/null +++ b/research/sage2_temporal_pair/run_spark_experiment.sh @@ -0,0 +1,148 @@ +#!/usr/bin/env bash +set -euo pipefail + +commit=d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5 +repo_url=https://github.com/thu-ml/SageAttention.git +root=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd) +patch="$root/research/sage2_temporal_pair/patches/0001-sm89-temporal-two-pair-qk-pv.patch" +validator="$root/research/sage2_temporal_pair/validate_short.py" +compose="$root/compose.spark.yml" + +work_root=${SAGE2_TEMPORAL_WORK_ROOT:-/home/daniel/aeon-spark-test/h3/sage2-temporal-pair} +run_id=${SAGE2_TEMPORAL_RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)} +run_root="$work_root/$run_id" +repo="$run_root/repo" +baseline_source="$run_root/baseline-source" +candidate_source="$run_root/candidate-source" +builds="$run_root/builds" +output_host=/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks +output_container=/output/h3-blackwell-runtime/benchmarks + +if [[ -e "$run_root" ]]; then + echo "run directory already exists: $run_root" >&2 + exit 2 +fi +mkdir -p "$run_root" "$builds/baseline" "$builds/temporal-pair" "$output_host" + +git clone --filter=blob:none --no-checkout "$repo_url" "$repo" +git -C "$repo" worktree add --detach "$baseline_source" "$commit" +git -C "$repo" worktree add --detach "$candidate_source" "$commit" +git -C "$candidate_source" apply --check "$patch" +git -C "$candidate_source" apply "$patch" + +build_variant() { + local name=$1 + local source=$2 + docker compose -f "$compose" run --rm --no-deps \ + -e TORCH_CUDA_ARCH_LIST=12.1 \ + -e MAX_JOBS="${MAX_JOBS:-8}" \ + -v "$source:/work/sageattention:rw" \ + -v "$builds:/variants:rw" \ + --workdir /work/sageattention \ + h3-blackwell-runtime \ + bash -lc "set -o pipefail; python setup.py build_ext --force --build-temp /variants/$name/temp --build-lib /variants/$name/lib 2>&1 | tee /variants/$name/build.log" +} + +build_variant baseline "$baseline_source" +build_variant temporal-pair "$candidate_source" + +baseline_matches=("$builds"/baseline/lib/sageattention/_qattn_sm89*.so) +candidate_matches=("$builds"/temporal-pair/lib/sageattention/_qattn_sm89*.so) +if [[ ${#baseline_matches[@]} -ne 1 || ! -f "${baseline_matches[0]}" ]]; then + echo "expected one baseline _qattn_sm89 extension" >&2 + exit 3 +fi +if [[ ${#candidate_matches[@]} -ne 1 || ! -f "${candidate_matches[0]}" ]]; then + echo "expected one candidate _qattn_sm89 extension" >&2 + exit 3 +fi +extension=$(basename "${baseline_matches[0]}") +if [[ $(basename "${candidate_matches[0]}") != "$extension" ]]; then + echo "baseline and candidate extension names differ" >&2 + exit 3 +fi + +short_json="$output_container/gb10-sage2-temporal-pair-short-$run_id.json" +timing_json="$output_container/gb10-sage2-temporal-pair-timing-$run_id.json" + +if [[ ${RUN_SANITIZER:-1} == 1 ]]; then + docker compose -f "$compose" run --rm --no-deps \ + -v "$builds:/variants:ro" \ + -v "$validator:/research/validate_short.py:ro" \ + h3-blackwell-runtime \ + compute-sanitizer --tool memcheck --error-exitcode=99 \ + python /research/validate_short.py \ + --quick \ + --baseline "/variants/baseline/lib/sageattention/$extension" \ + --candidate "/variants/temporal-pair/lib/sageattention/$extension" +fi + +docker compose -f "$compose" run --rm --no-deps \ + -v "$builds:/variants:ro" \ + -v "$validator:/research/validate_short.py:ro" \ + h3-blackwell-runtime \ + python /research/validate_short.py \ + --baseline "/variants/baseline/lib/sageattention/$extension" \ + --candidate "/variants/temporal-pair/lib/sageattention/$extension" \ + --output "$short_json" + +# The real harness first verifies the deployed baseline SHA, byte-compares both +# variants, then times them in rotating order. No extension is installed. +docker compose -f "$compose" run --rm --no-deps \ + -v "$root:/opt/h3-blackwell-runtime:ro" \ + -v "$builds:/variants:ro" \ + --workdir /opt/h3-blackwell-runtime \ + h3-blackwell-runtime \ + python tools/profile_sage2_register_variants.py \ + --output "$timing_json" \ + --rounds 5 \ + --warmup 3 \ + --iterations 10 \ + --variant "baseline=/variants/baseline/lib/sageattention/$extension" \ + --variant "temporal-pair=/variants/temporal-pair/lib/sageattention/$extension" + +timing_host="$output_host/gb10-sage2-temporal-pair-timing-$run_id.json" +python3 - "$timing_host" <<'PY' +import json +import pathlib +import sys + +report = json.loads(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8")) +variants = {item["name"]: item for item in report["variants"]} +candidate = variants["temporal-pair"] +latency_ms = candidate["timing"]["p50_s"] * 1000.0 +print(f"temporal-pair p50: {latency_ms:.3f} ms") +if not candidate["byte_exact"]: + raise SystemExit("candidate failed real SHA parity") +if latency_ms >= 220.0: + raise SystemExit("candidate failed the <220 ms mainloop gate") +PY + +if [[ ${RUN_NCU:-0} == 1 ]]; then + docker compose -f "$compose" run --rm --no-deps \ + --cap-add SYS_ADMIN --cap-add SYS_PTRACE \ + -v "$root:/opt/h3-blackwell-runtime:ro" \ + -v "${candidate_matches[0]}:/opt/venv/lib/python3.12/site-packages/sageattention/$extension:ro" \ + --workdir /opt/h3-blackwell-runtime \ + h3-blackwell-runtime \ + /opt/nvidia/nsight-compute/2025.3.1/ncu \ + --target-processes all \ + --profile-from-start off \ + --section LaunchStats \ + --section Occupancy \ + --section SchedulerStats \ + --section WarpStateStats \ + --section SpeedOfLight \ + --kernel-name regex:qk_int_sv_f8_attn_kernel \ + --launch-count 1 \ + --force-overwrite \ + -o "$output_container/gb10-sage2-temporal-pair-counters-$run_id" \ + python tools/profile_sage2_scheduler.py \ + --output "$output_container/gb10-sage2-temporal-pair-capture-$run_id.json" \ + --cuda-profiler-capture \ + --skip-tail-study +fi + +echo "experiment artifacts: $run_root" +echo "short parity: $output_host/gb10-sage2-temporal-pair-short-$run_id.json" +echo "timing: $timing_host" diff --git a/research/sage2_temporal_pair/validate_short.py b/research/sage2_temporal_pair/validate_short.py new file mode 100644 index 0000000..27c7c56 --- /dev/null +++ b/research/sage2_temporal_pair/validate_short.py @@ -0,0 +1,154 @@ +"""Byte-compare isolated SM89 extensions on adversarial short NHD shapes.""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +import json +from pathlib import Path + +import torch + + +CASES = ( + (1, 1), + (31, 63), + (32, 64), + (33, 65), + (63, 127), + (64, 128), + (65, 129), + (127, 63), + (128, 64), + (129, 65), + (191, 191), + (192, 192), + (193, 193), +) + + +def load_extension(path: Path): + spec = importlib.util.spec_from_file_location("sageattention._qattn_sm89", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load extension from {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def digest(value: torch.Tensor) -> str: + data = value.detach().contiguous().view(torch.uint8).cpu().numpy() + return hashlib.sha256(memoryview(data)).hexdigest() + + +def launch(extension, q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale): + output = torch.empty(q_int8.shape, dtype=torch.bfloat16, device=q_int8.device) + extension.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( + q_int8, + k_int8, + v_fp8, + output, + q_scale, + k_scale, + v_scale, + 0, + 0, + 2, + 128**-0.5, + 0, + ) + return output + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--baseline", type=Path, required=True) + parser.add_argument("--candidate", type=Path, required=True) + parser.add_argument("--output", type=Path) + parser.add_argument("--heads", type=int, default=2) + parser.add_argument("--seed", type=int, default=73021) + parser.add_argument("--quick", action="store_true", help="Run only one one-tile and one multi-tile case.") + args = parser.parse_args() + + import sageattention.core as sage_core + + baseline = load_extension(args.baseline) + candidate = load_extension(args.candidate) + cases = ((1, 1), (129, 193)) if args.quick else CASES + results = [] + + with torch.inference_mode(): + for index, (q_len, kv_len) in enumerate(cases): + generator = torch.Generator(device="cuda").manual_seed(args.seed + index) + q = torch.randn( + (1, q_len, args.heads, 128), + dtype=torch.bfloat16, + device="cuda", + generator=generator, + ) + k = torch.randn( + (1, kv_len, args.heads, 128), + dtype=torch.bfloat16, + device="cuda", + generator=generator, + ) + v = torch.randn( + (1, kv_len, args.heads, 128), + dtype=torch.bfloat16, + device="cuda", + generator=generator, + ) + k_mean = k.mean(dim=1, keepdim=True) + q_int8, q_scale, k_int8, k_scale = sage_core.per_warp_int8_cuda( + q, + k, + k_mean, + BLKQ=128, + WARPQ=32, + BLKK=64, + tensor_layout="NHD", + ) + v_fp8, v_scale, _ = sage_core.per_channel_fp8( + v, + tensor_layout="NHD", + scale_max=2.25, + smooth_v=False, + ) + expected = launch( + baseline, q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, + ) + actual = launch( + candidate, q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, + ) + torch.cuda.synchronize() + delta = actual.float() - expected.float() + result = { + "q_len": q_len, + "kv_len": kv_len, + "equal": torch.equal(actual, expected), + "different_elements": int(torch.count_nonzero(actual != expected).item()), + "max_abs": delta.abs().max().item(), + "baseline_sha256": digest(expected), + "candidate_sha256": digest(actual), + } + results.append(result) + print(json.dumps(result), flush=True) + if not result["equal"]: + raise RuntimeError(f"short-shape parity failed: {result}") + + report = { + "device": torch.cuda.get_device_name(), + "baseline": str(args.baseline), + "candidate": str(args.candidate), + "heads": args.heads, + "seed": args.seed, + "cases": results, + } + if args.output is not None: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/research/shared_cuda_experimental_worktree/README.md b/research/shared_cuda_experimental_worktree/README.md new file mode 100644 index 0000000..2e69e55 --- /dev/null +++ b/research/shared_cuda_experimental_worktree/README.md @@ -0,0 +1,21 @@ +# Shared CUDA Experimental Worktree Recovery + +This capsule preserves the current worktree-versus-HEAD CUDA extension changes. +The patch is intentionally not described as isolated: the same translation units +contain several requested experiments and shared helpers. + +Source baseline: `bee998d` (`HEAD` at recovery time). + +`patches/0001-shared-cuda-experiments.patch` contains: + +- Sage2 P1 fused Q/K RMSNorm, split-half RoPE, and Q INT8 preparation. +- Sage2 P2 direct V scale, E4M3 quantization, padding, and permutation. +- NVFP4 modulation-to-pack and SwiGLU-to-FC2-pack fused producers. +- Generalized physical block-scale swizzle and approximate reciprocal encoding. +- Caller-owned NVFP4 output buffers used by bounded-ring attempts. +- QKV split/layout kernels, including the vectorized 16-byte store path. + +The patch also contains the shared Python wrappers required to express those +entry points. It must not be applied wholesale to production without splitting +and revalidation. Nothing in this directory is connected to runtime dispatch or +build defaults. diff --git a/research/shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch b/research/shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch new file mode 100644 index 0000000..7584fb5 --- /dev/null +++ b/research/shared_cuda_experimental_worktree/patches/0001-shared-cuda-experiments.patch @@ -0,0 +1,1111 @@ +diff --git a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp +index 9291ed5..ecabdc8 100644 +--- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp ++++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp +@@ -5,7 +5,12 @@ + torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor); + torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divisor, torch::Tensor partials, torch::Tensor output, int64_t blocks, int64_t threads); + std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads); ++void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first); ++std::vector quantize_nvfp4_modulated_bf16_cuda(torch::Tensor input, torch::Tensor shift, torch::Tensor scale, torch::Tensor row_index, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads); ++std::vector quantize_nvfp4_swiglu_bf16_cuda(torch::Tensor input, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads); + std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim); ++std::vector sage2_prepare_qk_cuda(torch::Tensor q, torch::Tensor k, torch::Tensor rotation, torch::Tensor q_weight, torch::Tensor k_weight, double epsilon, bool materialize_q); ++std::vector sage2_prepare_v_cuda(torch::Tensor v, double scale_max); + + torch::Tensor nvfp4_activation_scale(torch::Tensor input, double divisor) { + TORCH_CHECK(input.is_cuda(), "nvfp4_activation_scale expects a CUDA tensor"); +@@ -42,6 +47,50 @@ std::vector quantize_nvfp4_bf16(torch::Tensor input, torch::Tenso + return quantize_nvfp4_bf16_cuda(input, scale, pad_16x, threads); + } + ++void quantize_nvfp4_bf16_into(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first) { ++ TORCH_CHECK(input.is_cuda() && scale.is_cuda() && qdata.is_cuda() && block_scale.is_cuda(), "quantize_nvfp4_bf16_into expects CUDA tensors"); ++ TORCH_CHECK(input.device() == scale.device() && input.device() == qdata.device() && input.device() == block_scale.device(), "quantize_nvfp4_bf16_into tensors must share a device"); ++ TORCH_CHECK(input.is_contiguous() && qdata.is_contiguous() && block_scale.is_contiguous(), "quantize_nvfp4_bf16_into tensors must be contiguous"); ++ TORCH_CHECK(input.dim() == 2 && qdata.dim() == 2 && block_scale.dim() == 2, "quantize_nvfp4_bf16_into expects 2D input and outputs"); ++ TORCH_CHECK(input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_bf16_into expects bfloat16 input"); ++ TORCH_CHECK(scale.scalar_type() == torch::kFloat32 && scale.numel() == 1, "quantize_nvfp4_bf16_into expects a scalar float32 scale"); ++ TORCH_CHECK(qdata.scalar_type() == torch::kUInt8, "quantize_nvfp4_bf16_into expects uint8 qdata"); ++ TORCH_CHECK(block_scale.scalar_type() == torch::kFloat8_e4m3fn, "quantize_nvfp4_bf16_into expects float8_e4m3fn block scales"); ++ TORCH_CHECK(qdata.size(0) >= input.size(0) && qdata.size(1) >= input.size(1) / 2, "quantize_nvfp4_bf16_into qdata output is too small"); ++ const int64_t required_scale_rows = ((qdata.size(0) + 127) / 128) * 128; ++ const int64_t required_scale_cols = (((qdata.size(1) / 8) + 3) / 4) * 4; ++ TORCH_CHECK(block_scale.size(0) >= required_scale_rows && block_scale.size(1) >= required_scale_cols, "quantize_nvfp4_bf16_into block-scale output is too small"); ++ TORCH_CHECK(threads == 64 || threads == 128 || threads == 256 || threads == 512, "quantize_nvfp4_bf16_into threads must be 64, 128, 256, or 512"); ++ quantize_nvfp4_bf16_into_cuda(input, scale, qdata, block_scale, threads, hi_first); ++} ++ ++std::vector quantize_nvfp4_modulated_bf16(torch::Tensor input, torch::Tensor shift, torch::Tensor scale, torch::Tensor row_index, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads) { ++ TORCH_CHECK(input.is_cuda() && shift.is_cuda() && scale.is_cuda() && row_index.is_cuda(), "quantize_nvfp4_modulated_bf16 expects CUDA tensors"); ++ TORCH_CHECK(input.device() == shift.device() && input.device() == scale.device() && input.device() == row_index.device(), "quantize_nvfp4_modulated_bf16 tensors must share a device"); ++ TORCH_CHECK(input.is_contiguous() && shift.is_contiguous() && scale.is_contiguous() && row_index.is_contiguous(), "quantize_nvfp4_modulated_bf16 tensors must be contiguous"); ++ TORCH_CHECK(input.dim() == 2 && shift.dim() == 2 && scale.dim() == 2, "quantize_nvfp4_modulated_bf16 expects 2D input and tables"); ++ TORCH_CHECK(input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_modulated_bf16 expects bfloat16 input"); ++ TORCH_CHECK(shift.scalar_type() == scale.scalar_type() && (shift.scalar_type() == torch::kBFloat16 || shift.scalar_type() == torch::kFloat32), "quantize_nvfp4_modulated_bf16 tables must share bfloat16 or float32 dtype"); ++ TORCH_CHECK(shift.sizes() == scale.sizes() && shift.size(1) == input.size(1), "quantize_nvfp4_modulated_bf16 table shapes must match the input width"); ++ TORCH_CHECK(row_index.scalar_type() == torch::kInt32 && row_index.dim() == 1 && row_index.numel() == input.size(0), "quantize_nvfp4_modulated_bf16 row_index must be int32 with one entry per row"); ++ TORCH_CHECK(divisor > 0.0, "quantize_nvfp4_modulated_bf16 divisor must be positive"); ++ TORCH_CHECK(blocks > 0, "quantize_nvfp4_modulated_bf16 blocks must be positive"); ++ TORCH_CHECK(scale_threads == 128 || scale_threads == 256 || scale_threads == 512, "quantize_nvfp4_modulated_bf16 scale_threads must be 128, 256, or 512"); ++ TORCH_CHECK(pack_threads == 64 || pack_threads == 128 || pack_threads == 256 || pack_threads == 512, "quantize_nvfp4_modulated_bf16 pack_threads must be 64, 128, 256, or 512"); ++ return quantize_nvfp4_modulated_bf16_cuda(input, shift, scale, row_index, divisor, pad_16x, blocks, scale_threads, pack_threads); ++} ++ ++std::vector quantize_nvfp4_swiglu_bf16(torch::Tensor input, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads) { ++ TORCH_CHECK(input.is_cuda() && input.is_contiguous(), "quantize_nvfp4_swiglu_bf16 expects contiguous CUDA input"); ++ TORCH_CHECK(input.dim() == 2 && input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_swiglu_bf16 expects 2D bfloat16 input"); ++ TORCH_CHECK(input.size(1) % 2 == 0, "quantize_nvfp4_swiglu_bf16 input width must be even"); ++ TORCH_CHECK(divisor > 0.0 && blocks > 0, "quantize_nvfp4_swiglu_bf16 divisor and blocks must be positive"); ++ TORCH_CHECK(scale_threads == 128 || scale_threads == 256 || scale_threads == 512, "quantize_nvfp4_swiglu_bf16 scale_threads must be 128, 256, or 512"); ++ TORCH_CHECK(pack_threads == 64 || pack_threads == 128 || pack_threads == 256 || pack_threads == 512, "quantize_nvfp4_swiglu_bf16 pack_threads must be 64, 128, 256, or 512"); ++ return quantize_nvfp4_swiglu_bf16_cuda(input, divisor, pad_16x, blocks, scale_threads, pack_threads); ++} ++ ++ + std::vector qkv_to_bshd(torch::Tensor qkv, int64_t heads, int64_t head_dim) { + TORCH_CHECK(qkv.is_cuda(), "qkv_to_bshd expects a CUDA tensor"); + TORCH_CHECK(qkv.is_contiguous(), "qkv_to_bshd expects contiguous input"); +@@ -51,9 +100,38 @@ std::vector qkv_to_bshd(torch::Tensor qkv, int64_t heads, int64_t + return qkv_to_bshd_cuda(qkv, heads, head_dim); + } + ++std::vector sage2_prepare_qk(torch::Tensor q, torch::Tensor k, torch::Tensor rotation, torch::Tensor q_weight, torch::Tensor k_weight, double epsilon, bool materialize_q) { ++ TORCH_CHECK(q.is_cuda() && k.is_cuda() && rotation.is_cuda() && q_weight.is_cuda() && k_weight.is_cuda(), "sage2_prepare_qk expects CUDA tensors"); ++ TORCH_CHECK(q.device() == k.device() && q.device() == rotation.device() && q.device() == q_weight.device() && q.device() == k_weight.device(), "sage2_prepare_qk tensors must share a device"); ++ TORCH_CHECK(q.scalar_type() == torch::kBFloat16 && k.scalar_type() == torch::kBFloat16, "sage2_prepare_qk expects BF16 Q and K"); ++ TORCH_CHECK(rotation.scalar_type() == torch::kBFloat16 && q_weight.scalar_type() == torch::kBFloat16 && k_weight.scalar_type() == torch::kBFloat16, "sage2_prepare_qk expects BF16 rotation and weights"); ++ TORCH_CHECK(q.dim() == 4 && k.dim() == 4 && q.sizes() == k.sizes(), "sage2_prepare_qk expects matching 4D Q and K"); ++ TORCH_CHECK(q.size(0) == 1 && q.size(3) == 128, "sage2_prepare_qk currently requires batch 1 and head dimension 128"); ++ TORCH_CHECK(q.stride(3) == 1 && k.stride(3) == 1, "sage2_prepare_qk requires contiguous head dimensions"); ++ TORCH_CHECK(rotation.dim() == 6 && rotation.size(0) == 1 && rotation.size(1) == q.size(1) && rotation.size(2) == 1 && rotation.size(3) > 0 && rotation.size(3) <= 64 && rotation.size(4) == 2 && rotation.size(5) == 2, "sage2_prepare_qk expects rotation shape [1, sequence, 1, pairs, 2, 2] with pairs <= 64; got rotation ", rotation.sizes(), " for Q ", q.sizes()); ++ TORCH_CHECK(q_weight.is_contiguous() && k_weight.is_contiguous() && q_weight.numel() == 128 && k_weight.numel() == 128, "sage2_prepare_qk expects contiguous 128-element weights"); ++ TORCH_CHECK(epsilon > 0.0, "sage2_prepare_qk epsilon must be positive"); ++ return sage2_prepare_qk_cuda(q, k, rotation, q_weight, k_weight, epsilon, materialize_q); ++} ++ ++std::vector sage2_prepare_v(torch::Tensor v, double scale_max) { ++ TORCH_CHECK(v.is_cuda(), "sage2_prepare_v expects a CUDA tensor"); ++ TORCH_CHECK(v.scalar_type() == torch::kBFloat16, "sage2_prepare_v expects BF16 V"); ++ TORCH_CHECK(v.dim() == 4, "sage2_prepare_v expects a 4D NHD tensor"); ++ TORCH_CHECK(v.size(0) == 1 && v.size(3) == 128, "sage2_prepare_v currently requires batch 1 and head dimension 128"); ++ TORCH_CHECK(v.stride(3) == 1, "sage2_prepare_v requires a contiguous head dimension"); ++ TORCH_CHECK(scale_max > 0.0, "sage2_prepare_v scale_max must be positive"); ++ return sage2_prepare_v_cuda(v, scale_max); ++} ++ + PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { + m.def("nvfp4_activation_scale", &nvfp4_activation_scale, "Vortex NVFP4 activation scale"); + m.def("nvfp4_activation_scale_into", &nvfp4_activation_scale_into, "Vortex NVFP4 activation scale with caller workspace"); + m.def("quantize_nvfp4_bf16", &quantize_nvfp4_bf16, "Vortex BF16 to TensorCore NVFP4 quantizer"); ++ m.def("quantize_nvfp4_bf16_into", &quantize_nvfp4_bf16_into, "Vortex BF16 to TensorCore NVFP4 quantizer with caller outputs", py::arg("input"), py::arg("scale"), py::arg("qdata"), py::arg("block_scale"), py::arg("threads"), py::arg("hi_first") = true); ++ m.def("quantize_nvfp4_modulated_bf16", &quantize_nvfp4_modulated_bf16, "Vortex fused H3 modulation and TensorCore NVFP4 quantizer"); ++ m.def("quantize_nvfp4_swiglu_bf16", &quantize_nvfp4_swiglu_bf16, "Vortex fused H3 SwiGLU and TensorCore NVFP4 quantizer"); + m.def("qkv_to_bshd", &qkv_to_bshd, "Fused H3 QKV split to BSHD tensors"); ++ m.def("sage2_prepare_qk", &sage2_prepare_qk, "Exact fused H3 Q/K RMSNorm, split-half RoPE, and Sage2 Q quantization", py::arg("q"), py::arg("k"), py::arg("rotation"), py::arg("q_weight"), py::arg("k_weight"), py::arg("epsilon"), py::arg("materialize_q") = false); ++ m.def("sage2_prepare_v", &sage2_prepare_v, "Direct Sage2 NHD V scale, FP8 quantization, and layout preparation", py::arg("v"), py::arg("scale_max") = 2.25); + } +diff --git a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu +index f6b85a5..3c323bf 100644 +--- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu ++++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu +@@ -3,6 +3,7 @@ + #include + + #include ++#include + + #include + #include +@@ -54,6 +55,63 @@ __inline__ __device__ float bf16_bits_to_float(uint16_t bits) { + return __uint_as_float(static_cast(bits) << 16); + } + ++__inline__ __device__ float round_bf16(float value) { ++ return __bfloat162float(__float2bfloat16_rn(value)); ++} ++ ++template ++__inline__ __device__ float modulated_bf16_value( ++ const uint16_t* __restrict__ input, ++ const table_t* __restrict__ shift, ++ const table_t* __restrict__ scale, ++ const int32_t* __restrict__ row_index, ++ int64_t row, ++ int64_t col, ++ int64_t cols) { ++ const int64_t table_offset = static_cast(row_index[row]) * cols + col; ++ const float x = bf16_bits_to_float(input[row * cols + col]); ++ const float scale_bf16 = round_bf16(static_cast(scale[table_offset])); ++ const float shift_bf16 = round_bf16(static_cast(shift[table_offset])); ++ const float factor = round_bf16(1.0f + scale_bf16); ++ const float multiplied = round_bf16(x * factor); ++ return round_bf16(multiplied + shift_bf16); ++} ++ ++__inline__ __device__ float swiglu_bf16_value( ++ const uint16_t* __restrict__ input, ++ int64_t row, ++ int64_t col, ++ int64_t output_cols) { ++ const int64_t row_offset = row * output_cols * 2; ++ const float gate = bf16_bits_to_float(input[row_offset + col]); ++ const float up = bf16_bits_to_float(input[row_offset + output_cols + col]); ++ const float silu = round_bf16(gate / (1.0f + expf(-gate))); ++ return round_bf16(silu * up); ++} ++ ++__inline__ __device__ uint32_t warp_reduce_max_u32(uint32_t value) { ++ for (int offset = 16; offset > 0; offset >>= 1) { ++ value = max(value, __shfl_down_sync(0xffffffff, value, offset)); ++ } ++ return value; ++} ++ ++__inline__ __device__ uint32_t block_reduce_max_u32(uint32_t value) { ++ __shared__ uint32_t warp_values[32]; ++ const int lane = threadIdx.x & 31; ++ const int warp = threadIdx.x >> 5; ++ value = warp_reduce_max_u32(value); ++ if (lane == 0) { ++ warp_values[warp] = value; ++ } ++ __syncthreads(); ++ value = threadIdx.x < ((blockDim.x + 31) >> 5) ? warp_values[lane] : 0; ++ if (warp == 0) { ++ value = warp_reduce_max_u32(value); ++ } ++ return value; ++} ++ + __inline__ __device__ uint8_t encode_fp4_e2m1(float value) { + const bool negative = signbit(value); + float abs_value = fabsf(value); +@@ -87,6 +145,12 @@ __inline__ __device__ float decode_fp8_e4m3(uint8_t value) { + return static_cast(encoded); + } + ++__inline__ __device__ float reciprocal_approx_ftz(float value) { ++ float result; ++ asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(result) : "f"(value)); ++ return result; ++} ++ + __global__ void quantize_nvfp4_bf16_kernel( + const uint16_t* __restrict__ input, + const float* __restrict__ scale, +@@ -97,7 +161,8 @@ __global__ void quantize_nvfp4_bf16_kernel( + int64_t q_rows, + int64_t q_cols, + int64_t scale_rows, +- int64_t scale_cols) { ++ int64_t scale_cols, ++ bool hi_first) { + const int64_t row = blockIdx.x; + const float tensor_scale = scale[0]; + for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { +@@ -113,43 +178,197 @@ __global__ void quantize_nvfp4_bf16_kernel( + values[i] = value; + local_max = fmaxf(local_max, fabsf(value)); + } +- uint8_t scale_byte = 0; +- float block_scale_value = 0.0f; +- if (local_max > 0.0f && tensor_scale > 0.0f) { +- scale_byte = encode_fp8_e4m3(local_max / (tensor_scale * 6.0f)); +- block_scale_value = decode_fp8_e4m3(scale_byte); +- } ++ const float raw_block_scale = (local_max / 6.0f) / tensor_scale; ++ const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); ++ const float block_scale_value = decode_fp8_e4m3(scale_byte); + if (row < scale_rows) { +- int64_t scale_row = row; +- int64_t scale_col = block_col; +- if (scale_cols == 336 || scale_cols == 896) { +- const int64_t row_in_tile = row % 128; +- const int64_t group = block_col / 4; +- const int64_t pair = group / 2; +- const int64_t phase = group % 2; +- const int64_t col_stride = scale_cols == 336 ? 16 : 128; +- const int64_t phase_offset = scale_cols == 336 ? 176 : 512; +- int64_t swizzled_col_base = pair * col_stride + phase * phase_offset; +- int64_t swizzled_col = (swizzled_col_base % scale_cols) + (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 32) * 16; +- scale_row = (row / 128) * 128 + pair * (scale_cols == 336 ? 3 : 1) + (scale_cols == 336 ? phase : 0) + swizzled_col_base / scale_cols + swizzled_col / scale_cols; +- scale_col = swizzled_col % scale_cols; +- if (scale_col >= scale_cols) { +- scale_col -= scale_cols; +- scale_row += 1; +- } +- } else if (scale_cols >= 32) { +- const int64_t row_in_tile = row % 128; +- scale_row = (row / 128) * 128 + (block_col / 4) * 16 + ((row_in_tile % 32) / 2); +- scale_col = (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 2) * 16; ++ const int64_t row_in_tile = row % 128; ++ const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); ++ const int64_t within_tile = ++ ((row_in_tile % 32) / 2) * 32 + ++ (block_col % 4) + ++ (row_in_tile / 32) * 4 + ++ (row_in_tile % 2) * 16; ++ block_scale[tile * 512 + within_tile] = scale_byte; ++ } ++ if (row < q_rows && block_col * 8 < q_cols) { ++ #pragma unroll ++ for (int pair = 0; pair < 8; ++pair) { ++ const float encode_scale = fminf( ++ reciprocal_approx_ftz(tensor_scale * block_scale_value), ++ 3.402823466e+38f); ++ const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); ++ const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); ++ qdata[row * q_cols + block_col * 8 + pair] = hi_first ++ ? static_cast((even << 4) | odd) ++ : static_cast(even | (odd << 4)); + } +- block_scale[scale_row * scale_cols + scale_col] = scale_byte; + } ++ } ++} ++ ++template ++__global__ void partial_absmax_modulated_bf16_kernel( ++ const uint16_t* __restrict__ input, ++ const table_t* __restrict__ shift, ++ const table_t* __restrict__ scale, ++ const int32_t* __restrict__ row_index, ++ float* __restrict__ partials, ++ int64_t rows, ++ int64_t cols) { ++ const int64_t numel = rows * cols; ++ const int64_t stride = static_cast(blockDim.x) * gridDim.x; ++ uint32_t local_bits = 0; ++ for (int64_t index = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; ++ index < numel; ++ index += stride) { ++ const int64_t row = index / cols; ++ const int64_t col = index - row * cols; ++ const float value = modulated_bf16_value(input, shift, scale, row_index, row, col, cols); ++ local_bits = max(local_bits, __float_as_uint(value) >> 16 & 0x7fffu); ++ } ++ const uint32_t block_bits = block_reduce_max_u32(local_bits); ++ if (threadIdx.x == 0) { ++ partials[blockIdx.x] = bf16_abs_bits_to_float(block_bits); ++ } ++} ++ ++__global__ void final_scale_bf16_compat_kernel( ++ const float* __restrict__ partials, ++ float* __restrict__ output, ++ int64_t count, ++ float divisor) { ++ float local_max = 0.0f; ++ for (int64_t index = threadIdx.x; index < count; index += blockDim.x) { ++ local_max = fmaxf(local_max, partials[index]); ++ } ++ const float block_max = block_reduce_max(local_max); ++ if (threadIdx.x == 0) { ++ output[0] = round_bf16(block_max / divisor); ++ } ++} ++ ++template ++__global__ void quantize_nvfp4_modulated_bf16_kernel( ++ const uint16_t* __restrict__ input, ++ const table_t* __restrict__ shift, ++ const table_t* __restrict__ adaln_scale, ++ const int32_t* __restrict__ row_index, ++ const float* __restrict__ tensor_scale_ptr, ++ uint8_t* __restrict__ qdata, ++ uint8_t* __restrict__ block_scale, ++ int64_t rows, ++ int64_t cols, ++ int64_t q_rows, ++ int64_t q_cols, ++ int64_t scale_rows, ++ int64_t scale_cols) { ++ const int64_t row = blockIdx.x; ++ const float tensor_scale = tensor_scale_ptr[0]; ++ for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { ++ float local_max = 0.0f; ++ float values[16]; ++ #pragma unroll ++ for (int i = 0; i < 16; ++i) { ++ const int64_t col = block_col * 16 + i; ++ float value = 0.0f; ++ if (row < rows && col < cols) { ++ value = modulated_bf16_value(input, shift, adaln_scale, row_index, row, col, cols); ++ } ++ values[i] = value; ++ local_max = fmaxf(local_max, fabsf(value)); ++ } ++ const float raw_block_scale = (local_max / 6.0f) / tensor_scale; ++ const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); ++ const float block_scale_value = decode_fp8_e4m3(scale_byte); ++ const int64_t row_in_tile = row % 128; ++ const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); ++ const int64_t within_tile = ++ ((row_in_tile % 32) / 2) * 32 + ++ (block_col % 4) + ++ (row_in_tile / 32) * 4 + ++ (row_in_tile % 2) * 16; ++ block_scale[tile * 512 + within_tile] = scale_byte; + if (row < q_rows && block_col * 8 < q_cols) { ++ const float encode_scale = fminf( ++ reciprocal_approx_ftz(tensor_scale * block_scale_value), ++ 3.402823466e+38f); + #pragma unroll + for (int pair = 0; pair < 8; ++pair) { +- const float denom = tensor_scale * block_scale_value; +- const uint8_t even = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2] / denom) : 0; +- const uint8_t odd = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2 + 1] / denom) : 0; ++ const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); ++ const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); ++ qdata[row * q_cols + block_col * 8 + pair] = static_cast((even << 4) | odd); ++ } ++ } ++ } ++} ++ ++__global__ void partial_absmax_swiglu_bf16_kernel( ++ const uint16_t* __restrict__ input, ++ float* __restrict__ partials, ++ int64_t rows, ++ int64_t output_cols) { ++ const int64_t numel = rows * output_cols; ++ const int64_t stride = static_cast(blockDim.x) * gridDim.x; ++ uint32_t local_bits = 0; ++ for (int64_t index = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; ++ index < numel; ++ index += stride) { ++ const int64_t row = index / output_cols; ++ const int64_t col = index - row * output_cols; ++ const float value = swiglu_bf16_value(input, row, col, output_cols); ++ local_bits = max(local_bits, __float_as_uint(value) >> 16 & 0x7fffu); ++ } ++ const uint32_t block_bits = block_reduce_max_u32(local_bits); ++ if (threadIdx.x == 0) { ++ partials[blockIdx.x] = bf16_abs_bits_to_float(block_bits); ++ } ++} ++ ++__global__ void quantize_nvfp4_swiglu_bf16_kernel( ++ const uint16_t* __restrict__ input, ++ const float* __restrict__ tensor_scale_ptr, ++ uint8_t* __restrict__ qdata, ++ uint8_t* __restrict__ block_scale, ++ int64_t rows, ++ int64_t output_cols, ++ int64_t q_rows, ++ int64_t q_cols, ++ int64_t scale_cols) { ++ const int64_t row = blockIdx.x; ++ const float tensor_scale = tensor_scale_ptr[0]; ++ for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { ++ float local_max = 0.0f; ++ float values[16]; ++ #pragma unroll ++ for (int i = 0; i < 16; ++i) { ++ const int64_t col = block_col * 16 + i; ++ float value = 0.0f; ++ if (row < rows && col < output_cols) { ++ value = swiglu_bf16_value(input, row, col, output_cols); ++ } ++ values[i] = value; ++ local_max = fmaxf(local_max, fabsf(value)); ++ } ++ const float raw_block_scale = (local_max / 6.0f) / tensor_scale; ++ const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); ++ const float block_scale_value = decode_fp8_e4m3(scale_byte); ++ const int64_t row_in_tile = row % 128; ++ const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); ++ const int64_t within_tile = ++ ((row_in_tile % 32) / 2) * 32 + ++ (block_col % 4) + ++ (row_in_tile / 32) * 4 + ++ (row_in_tile % 2) * 16; ++ block_scale[tile * 512 + within_tile] = scale_byte; ++ if (row < q_rows && block_col * 8 < q_cols) { ++ const float encode_scale = fminf( ++ reciprocal_approx_ftz(tensor_scale * block_scale_value), ++ 3.402823466e+38f); ++ #pragma unroll ++ for (int pair = 0; pair < 8; ++pair) { ++ const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); ++ const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); + qdata[row * q_cols + block_col * 8 + pair] = static_cast((even << 4) | odd); + } + } +@@ -277,6 +496,298 @@ __global__ void qkv_to_bshd_vec16_kernel( + } + } + ++__device__ __forceinline__ float sage2_warp_sum(float value) { ++ #pragma unroll ++ for (int offset = 16; offset > 0; offset >>= 1) { ++ value += __shfl_down_sync(0xffffffffu, value, offset); ++ } ++ return __shfl_sync(0xffffffffu, value, 0); ++} ++ ++__device__ __forceinline__ float sage2_warp_max(float value) { ++ #pragma unroll ++ for (int mask = 16; mask > 0; mask >>= 1) { ++ value = fmaxf(value, __shfl_xor_sync(0xffffffffu, value, mask)); ++ } ++ return value; ++} ++ ++__device__ __forceinline__ int8_t sage2_int8_rn(float value) { ++ uint32_t result; ++ asm volatile("cvt.rni.sat.s8.f32 %0, %1;" : "=r"(result) : "f"(value)); ++ return static_cast(result); ++} ++ ++__device__ __forceinline__ nv_bfloat16 sage2_rotate_value( ++ const nv_bfloat16* __restrict__ normalized, ++ const nv_bfloat16* __restrict__ rotation, ++ int dimension, ++ int rotation_pairs, ++ int64_t rotation_pair_stride, ++ int64_t rotation_row_stride, ++ int64_t rotation_component_stride) { ++ if (dimension >= rotation_pairs * 2) { ++ return normalized[dimension]; ++ } ++ const int pair = dimension < rotation_pairs ? dimension : dimension - rotation_pairs; ++ const float x0 = __bfloat162float(normalized[pair]); ++ const float x1 = __bfloat162float(normalized[pair + rotation_pairs]); ++ const nv_bfloat16* pair_rotation = rotation + static_cast(pair) * rotation_pair_stride; ++ const int row = dimension < rotation_pairs ? 0 : 1; ++ const float f0 = __bfloat162float(pair_rotation[static_cast(row) * rotation_row_stride]); ++ const float f1 = __bfloat162float(pair_rotation[static_cast(row) * rotation_row_stride + rotation_component_stride]); ++ return __float2bfloat16_rn(f0 * x0 + f1 * x1); ++} ++ ++__global__ __launch_bounds__(1024) void sage2_prepare_qk_bf16_kernel( ++ const nv_bfloat16* __restrict__ q, ++ nv_bfloat16* __restrict__ k, ++ const nv_bfloat16* __restrict__ rotation, ++ const nv_bfloat16* __restrict__ q_weight, ++ const nv_bfloat16* __restrict__ k_weight, ++ int8_t* __restrict__ q_int8, ++ float* __restrict__ q_scale, ++ nv_bfloat16* __restrict__ q_prepared, ++ int64_t sequence, ++ int64_t heads, ++ int64_t q_s0, ++ int64_t q_s1, ++ int64_t q_s2, ++ int64_t k_s0, ++ int64_t k_s1, ++ int64_t k_s2, ++ int64_t f_s0, ++ int64_t f_s1, ++ int64_t f_s2, ++ int64_t f_s3, ++ int64_t f_s4, ++ int64_t f_s5, ++ int rotation_pairs, ++ float epsilon) { ++ constexpr int kHeadDim = 128; ++ constexpr int kTokensPerBlock = 32; ++ __shared__ nv_bfloat16 normalized[kTokensPerBlock][kHeadDim]; ++ __shared__ float warp_maxima[32]; ++ __shared__ float block_amax; ++ ++ const int lane = threadIdx.x & 31; ++ const int token_slot = threadIdx.x >> 5; ++ const int64_t token = static_cast(blockIdx.x) * kTokensPerBlock + token_slot; ++ const int64_t head = blockIdx.y; ++ const int64_t batch = blockIdx.z; ++ const bool valid = token < sequence; ++ const int64_t q_base = batch * q_s0 + token * q_s1 + head * q_s2; ++ const int64_t k_base = batch * k_s0 + token * k_s1 + head * k_s2; ++ const int64_t out_base = (batch * sequence * heads + token * heads + head) * kHeadDim; ++ const int64_t freq_base = batch * f_s0 + token * f_s1; ++ ++ float q_raw[4]; ++ float q_sum = 0.0f; ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ const float value = valid ? __bfloat162float(q[q_base + dimension]) : 0.0f; ++ q_raw[item] = value; ++ q_sum = fmaf(value, value, q_sum); ++ } ++ const float q_rrms = rsqrtf(sage2_warp_sum(q_sum) / static_cast(kHeadDim) + epsilon); ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ normalized[token_slot][dimension] = __float2bfloat16_rn( ++ q_raw[item] * q_rrms * __bfloat162float(q_weight[dimension])); ++ } ++ __syncthreads(); ++ ++ const nv_bfloat16* token_rotation = rotation + freq_base; ++ nv_bfloat16 q_values[4]; ++ float local_amax = 0.0000001f; ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ const nv_bfloat16 value = valid ++ ? sage2_rotate_value(normalized[token_slot], token_rotation, dimension, rotation_pairs, f_s3, f_s4, f_s5) ++ : __float2bfloat16_rn(0.0f); ++ q_values[item] = value; ++ local_amax = fmaxf(local_amax, fabsf(__bfloat162float(value))); ++ if (valid && q_prepared != nullptr) { ++ q_prepared[out_base + dimension] = value; ++ } ++ } ++ ++ const float warp_amax = sage2_warp_max(local_amax); ++ if (lane == 0) { ++ warp_maxima[token_slot] = warp_amax; ++ } ++ __syncthreads(); ++ float block_value = threadIdx.x < 32 ? warp_maxima[lane] : -1e20f; ++ block_value = sage2_warp_max(block_value); ++ if (threadIdx.x == 0) { ++ block_amax = block_value; ++ q_scale[(batch * heads + head) * gridDim.x + blockIdx.x] = ++ block_value * reciprocal_approx_ftz(127.0f); ++ } ++ __syncthreads(); ++ const float reciprocal_scale = 127.0f * reciprocal_approx_ftz(block_amax); ++ if (valid) { ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ q_int8[out_base + dimension] = sage2_int8_rn(__bfloat162float(q_values[item]) * reciprocal_scale); ++ } ++ } ++ __syncthreads(); ++ ++ float k_raw[4]; ++ float k_sum = 0.0f; ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ const float value = valid ? __bfloat162float(k[k_base + dimension]) : 0.0f; ++ k_raw[item] = value; ++ k_sum = fmaf(value, value, k_sum); ++ } ++ const float k_rrms = rsqrtf(sage2_warp_sum(k_sum) / static_cast(kHeadDim) + epsilon); ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ normalized[token_slot][dimension] = __float2bfloat16_rn( ++ k_raw[item] * k_rrms * __bfloat162float(k_weight[dimension])); ++ } ++ __syncthreads(); ++ if (valid) { ++ #pragma unroll ++ for (int item = 0; item < 4; ++item) { ++ const int dimension = lane + item * 32; ++ k[k_base + dimension] = sage2_rotate_value( ++ normalized[token_slot], token_rotation, dimension, rotation_pairs, f_s3, f_s4, f_s5); ++ } ++ } ++} ++ ++__global__ void sage2_v_partial_absmax_bf16_kernel( ++ const nv_bfloat16* __restrict__ v, ++ float* __restrict__ partials, ++ int64_t sequence, ++ int64_t heads, ++ int64_t tiles, ++ int64_t v_s0, ++ int64_t v_s1, ++ int64_t v_s2) { ++ constexpr int kHeadDim = 128; ++ constexpr int kTileRows = 64; ++ const int dimension = threadIdx.x; ++ const int64_t tile = blockIdx.x; ++ const int64_t head = blockIdx.y; ++ const int64_t batch = blockIdx.z; ++ const int64_t first_token = tile * kTileRows; ++ float local_max = 0.0f; ++ ++ if (dimension < kHeadDim) { ++ #pragma unroll ++ for (int row = 0; row < kTileRows; ++row) { ++ const int64_t token = first_token + row; ++ if (token < sequence) { ++ const int64_t offset = batch * v_s0 + token * v_s1 + head * v_s2 + dimension; ++ local_max = fmaxf(local_max, fabsf(__bfloat162float(v[offset]))); ++ } ++ } ++ partials[((batch * heads + head) * kHeadDim + dimension) * tiles + tile] = local_max; ++ } ++} ++ ++__global__ void sage2_v_finalize_scale_kernel( ++ const float* __restrict__ partials, ++ float* __restrict__ scale, ++ float* __restrict__ reciprocal_scale, ++ int64_t heads, ++ int64_t tiles, ++ float scale_max) { ++ constexpr int kHeadDim = 128; ++ const int64_t head = blockIdx.x; ++ const int64_t batch = blockIdx.y; ++ const int64_t dimension = blockIdx.z; ++ const int64_t partial_base = ((batch * heads + head) * kHeadDim + dimension) * tiles; ++ float local_max = 0.0f; ++ for (int64_t tile = threadIdx.x; tile < tiles; tile += blockDim.x) { ++ local_max = fmaxf(local_max, partials[partial_base + tile]); ++ } ++ const float channel_max = block_reduce_max(local_max); ++ if (threadIdx.x == 0) { ++ scale[(batch * heads + head) * kHeadDim + dimension] = ++ channel_max * reciprocal_approx_ftz(scale_max); ++ reciprocal_scale[(batch * heads + head) * kHeadDim + dimension] = ++ channel_max == 0.0f ? 0.0f : scale_max * reciprocal_approx_ftz(channel_max); ++ } ++} ++ ++__device__ __forceinline__ uint32_t sage2_e4m3x4(float x0, float x1, float x2, float x3) { ++ uint32_t packed; ++ asm volatile( ++ "{\n" ++ ".reg .b16 lo;\n" ++ ".reg .b16 hi;\n" ++ "cvt.rn.satfinite.e4m3x2.f32 lo, %2, %1;\n" ++ "cvt.rn.satfinite.e4m3x2.f32 hi, %4, %3;\n" ++ "mov.b32 %0, {lo, hi};\n" ++ "}\n" ++ : "=r"(packed) : "f"(x0), "f"(x1), "f"(x2), "f"(x3)); ++ return packed; ++} ++ ++__global__ __launch_bounds__(1024) void sage2_v_quantize_permute_bf16_kernel( ++ const nv_bfloat16* __restrict__ v, ++ const float* __restrict__ reciprocal_scale, ++ uint8_t* __restrict__ output, ++ int64_t sequence, ++ int64_t heads, ++ int64_t padded_sequence, ++ int64_t v_s0, ++ int64_t v_s1, ++ int64_t v_s2, ++ float scale_max) { ++ constexpr int kHeadDim = 128; ++ constexpr int kTileRows = 64; ++ __shared__ nv_bfloat16 tile[kTileRows][kHeadDim]; ++ ++ const int lane_group = threadIdx.x & 15; ++ const int source_row = threadIdx.x >> 4; ++ const int dimension_base = lane_group * 8; ++ const int64_t tile_index = blockIdx.x; ++ const int64_t head = blockIdx.y; ++ const int64_t batch = blockIdx.z; ++ const int64_t token = tile_index * kTileRows + source_row; ++ const int row_mod = source_row & 15; ++ const int permuted_row = (source_row & ~15) + (row_mod / 8) * 2 + ((row_mod / 2) & 3) * 4 + (row_mod & 1); ++ ++ #pragma unroll ++ for (int item = 0; item < 8; ++item) { ++ const int dimension = dimension_base + item; ++ tile[permuted_row][dimension] = token < sequence ++ ? v[batch * v_s0 + token * v_s1 + head * v_s2 + dimension] ++ : __float2bfloat16_rn(0.0f); ++ } ++ __syncthreads(); ++ ++ const int output_dimension = threadIdx.x >> 3; ++ const int output_group = threadIdx.x & 7; ++ const int output_row = output_group * 8; ++ const float channel_reciprocal_scale = ++ reciprocal_scale[(batch * heads + head) * kHeadDim + output_dimension]; ++ float values[8]; ++ #pragma unroll ++ for (int item = 0; item < 8; ++item) { ++ values[item] = __bfloat162float(tile[output_row + item][output_dimension]) * channel_reciprocal_scale; ++ } ++ const uint2 packed = make_uint2( ++ sage2_e4m3x4(values[0], values[1], values[2], values[3]), ++ sage2_e4m3x4(values[4], values[5], values[6], values[7])); ++ const int64_t output_base = ++ ((batch * kHeadDim + output_dimension) * heads + head) * padded_sequence ++ + tile_index * kTileRows + output_row; ++ *reinterpret_cast(output + output_base) = packed; ++} ++ + } // namespace + + torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor) { +@@ -327,6 +838,8 @@ torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divis + return output; + } + ++void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first); ++ + std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads) { + c10::cuda::CUDAGuard device_guard(input.device()); + const int64_t rows = input.size(0); +@@ -336,8 +849,21 @@ std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch:: + const int64_t q_cols = pad_16x ? roundup(cols, 16) / 2 : cols / 2; + const int64_t scale_rows = roundup(q_rows, 128); + const int64_t scale_cols = roundup(q_cols / 8, 4); ++ TORCH_CHECK(scale_cols % 4 == 0, "NVFP4 block-scale width must be divisible by four"); + auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); + auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); ++ quantize_nvfp4_bf16_into_cuda(input, scale, qdata, block_scale, threads, true); ++ return {qdata, block_scale}; ++} ++ ++void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first) { ++ c10::cuda::CUDAGuard device_guard(input.device()); ++ const int64_t rows = input.size(0); ++ const int64_t cols = input.size(1); ++ const int64_t q_rows = qdata.size(0); ++ const int64_t q_cols = qdata.size(1); ++ const int64_t scale_rows = block_scale.size(0); ++ const int64_t scale_cols = block_scale.size(1); + auto stream = at::cuda::getCurrentCUDAStream(); + quantize_nvfp4_bf16_kernel<<(threads), 0, stream>>>( + reinterpret_cast(input.data_ptr()), +@@ -349,11 +875,110 @@ std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch:: + q_rows, + q_cols, + scale_rows, ++ scale_cols, ++ hi_first); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++} ++ ++std::vector quantize_nvfp4_modulated_bf16_cuda( ++ torch::Tensor input, ++ torch::Tensor shift, ++ torch::Tensor adaln_scale, ++ torch::Tensor row_index, ++ double divisor, ++ bool pad_16x, ++ int64_t blocks, ++ int64_t scale_threads, ++ int64_t pack_threads) { ++ c10::cuda::CUDAGuard device_guard(input.device()); ++ const int64_t rows = input.size(0); ++ const int64_t cols = input.size(1); ++ const int64_t q_rows = pad_16x ? roundup(rows, 16) : rows; ++ const int64_t q_cols = pad_16x ? roundup(cols, 16) / 2 : cols / 2; ++ const int64_t scale_rows = roundup(q_rows, 128); ++ const int64_t scale_cols = roundup(q_cols / 8, 4); ++ auto partials = torch::empty({blocks}, input.options().dtype(torch::kFloat32)); ++ auto tensor_scale = torch::empty({}, input.options().dtype(torch::kFloat32)); ++ auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); ++ auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); ++ auto stream = at::cuda::getCurrentCUDAStream(); ++ ++ AT_DISPATCH_FLOATING_TYPES_AND(at::ScalarType::BFloat16, shift.scalar_type(), "vortex_nvfp4_modulated_bf16", [&] { ++ partial_absmax_modulated_bf16_kernel<<(blocks), static_cast(scale_threads), 0, stream>>>( ++ reinterpret_cast(input.data_ptr()), ++ shift.data_ptr(), ++ adaln_scale.data_ptr(), ++ row_index.data_ptr(), ++ partials.data_ptr(), ++ rows, ++ cols); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ final_scale_bf16_compat_kernel<<<1, static_cast(scale_threads), 0, stream>>>( ++ partials.data_ptr(), tensor_scale.data_ptr(), blocks, static_cast(divisor)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ quantize_nvfp4_modulated_bf16_kernel<<(scale_rows), static_cast(pack_threads), 0, stream>>>( ++ reinterpret_cast(input.data_ptr()), ++ shift.data_ptr(), ++ adaln_scale.data_ptr(), ++ row_index.data_ptr(), ++ tensor_scale.data_ptr(), ++ qdata.data_ptr(), ++ reinterpret_cast(block_scale.data_ptr()), ++ rows, ++ cols, ++ q_rows, ++ q_cols, ++ scale_rows, ++ scale_cols); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ }); ++ return {tensor_scale, qdata, block_scale}; ++} ++ ++std::vector quantize_nvfp4_swiglu_bf16_cuda( ++ torch::Tensor input, ++ double divisor, ++ bool pad_16x, ++ int64_t blocks, ++ int64_t scale_threads, ++ int64_t pack_threads) { ++ c10::cuda::CUDAGuard device_guard(input.device()); ++ const int64_t rows = input.size(0); ++ const int64_t output_cols = input.size(1) / 2; ++ const int64_t q_rows = pad_16x ? roundup(rows, 16) : rows; ++ const int64_t q_cols = pad_16x ? roundup(output_cols, 16) / 2 : output_cols / 2; ++ const int64_t scale_rows = roundup(q_rows, 128); ++ const int64_t scale_cols = roundup(q_cols / 8, 4); ++ auto partials = torch::empty({blocks}, input.options().dtype(torch::kFloat32)); ++ auto tensor_scale = torch::empty({}, input.options().dtype(torch::kFloat32)); ++ auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); ++ auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); ++ auto stream = at::cuda::getCurrentCUDAStream(); ++ ++ partial_absmax_swiglu_bf16_kernel<<(blocks), static_cast(scale_threads), 0, stream>>>( ++ reinterpret_cast(input.data_ptr()), ++ partials.data_ptr(), ++ rows, ++ output_cols); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ final_scale_bf16_compat_kernel<<<1, static_cast(scale_threads), 0, stream>>>( ++ partials.data_ptr(), tensor_scale.data_ptr(), blocks, static_cast(divisor)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ quantize_nvfp4_swiglu_bf16_kernel<<(scale_rows), static_cast(pack_threads), 0, stream>>>( ++ reinterpret_cast(input.data_ptr()), ++ tensor_scale.data_ptr(), ++ qdata.data_ptr(), ++ reinterpret_cast(block_scale.data_ptr()), ++ rows, ++ output_cols, ++ q_rows, ++ q_cols, + scale_cols); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +- return {qdata, block_scale}; ++ return {tensor_scale, qdata, block_scale}; + } + ++ + std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim) { + c10::cuda::CUDAGuard device_guard(qkv.device()); + const int64_t sequence = qkv.size(0); +@@ -396,3 +1021,97 @@ std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, in + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return {q, k, v}; + } ++ ++std::vector sage2_prepare_qk_cuda( ++ torch::Tensor q, ++ torch::Tensor k, ++ torch::Tensor rotation, ++ torch::Tensor q_weight, ++ torch::Tensor k_weight, ++ double epsilon, ++ bool materialize_q) { ++ c10::cuda::CUDAGuard device_guard(q.device()); ++ const int64_t batch = q.size(0); ++ const int64_t sequence = q.size(1); ++ const int64_t heads = q.size(2); ++ const int64_t groups = ((sequence + 127) / 128) * 4; ++ auto q_int8 = torch::empty(q.sizes(), q.options().dtype(torch::kInt8)); ++ auto q_scale = torch::empty({batch, heads, groups}, q.options().dtype(torch::kFloat32)); ++ auto q_prepared = materialize_q ++ ? torch::empty(q.sizes(), q.options()) ++ : torch::empty({0}, q.options()); ++ auto stream = at::cuda::getCurrentCUDAStream(); ++ const dim3 grid(static_cast(groups), static_cast(heads), static_cast(batch)); ++ sage2_prepare_qk_bf16_kernel<<>>( ++ reinterpret_cast(q.data_ptr()), ++ reinterpret_cast(k.data_ptr()), ++ reinterpret_cast(rotation.data_ptr()), ++ reinterpret_cast(q_weight.data_ptr()), ++ reinterpret_cast(k_weight.data_ptr()), ++ q_int8.data_ptr(), ++ q_scale.data_ptr(), ++ materialize_q ? reinterpret_cast(q_prepared.data_ptr()) : nullptr, ++ sequence, ++ heads, ++ q.stride(0), q.stride(1), q.stride(2), ++ k.stride(0), k.stride(1), k.stride(2), ++ rotation.stride(0), rotation.stride(1), rotation.stride(2), ++ rotation.stride(3), rotation.stride(4), rotation.stride(5), ++ static_cast(rotation.size(3)), ++ static_cast(epsilon)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ return {q_int8, q_scale, q_prepared}; ++} ++ ++std::vector sage2_prepare_v_cuda(torch::Tensor v, double scale_max) { ++ c10::cuda::CUDAGuard device_guard(v.device()); ++ constexpr int64_t kHeadDim = 128; ++ constexpr int64_t kTileRows = 64; ++ const int64_t batch = v.size(0); ++ const int64_t sequence = v.size(1); ++ const int64_t heads = v.size(2); ++ const int64_t tiles = (sequence + kTileRows - 1) / kTileRows; ++ const int64_t padded_sequence = tiles * kTileRows; ++ auto partials = torch::empty({batch, heads, kHeadDim, tiles}, v.options().dtype(torch::kFloat32)); ++ auto v_scale = torch::empty({batch, heads, kHeadDim}, v.options().dtype(torch::kFloat32)); ++ auto reciprocal_scale = torch::empty_like(v_scale); ++ auto v_fp8 = torch::empty( ++ {batch, kHeadDim, heads, padded_sequence}, ++ v.options().dtype(torch::kFloat8_e4m3fn)); ++ auto stream = at::cuda::getCurrentCUDAStream(); ++ const dim3 tile_grid( ++ static_cast(tiles), ++ static_cast(heads), ++ static_cast(batch)); ++ sage2_v_partial_absmax_bf16_kernel<<>>( ++ reinterpret_cast(v.data_ptr()), ++ partials.data_ptr(), ++ sequence, ++ heads, ++ tiles, ++ v.stride(0), v.stride(1), v.stride(2)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ const dim3 scale_grid( ++ static_cast(heads), ++ static_cast(batch), ++ static_cast(kHeadDim)); ++ sage2_v_finalize_scale_kernel<<>>( ++ partials.data_ptr(), ++ v_scale.data_ptr(), ++ reciprocal_scale.data_ptr(), ++ heads, ++ tiles, ++ static_cast(scale_max)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ sage2_v_quantize_permute_bf16_kernel<<>>( ++ reinterpret_cast(v.data_ptr()), ++ reciprocal_scale.data_ptr(), ++ reinterpret_cast(v_fp8.data_ptr()), ++ sequence, ++ heads, ++ padded_sequence, ++ v.stride(0), v.stride(1), v.stride(2), ++ static_cast(scale_max)); ++ C10_CUDA_KERNEL_LAUNCH_CHECK(); ++ return {v_fp8, v_scale}; ++} +diff --git a/src/h3_blackwell_runtime/nvfp4_quant.py b/src/h3_blackwell_runtime/nvfp4_quant.py +index 97ea031..b964944 100644 +--- a/src/h3_blackwell_runtime/nvfp4_quant.py ++++ b/src/h3_blackwell_runtime/nvfp4_quant.py +@@ -214,3 +214,131 @@ def vortex_native_quantize_nvfp4( + ), + ), + ) ++ ++ ++def vortex_native_quantize_nvfp4_into( ++ tensor: torch.Tensor, ++ scale: torch.Tensor, ++ qdata: torch.Tensor, ++ block_scale: torch.Tensor, ++ *, ++ hi_first: bool = True, ++) -> None: ++ """Pack BF16 activation into caller-owned NVFP4 ring buffers.""" ++ if tensor.dim() != 2 or tensor.dtype != torch.bfloat16: ++ raise ValueError("vortex_native_quantize_nvfp4_into expects a 2D BF16 tensor") ++ if not tensor.is_cuda or not tensor.is_contiguous(): ++ raise ValueError("vortex_native_quantize_nvfp4_into expects contiguous CUDA input") ++ _vortex_scale_extension().quantize_nvfp4_bf16_into( ++ tensor, ++ torch.as_tensor(scale, device=tensor.device, dtype=torch.float32), ++ qdata, ++ block_scale, ++ _env_int("H3_NVFP4_PACK_THREADS", 256), ++ hi_first, ++ ) ++ ++ ++def vortex_native_quantize_modulated_nvfp4( ++ tensor: torch.Tensor, ++ shift: torch.Tensor, ++ scale: torch.Tensor, ++ row_index: torch.Tensor, ++): ++ """Produce exact Comfy-layout NVFP4 bytes without materializing modulation.""" ++ if tensor.dim() != 2 or tensor.dtype != torch.bfloat16: ++ raise ValueError("vortex_native_quantize_modulated_nvfp4 expects a 2D BF16 tensor") ++ if not tensor.is_cuda or not tensor.is_contiguous(): ++ raise ValueError("vortex_native_quantize_modulated_nvfp4 expects contiguous CUDA input") ++ if shift.shape != scale.shape or shift.dim() != 2 or shift.shape[1] != tensor.shape[1]: ++ raise ValueError("shift and scale must have matching [table_rows, hidden] shapes") ++ if shift.device != tensor.device or scale.device != tensor.device: ++ raise ValueError("modulation tables must be on the input device") ++ if shift.dtype != scale.dtype or shift.dtype not in {torch.bfloat16, torch.float32}: ++ raise ValueError("modulation tables must share BF16 or FP32 dtype") ++ shift = shift.contiguous() ++ scale = scale.contiguous() ++ if row_index.shape != (tensor.shape[0],) or row_index.dtype != torch.int32: ++ raise ValueError("row_index must be int32 with one entry per input row") ++ if row_index.device != tensor.device or not row_index.is_contiguous(): ++ raise ValueError("row_index must be contiguous on the input device") ++ ++ from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ blocks, scale_threads = _vortex_scale_geometry(tensor.numel()) ++ return _vortex_scale_extension().quantize_nvfp4_modulated_bf16( ++ tensor, ++ shift, ++ scale, ++ row_index, ++ float(F8_E4M3_MAX * F4_E2M1_MAX), ++ TensorCoreNVFP4Layout.get_padded_shape(tuple(tensor.shape)) != tuple(tensor.shape), ++ blocks, ++ scale_threads, ++ _env_int("H3_NVFP4_PACK_THREADS", 256), ++ ) ++ ++ ++def vortex_quantize_modulated_nvfp4( ++ tensor: torch.Tensor, ++ shift: torch.Tensor, ++ scale: torch.Tensor, ++ row_index: torch.Tensor, ++): ++ """Wrap fused modulation output as a Comfy TensorCoreNVFP4Layout tensor.""" ++ from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout ++ ++ tensor_scale, qdata, block_scale = vortex_native_quantize_modulated_nvfp4( ++ tensor, shift, scale, row_index, ++ ) ++ return QuantizedTensor( ++ qdata, ++ "TensorCoreNVFP4Layout", ++ TensorCoreNVFP4Layout.Params( ++ scale=tensor_scale, ++ orig_dtype=tensor.dtype, ++ orig_shape=tuple(tensor.shape), ++ block_scale=block_scale, ++ ), ++ ) ++ ++ ++def vortex_native_quantize_swiglu_nvfp4(tensor: torch.Tensor): ++ """Produce exact Comfy-layout NVFP4 bytes without materializing SwiGLU.""" ++ if tensor.dim() != 2 or tensor.dtype != torch.bfloat16 or tensor.shape[1] % 2: ++ raise ValueError("vortex_native_quantize_swiglu_nvfp4 expects even-width 2D BF16 input") ++ if not tensor.is_cuda or not tensor.is_contiguous(): ++ raise ValueError("vortex_native_quantize_swiglu_nvfp4 expects contiguous CUDA input") ++ ++ from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX ++ from comfy_kitchen.tensor import TensorCoreNVFP4Layout ++ ++ output_shape = (tensor.shape[0], tensor.shape[1] // 2) ++ blocks, scale_threads = _vortex_scale_geometry(tensor.numel() // 2) ++ return _vortex_scale_extension().quantize_nvfp4_swiglu_bf16( ++ tensor, ++ float(F8_E4M3_MAX * F4_E2M1_MAX), ++ TensorCoreNVFP4Layout.get_padded_shape(output_shape) != output_shape, ++ blocks, ++ scale_threads, ++ _env_int("H3_NVFP4_PACK_THREADS", 256), ++ ) ++ ++ ++def vortex_quantize_swiglu_nvfp4(tensor: torch.Tensor): ++ """Wrap fused SwiGLU output as a Comfy TensorCoreNVFP4Layout tensor.""" ++ from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout ++ ++ tensor_scale, qdata, block_scale = vortex_native_quantize_swiglu_nvfp4(tensor) ++ output_shape = (tensor.shape[0], tensor.shape[1] // 2) ++ return QuantizedTensor( ++ qdata, ++ "TensorCoreNVFP4Layout", ++ TensorCoreNVFP4Layout.Params( ++ scale=tensor_scale, ++ orig_dtype=tensor.dtype, ++ orig_shape=output_shape, ++ block_scale=block_scale, ++ ), ++ ) diff --git a/src/h3_blackwell_runtime/attention.py b/src/h3_blackwell_runtime/attention.py index 7b7560d..a5372f8 100644 --- a/src/h3_blackwell_runtime/attention.py +++ b/src/h3_blackwell_runtime/attention.py @@ -169,6 +169,13 @@ def run_attention(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, *, backend: raise ValueError(f"Unsupported H3 attention backend: {backend}") +def run_sage_attention_nhd(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor: + """Run Sage2 directly on projection-strided NHD Q/K/V views.""" + from sageattention import sageattn + + return sageattn(q, k, v, is_causal=False, tensor_layout="NHD", smooth_k=False) + + def apply_split_half_rope(x: torch.Tensor, rotation: torch.Tensor) -> torch.Tensor: """Apply H3's split-half rotary table to `[batch, sequence, heads, dim]`.""" rotated_width = rotation.shape[-3] * 2 @@ -231,8 +238,8 @@ class H3SageAttention(nn.Module): @classmethod def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16, backend: str = DEFAULT_ATTENTION_BACKEND): return cls( - checkpoint.nvfp4_linear(f"{prefix}.qkv_proj", output_dtype=output_dtype), - checkpoint.nvfp4_linear(f"{prefix}.out_proj", output_dtype=output_dtype), + checkpoint.nvfp4_linear(f"{prefix}.qkv_proj", output_dtype=output_dtype, role="h3_attn_qkv"), + checkpoint.nvfp4_linear(f"{prefix}.out_proj", output_dtype=output_dtype, role="h3_attn_out"), checkpoint.tensor(f"{prefix}.q_norm.weight", dtype=output_dtype), checkpoint.tensor(f"{prefix}.k_norm.weight", dtype=output_dtype), backend=backend, @@ -244,6 +251,7 @@ class H3SageAttention(nn.Module): rope_rotation: torch.Tensor, sequence_parallel: "SequenceParallelContext | None" = None, tensor_parallel: "SequenceParallelContext | None" = None, + modulation: tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None = None, ) -> torch.Tensor: if x.ndim != 2: raise ValueError("H3 attention expects `[sequence, hidden]` input.") @@ -253,13 +261,22 @@ class H3SageAttention(nn.Module): return self._forward_tensor_parallel(x, rope_rotation, tensor_parallel) sequence = x.shape[0] inner = self.heads * self.head_dim - qkv = self.qkv_proj(x) + qkv = self.qkv_proj.forward_modulated(x, *modulation) if modulation is not None else self.qkv_proj(x) q, k, v = qkv.split(inner, dim=-1) q = q.view(1, sequence, self.heads, self.head_dim) k = k.view(1, sequence, self.heads, self.head_dim) v = v.view(1, sequence, self.heads, self.head_dim) q, k = rms_rope_split_half_(q, k, rope_rotation, self.q_norm_weight, self.k_norm_weight, self.eps) + if ( + sequence_parallel is None + and self.backend == "sage2" + and os.getenv("H3_SAGE_QKV_LAYOUT", "hnd").lower() == "strided_nhd" + ): + out = run_sage_attention_nhd(q, k, v) + if not out.is_contiguous(): + raise RuntimeError("Sage2 NHD output must be contiguous for zero-copy output projection") + return self.out_proj(out.reshape(sequence, inner)) if sequence_parallel is not None: q, k, v = sequence_parallel.seq_to_heads(q, k, v) if self.backend == "sol_attn": diff --git a/src/h3_blackwell_runtime/block.py b/src/h3_blackwell_runtime/block.py index 6177e73..2c3650a 100644 --- a/src/h3_blackwell_runtime/block.py +++ b/src/h3_blackwell_runtime/block.py @@ -1,5 +1,7 @@ """Direct MiniMax H3 DiT block over the standalone Sage3 attention unit.""" +import os + import torch from torch import nn from typing import TYPE_CHECKING @@ -45,12 +47,15 @@ class H3SwiGLU(nn.Module): self.fc2 = fc2 self.chunks = 1 self.chunk_threshold = 4096 + self.fused_nvfp4_swiglu = os.getenv("H3_NVFP4_SWIGLU_FUSION", "").lower() in { + "1", "true", "yes", "on", + } @classmethod def from_checkpoint(cls, checkpoint: H3Checkpoint, prefix: str, *, output_dtype=torch.bfloat16): return cls( - checkpoint.nvfp4_linear(f"{prefix}.fc1", output_dtype=output_dtype), - checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype), + checkpoint.nvfp4_linear(f"{prefix}.fc1", output_dtype=output_dtype, role="h3_mlp_fc1"), + checkpoint.nvfp4_linear(f"{prefix}.fc2", output_dtype=output_dtype, role="h3_mlp_fc2"), ) def forward( @@ -71,7 +76,27 @@ class H3SwiGLU(nn.Module): return self._forward_chunk(x) def _forward_chunk(self, x: torch.Tensor) -> torch.Tensor: - gate, up = self.fc1(x).chunk(2, dim=-1) + gate_up = self.fc1(x) + if self.fused_nvfp4_swiglu: + return self.fc2.forward_swiglu(gate_up) + gate, up = gate_up.chunk(2, dim=-1) + return self.fc2(torch.nn.functional.silu(gate).mul_(up)) + + def forward_modulated( + self, + x: torch.Tensor, + shift: torch.Tensor, + scale: torch.Tensor, + row_index: torch.Tensor, + ) -> torch.Tensor: + if self.chunks != 1: + from .h3_fusion import fused_modulate_ + + return self(fused_modulate_(x, shift, scale, row_index)) + gate_up = self.fc1.forward_modulated(x, shift, scale, row_index) + if self.fused_nvfp4_swiglu: + return self.fc2.forward_swiglu(gate_up) + gate, up = gate_up.chunk(2, dim=-1) return self.fc2(torch.nn.functional.silu(gate).mul_(up)) @@ -101,6 +126,12 @@ class H3DiTBlock(nn.Module): self.attention = attention self.mlp = mlp self.norm_eps = norm_eps + self.fused_elementwise = os.getenv("H3_FUSED_ELEMENTWISE", "").lower() in { + "1", "true", "yes", "on", + } + self.fused_nvfp4_modulation = os.getenv("H3_NVFP4_MODULATE_FUSION", "").lower() in { + "1", "true", "yes", "on", + } self.register_buffer("norm1_weight", norm1_weight, persistent=False) self.register_buffer("norm2_weight", norm2_weight, persistent=False) @@ -128,6 +159,46 @@ class H3DiTBlock(nn.Module): sequence_parallel: "SequenceParallelContext | None" = None, tensor_parallel: "SequenceParallelContext | None" = None, ) -> torch.Tensor: + if self.fused_elementwise and x.is_cuda and x.dtype == torch.bfloat16 and not x.requires_grad: + from .h3_fusion import fused_gate_add_, fused_modulate_, segment_index + + row_index = segment_index(x.shape[0], segments, x.device) + if ( + self.fused_nvfp4_modulation + and sequence_parallel is None + and tensor_parallel is None + ): + h = rms_norm(x, self.norm1_weight, self.norm_eps) + x = fused_gate_add_( + x, + self.attention( + h, + rope_rotation, + modulation=(shift_msa, scale_msa, row_index), + ), + gate_msa, + row_index, + ) + h = rms_norm(x, self.norm2_weight, self.norm_eps) + return fused_gate_add_( + x, + self.mlp.forward_modulated(h, shift_mlp, scale_mlp, row_index), + gate_mlp, + row_index, + ) + h = fused_modulate_( + rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, row_index, + ) + x = fused_gate_add_( + x, + self.attention(h, rope_rotation, sequence_parallel, tensor_parallel), + gate_msa, + row_index, + ) + h = fused_modulate_( + rms_norm(x, self.norm2_weight, self.norm_eps), shift_mlp, scale_mlp, row_index, + ) + return fused_gate_add_(x, self.mlp(h, tensor_parallel), gate_mlp, row_index) h = modulate_segments(rms_norm(x, self.norm1_weight, self.norm_eps), shift_msa, scale_msa, segments) x = gate_segments( x, diff --git a/src/h3_blackwell_runtime/checkpoint.py b/src/h3_blackwell_runtime/checkpoint.py index 661bc8c..dbcfc68 100644 --- a/src/h3_blackwell_runtime/checkpoint.py +++ b/src/h3_blackwell_runtime/checkpoint.py @@ -63,7 +63,7 @@ class H3Checkpoint: """Release tensors retained by whole-file loading after modules are built.""" self._no_mmap_tensors = None - def nvfp4_linear(self, prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear: + def nvfp4_linear(self, prefix: str, *, output_dtype=torch.bfloat16, role: str = "other") -> Nvfp4Linear: names = ("comfy_quant", "weight", "weight_scale", "weight_scale_2", "bias", "pre_quant_scale") tensors = {} if self._use_fast_safetensors(): @@ -72,14 +72,14 @@ class H3Checkpoint: name = f"{prefix}.{suffix}" if name in available_tensors: tensors[name] = available_tensors[name] - return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype) + return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype, role=role) if self._disable_mmap(): available_tensors = self._all_tensors_no_mmap() for suffix in names: name = f"{prefix}.{suffix}" if name in available_tensors: tensors[name] = available_tensors[name] - return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype) + return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype, role=role) from safetensors import safe_open @@ -89,4 +89,4 @@ class H3Checkpoint: name = f"{prefix}.{suffix}" if name in available: tensors[name] = checkpoint.get_tensor(name) - return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype) + return load_nvfp4_linear(tensors, prefix, output_dtype=output_dtype, role=role) diff --git a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp index 9291ed5..ecabdc8 100644 --- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp +++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cpp @@ -5,7 +5,12 @@ torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor); torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divisor, torch::Tensor partials, torch::Tensor output, int64_t blocks, int64_t threads); std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads); +void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first); +std::vector quantize_nvfp4_modulated_bf16_cuda(torch::Tensor input, torch::Tensor shift, torch::Tensor scale, torch::Tensor row_index, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads); +std::vector quantize_nvfp4_swiglu_bf16_cuda(torch::Tensor input, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads); std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim); +std::vector sage2_prepare_qk_cuda(torch::Tensor q, torch::Tensor k, torch::Tensor rotation, torch::Tensor q_weight, torch::Tensor k_weight, double epsilon, bool materialize_q); +std::vector sage2_prepare_v_cuda(torch::Tensor v, double scale_max); torch::Tensor nvfp4_activation_scale(torch::Tensor input, double divisor) { TORCH_CHECK(input.is_cuda(), "nvfp4_activation_scale expects a CUDA tensor"); @@ -42,6 +47,50 @@ std::vector quantize_nvfp4_bf16(torch::Tensor input, torch::Tenso return quantize_nvfp4_bf16_cuda(input, scale, pad_16x, threads); } +void quantize_nvfp4_bf16_into(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first) { + TORCH_CHECK(input.is_cuda() && scale.is_cuda() && qdata.is_cuda() && block_scale.is_cuda(), "quantize_nvfp4_bf16_into expects CUDA tensors"); + TORCH_CHECK(input.device() == scale.device() && input.device() == qdata.device() && input.device() == block_scale.device(), "quantize_nvfp4_bf16_into tensors must share a device"); + TORCH_CHECK(input.is_contiguous() && qdata.is_contiguous() && block_scale.is_contiguous(), "quantize_nvfp4_bf16_into tensors must be contiguous"); + TORCH_CHECK(input.dim() == 2 && qdata.dim() == 2 && block_scale.dim() == 2, "quantize_nvfp4_bf16_into expects 2D input and outputs"); + TORCH_CHECK(input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_bf16_into expects bfloat16 input"); + TORCH_CHECK(scale.scalar_type() == torch::kFloat32 && scale.numel() == 1, "quantize_nvfp4_bf16_into expects a scalar float32 scale"); + TORCH_CHECK(qdata.scalar_type() == torch::kUInt8, "quantize_nvfp4_bf16_into expects uint8 qdata"); + TORCH_CHECK(block_scale.scalar_type() == torch::kFloat8_e4m3fn, "quantize_nvfp4_bf16_into expects float8_e4m3fn block scales"); + TORCH_CHECK(qdata.size(0) >= input.size(0) && qdata.size(1) >= input.size(1) / 2, "quantize_nvfp4_bf16_into qdata output is too small"); + const int64_t required_scale_rows = ((qdata.size(0) + 127) / 128) * 128; + const int64_t required_scale_cols = (((qdata.size(1) / 8) + 3) / 4) * 4; + TORCH_CHECK(block_scale.size(0) >= required_scale_rows && block_scale.size(1) >= required_scale_cols, "quantize_nvfp4_bf16_into block-scale output is too small"); + TORCH_CHECK(threads == 64 || threads == 128 || threads == 256 || threads == 512, "quantize_nvfp4_bf16_into threads must be 64, 128, 256, or 512"); + quantize_nvfp4_bf16_into_cuda(input, scale, qdata, block_scale, threads, hi_first); +} + +std::vector quantize_nvfp4_modulated_bf16(torch::Tensor input, torch::Tensor shift, torch::Tensor scale, torch::Tensor row_index, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads) { + TORCH_CHECK(input.is_cuda() && shift.is_cuda() && scale.is_cuda() && row_index.is_cuda(), "quantize_nvfp4_modulated_bf16 expects CUDA tensors"); + TORCH_CHECK(input.device() == shift.device() && input.device() == scale.device() && input.device() == row_index.device(), "quantize_nvfp4_modulated_bf16 tensors must share a device"); + TORCH_CHECK(input.is_contiguous() && shift.is_contiguous() && scale.is_contiguous() && row_index.is_contiguous(), "quantize_nvfp4_modulated_bf16 tensors must be contiguous"); + TORCH_CHECK(input.dim() == 2 && shift.dim() == 2 && scale.dim() == 2, "quantize_nvfp4_modulated_bf16 expects 2D input and tables"); + TORCH_CHECK(input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_modulated_bf16 expects bfloat16 input"); + TORCH_CHECK(shift.scalar_type() == scale.scalar_type() && (shift.scalar_type() == torch::kBFloat16 || shift.scalar_type() == torch::kFloat32), "quantize_nvfp4_modulated_bf16 tables must share bfloat16 or float32 dtype"); + TORCH_CHECK(shift.sizes() == scale.sizes() && shift.size(1) == input.size(1), "quantize_nvfp4_modulated_bf16 table shapes must match the input width"); + TORCH_CHECK(row_index.scalar_type() == torch::kInt32 && row_index.dim() == 1 && row_index.numel() == input.size(0), "quantize_nvfp4_modulated_bf16 row_index must be int32 with one entry per row"); + TORCH_CHECK(divisor > 0.0, "quantize_nvfp4_modulated_bf16 divisor must be positive"); + TORCH_CHECK(blocks > 0, "quantize_nvfp4_modulated_bf16 blocks must be positive"); + TORCH_CHECK(scale_threads == 128 || scale_threads == 256 || scale_threads == 512, "quantize_nvfp4_modulated_bf16 scale_threads must be 128, 256, or 512"); + TORCH_CHECK(pack_threads == 64 || pack_threads == 128 || pack_threads == 256 || pack_threads == 512, "quantize_nvfp4_modulated_bf16 pack_threads must be 64, 128, 256, or 512"); + return quantize_nvfp4_modulated_bf16_cuda(input, shift, scale, row_index, divisor, pad_16x, blocks, scale_threads, pack_threads); +} + +std::vector quantize_nvfp4_swiglu_bf16(torch::Tensor input, double divisor, bool pad_16x, int64_t blocks, int64_t scale_threads, int64_t pack_threads) { + TORCH_CHECK(input.is_cuda() && input.is_contiguous(), "quantize_nvfp4_swiglu_bf16 expects contiguous CUDA input"); + TORCH_CHECK(input.dim() == 2 && input.scalar_type() == torch::kBFloat16, "quantize_nvfp4_swiglu_bf16 expects 2D bfloat16 input"); + TORCH_CHECK(input.size(1) % 2 == 0, "quantize_nvfp4_swiglu_bf16 input width must be even"); + TORCH_CHECK(divisor > 0.0 && blocks > 0, "quantize_nvfp4_swiglu_bf16 divisor and blocks must be positive"); + TORCH_CHECK(scale_threads == 128 || scale_threads == 256 || scale_threads == 512, "quantize_nvfp4_swiglu_bf16 scale_threads must be 128, 256, or 512"); + TORCH_CHECK(pack_threads == 64 || pack_threads == 128 || pack_threads == 256 || pack_threads == 512, "quantize_nvfp4_swiglu_bf16 pack_threads must be 64, 128, 256, or 512"); + return quantize_nvfp4_swiglu_bf16_cuda(input, divisor, pad_16x, blocks, scale_threads, pack_threads); +} + + std::vector qkv_to_bshd(torch::Tensor qkv, int64_t heads, int64_t head_dim) { TORCH_CHECK(qkv.is_cuda(), "qkv_to_bshd expects a CUDA tensor"); TORCH_CHECK(qkv.is_contiguous(), "qkv_to_bshd expects contiguous input"); @@ -51,9 +100,38 @@ std::vector qkv_to_bshd(torch::Tensor qkv, int64_t heads, int64_t return qkv_to_bshd_cuda(qkv, heads, head_dim); } +std::vector sage2_prepare_qk(torch::Tensor q, torch::Tensor k, torch::Tensor rotation, torch::Tensor q_weight, torch::Tensor k_weight, double epsilon, bool materialize_q) { + TORCH_CHECK(q.is_cuda() && k.is_cuda() && rotation.is_cuda() && q_weight.is_cuda() && k_weight.is_cuda(), "sage2_prepare_qk expects CUDA tensors"); + TORCH_CHECK(q.device() == k.device() && q.device() == rotation.device() && q.device() == q_weight.device() && q.device() == k_weight.device(), "sage2_prepare_qk tensors must share a device"); + TORCH_CHECK(q.scalar_type() == torch::kBFloat16 && k.scalar_type() == torch::kBFloat16, "sage2_prepare_qk expects BF16 Q and K"); + TORCH_CHECK(rotation.scalar_type() == torch::kBFloat16 && q_weight.scalar_type() == torch::kBFloat16 && k_weight.scalar_type() == torch::kBFloat16, "sage2_prepare_qk expects BF16 rotation and weights"); + TORCH_CHECK(q.dim() == 4 && k.dim() == 4 && q.sizes() == k.sizes(), "sage2_prepare_qk expects matching 4D Q and K"); + TORCH_CHECK(q.size(0) == 1 && q.size(3) == 128, "sage2_prepare_qk currently requires batch 1 and head dimension 128"); + TORCH_CHECK(q.stride(3) == 1 && k.stride(3) == 1, "sage2_prepare_qk requires contiguous head dimensions"); + TORCH_CHECK(rotation.dim() == 6 && rotation.size(0) == 1 && rotation.size(1) == q.size(1) && rotation.size(2) == 1 && rotation.size(3) > 0 && rotation.size(3) <= 64 && rotation.size(4) == 2 && rotation.size(5) == 2, "sage2_prepare_qk expects rotation shape [1, sequence, 1, pairs, 2, 2] with pairs <= 64; got rotation ", rotation.sizes(), " for Q ", q.sizes()); + TORCH_CHECK(q_weight.is_contiguous() && k_weight.is_contiguous() && q_weight.numel() == 128 && k_weight.numel() == 128, "sage2_prepare_qk expects contiguous 128-element weights"); + TORCH_CHECK(epsilon > 0.0, "sage2_prepare_qk epsilon must be positive"); + return sage2_prepare_qk_cuda(q, k, rotation, q_weight, k_weight, epsilon, materialize_q); +} + +std::vector sage2_prepare_v(torch::Tensor v, double scale_max) { + TORCH_CHECK(v.is_cuda(), "sage2_prepare_v expects a CUDA tensor"); + TORCH_CHECK(v.scalar_type() == torch::kBFloat16, "sage2_prepare_v expects BF16 V"); + TORCH_CHECK(v.dim() == 4, "sage2_prepare_v expects a 4D NHD tensor"); + TORCH_CHECK(v.size(0) == 1 && v.size(3) == 128, "sage2_prepare_v currently requires batch 1 and head dimension 128"); + TORCH_CHECK(v.stride(3) == 1, "sage2_prepare_v requires a contiguous head dimension"); + TORCH_CHECK(scale_max > 0.0, "sage2_prepare_v scale_max must be positive"); + return sage2_prepare_v_cuda(v, scale_max); +} + PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) { m.def("nvfp4_activation_scale", &nvfp4_activation_scale, "Vortex NVFP4 activation scale"); m.def("nvfp4_activation_scale_into", &nvfp4_activation_scale_into, "Vortex NVFP4 activation scale with caller workspace"); m.def("quantize_nvfp4_bf16", &quantize_nvfp4_bf16, "Vortex BF16 to TensorCore NVFP4 quantizer"); + m.def("quantize_nvfp4_bf16_into", &quantize_nvfp4_bf16_into, "Vortex BF16 to TensorCore NVFP4 quantizer with caller outputs", py::arg("input"), py::arg("scale"), py::arg("qdata"), py::arg("block_scale"), py::arg("threads"), py::arg("hi_first") = true); + m.def("quantize_nvfp4_modulated_bf16", &quantize_nvfp4_modulated_bf16, "Vortex fused H3 modulation and TensorCore NVFP4 quantizer"); + m.def("quantize_nvfp4_swiglu_bf16", &quantize_nvfp4_swiglu_bf16, "Vortex fused H3 SwiGLU and TensorCore NVFP4 quantizer"); m.def("qkv_to_bshd", &qkv_to_bshd, "Fused H3 QKV split to BSHD tensors"); + m.def("sage2_prepare_qk", &sage2_prepare_qk, "Exact fused H3 Q/K RMSNorm, split-half RoPE, and Sage2 Q quantization", py::arg("q"), py::arg("k"), py::arg("rotation"), py::arg("q_weight"), py::arg("k_weight"), py::arg("epsilon"), py::arg("materialize_q") = false); + m.def("sage2_prepare_v", &sage2_prepare_v, "Direct Sage2 NHD V scale, FP8 quantization, and layout preparation", py::arg("v"), py::arg("scale_max") = 2.25); } diff --git a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu index f6b85a5..3c323bf 100644 --- a/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu +++ b/src/h3_blackwell_runtime/csrc/nvfp4_scale.cu @@ -3,6 +3,7 @@ #include #include +#include #include #include @@ -54,6 +55,63 @@ __inline__ __device__ float bf16_bits_to_float(uint16_t bits) { return __uint_as_float(static_cast(bits) << 16); } +__inline__ __device__ float round_bf16(float value) { + return __bfloat162float(__float2bfloat16_rn(value)); +} + +template +__inline__ __device__ float modulated_bf16_value( + const uint16_t* __restrict__ input, + const table_t* __restrict__ shift, + const table_t* __restrict__ scale, + const int32_t* __restrict__ row_index, + int64_t row, + int64_t col, + int64_t cols) { + const int64_t table_offset = static_cast(row_index[row]) * cols + col; + const float x = bf16_bits_to_float(input[row * cols + col]); + const float scale_bf16 = round_bf16(static_cast(scale[table_offset])); + const float shift_bf16 = round_bf16(static_cast(shift[table_offset])); + const float factor = round_bf16(1.0f + scale_bf16); + const float multiplied = round_bf16(x * factor); + return round_bf16(multiplied + shift_bf16); +} + +__inline__ __device__ float swiglu_bf16_value( + const uint16_t* __restrict__ input, + int64_t row, + int64_t col, + int64_t output_cols) { + const int64_t row_offset = row * output_cols * 2; + const float gate = bf16_bits_to_float(input[row_offset + col]); + const float up = bf16_bits_to_float(input[row_offset + output_cols + col]); + const float silu = round_bf16(gate / (1.0f + expf(-gate))); + return round_bf16(silu * up); +} + +__inline__ __device__ uint32_t warp_reduce_max_u32(uint32_t value) { + for (int offset = 16; offset > 0; offset >>= 1) { + value = max(value, __shfl_down_sync(0xffffffff, value, offset)); + } + return value; +} + +__inline__ __device__ uint32_t block_reduce_max_u32(uint32_t value) { + __shared__ uint32_t warp_values[32]; + const int lane = threadIdx.x & 31; + const int warp = threadIdx.x >> 5; + value = warp_reduce_max_u32(value); + if (lane == 0) { + warp_values[warp] = value; + } + __syncthreads(); + value = threadIdx.x < ((blockDim.x + 31) >> 5) ? warp_values[lane] : 0; + if (warp == 0) { + value = warp_reduce_max_u32(value); + } + return value; +} + __inline__ __device__ uint8_t encode_fp4_e2m1(float value) { const bool negative = signbit(value); float abs_value = fabsf(value); @@ -87,6 +145,12 @@ __inline__ __device__ float decode_fp8_e4m3(uint8_t value) { return static_cast(encoded); } +__inline__ __device__ float reciprocal_approx_ftz(float value) { + float result; + asm("rcp.approx.ftz.f32 %0, %1;" : "=f"(result) : "f"(value)); + return result; +} + __global__ void quantize_nvfp4_bf16_kernel( const uint16_t* __restrict__ input, const float* __restrict__ scale, @@ -97,7 +161,8 @@ __global__ void quantize_nvfp4_bf16_kernel( int64_t q_rows, int64_t q_cols, int64_t scale_rows, - int64_t scale_cols) { + int64_t scale_cols, + bool hi_first) { const int64_t row = blockIdx.x; const float tensor_scale = scale[0]; for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { @@ -113,43 +178,197 @@ __global__ void quantize_nvfp4_bf16_kernel( values[i] = value; local_max = fmaxf(local_max, fabsf(value)); } - uint8_t scale_byte = 0; - float block_scale_value = 0.0f; - if (local_max > 0.0f && tensor_scale > 0.0f) { - scale_byte = encode_fp8_e4m3(local_max / (tensor_scale * 6.0f)); - block_scale_value = decode_fp8_e4m3(scale_byte); - } + const float raw_block_scale = (local_max / 6.0f) / tensor_scale; + const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); + const float block_scale_value = decode_fp8_e4m3(scale_byte); if (row < scale_rows) { - int64_t scale_row = row; - int64_t scale_col = block_col; - if (scale_cols == 336 || scale_cols == 896) { - const int64_t row_in_tile = row % 128; - const int64_t group = block_col / 4; - const int64_t pair = group / 2; - const int64_t phase = group % 2; - const int64_t col_stride = scale_cols == 336 ? 16 : 128; - const int64_t phase_offset = scale_cols == 336 ? 176 : 512; - int64_t swizzled_col_base = pair * col_stride + phase * phase_offset; - int64_t swizzled_col = (swizzled_col_base % scale_cols) + (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 32) * 16; - scale_row = (row / 128) * 128 + pair * (scale_cols == 336 ? 3 : 1) + (scale_cols == 336 ? phase : 0) + swizzled_col_base / scale_cols + swizzled_col / scale_cols; - scale_col = swizzled_col % scale_cols; - if (scale_col >= scale_cols) { - scale_col -= scale_cols; - scale_row += 1; - } - } else if (scale_cols >= 32) { - const int64_t row_in_tile = row % 128; - scale_row = (row / 128) * 128 + (block_col / 4) * 16 + ((row_in_tile % 32) / 2); - scale_col = (block_col % 4) + (row_in_tile / 32) * 4 + (row_in_tile % 2) * 16; - } - block_scale[scale_row * scale_cols + scale_col] = scale_byte; + const int64_t row_in_tile = row % 128; + const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); + const int64_t within_tile = + ((row_in_tile % 32) / 2) * 32 + + (block_col % 4) + + (row_in_tile / 32) * 4 + + (row_in_tile % 2) * 16; + block_scale[tile * 512 + within_tile] = scale_byte; } if (row < q_rows && block_col * 8 < q_cols) { #pragma unroll for (int pair = 0; pair < 8; ++pair) { - const float denom = tensor_scale * block_scale_value; - const uint8_t even = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2] / denom) : 0; - const uint8_t odd = denom > 0.0f ? encode_fp4_e2m1(values[pair * 2 + 1] / denom) : 0; + const float encode_scale = fminf( + reciprocal_approx_ftz(tensor_scale * block_scale_value), + 3.402823466e+38f); + const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); + const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); + qdata[row * q_cols + block_col * 8 + pair] = hi_first + ? static_cast((even << 4) | odd) + : static_cast(even | (odd << 4)); + } + } + } +} + +template +__global__ void partial_absmax_modulated_bf16_kernel( + const uint16_t* __restrict__ input, + const table_t* __restrict__ shift, + const table_t* __restrict__ scale, + const int32_t* __restrict__ row_index, + float* __restrict__ partials, + int64_t rows, + int64_t cols) { + const int64_t numel = rows * cols; + const int64_t stride = static_cast(blockDim.x) * gridDim.x; + uint32_t local_bits = 0; + for (int64_t index = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; + index < numel; + index += stride) { + const int64_t row = index / cols; + const int64_t col = index - row * cols; + const float value = modulated_bf16_value(input, shift, scale, row_index, row, col, cols); + local_bits = max(local_bits, __float_as_uint(value) >> 16 & 0x7fffu); + } + const uint32_t block_bits = block_reduce_max_u32(local_bits); + if (threadIdx.x == 0) { + partials[blockIdx.x] = bf16_abs_bits_to_float(block_bits); + } +} + +__global__ void final_scale_bf16_compat_kernel( + const float* __restrict__ partials, + float* __restrict__ output, + int64_t count, + float divisor) { + float local_max = 0.0f; + for (int64_t index = threadIdx.x; index < count; index += blockDim.x) { + local_max = fmaxf(local_max, partials[index]); + } + const float block_max = block_reduce_max(local_max); + if (threadIdx.x == 0) { + output[0] = round_bf16(block_max / divisor); + } +} + +template +__global__ void quantize_nvfp4_modulated_bf16_kernel( + const uint16_t* __restrict__ input, + const table_t* __restrict__ shift, + const table_t* __restrict__ adaln_scale, + const int32_t* __restrict__ row_index, + const float* __restrict__ tensor_scale_ptr, + uint8_t* __restrict__ qdata, + uint8_t* __restrict__ block_scale, + int64_t rows, + int64_t cols, + int64_t q_rows, + int64_t q_cols, + int64_t scale_rows, + int64_t scale_cols) { + const int64_t row = blockIdx.x; + const float tensor_scale = tensor_scale_ptr[0]; + for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { + float local_max = 0.0f; + float values[16]; + #pragma unroll + for (int i = 0; i < 16; ++i) { + const int64_t col = block_col * 16 + i; + float value = 0.0f; + if (row < rows && col < cols) { + value = modulated_bf16_value(input, shift, adaln_scale, row_index, row, col, cols); + } + values[i] = value; + local_max = fmaxf(local_max, fabsf(value)); + } + const float raw_block_scale = (local_max / 6.0f) / tensor_scale; + const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); + const float block_scale_value = decode_fp8_e4m3(scale_byte); + const int64_t row_in_tile = row % 128; + const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); + const int64_t within_tile = + ((row_in_tile % 32) / 2) * 32 + + (block_col % 4) + + (row_in_tile / 32) * 4 + + (row_in_tile % 2) * 16; + block_scale[tile * 512 + within_tile] = scale_byte; + if (row < q_rows && block_col * 8 < q_cols) { + const float encode_scale = fminf( + reciprocal_approx_ftz(tensor_scale * block_scale_value), + 3.402823466e+38f); + #pragma unroll + for (int pair = 0; pair < 8; ++pair) { + const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); + const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); + qdata[row * q_cols + block_col * 8 + pair] = static_cast((even << 4) | odd); + } + } + } +} + +__global__ void partial_absmax_swiglu_bf16_kernel( + const uint16_t* __restrict__ input, + float* __restrict__ partials, + int64_t rows, + int64_t output_cols) { + const int64_t numel = rows * output_cols; + const int64_t stride = static_cast(blockDim.x) * gridDim.x; + uint32_t local_bits = 0; + for (int64_t index = static_cast(blockIdx.x) * blockDim.x + threadIdx.x; + index < numel; + index += stride) { + const int64_t row = index / output_cols; + const int64_t col = index - row * output_cols; + const float value = swiglu_bf16_value(input, row, col, output_cols); + local_bits = max(local_bits, __float_as_uint(value) >> 16 & 0x7fffu); + } + const uint32_t block_bits = block_reduce_max_u32(local_bits); + if (threadIdx.x == 0) { + partials[blockIdx.x] = bf16_abs_bits_to_float(block_bits); + } +} + +__global__ void quantize_nvfp4_swiglu_bf16_kernel( + const uint16_t* __restrict__ input, + const float* __restrict__ tensor_scale_ptr, + uint8_t* __restrict__ qdata, + uint8_t* __restrict__ block_scale, + int64_t rows, + int64_t output_cols, + int64_t q_rows, + int64_t q_cols, + int64_t scale_cols) { + const int64_t row = blockIdx.x; + const float tensor_scale = tensor_scale_ptr[0]; + for (int64_t block_col = threadIdx.x; block_col < scale_cols; block_col += blockDim.x) { + float local_max = 0.0f; + float values[16]; + #pragma unroll + for (int i = 0; i < 16; ++i) { + const int64_t col = block_col * 16 + i; + float value = 0.0f; + if (row < rows && col < output_cols) { + value = swiglu_bf16_value(input, row, col, output_cols); + } + values[i] = value; + local_max = fmaxf(local_max, fabsf(value)); + } + const float raw_block_scale = (local_max / 6.0f) / tensor_scale; + const uint8_t scale_byte = encode_fp8_e4m3(fminf(raw_block_scale, 448.0f)); + const float block_scale_value = decode_fp8_e4m3(scale_byte); + const int64_t row_in_tile = row % 128; + const int64_t tile = ((row / 128) * (scale_cols / 4)) + (block_col / 4); + const int64_t within_tile = + ((row_in_tile % 32) / 2) * 32 + + (block_col % 4) + + (row_in_tile / 32) * 4 + + (row_in_tile % 2) * 16; + block_scale[tile * 512 + within_tile] = scale_byte; + if (row < q_rows && block_col * 8 < q_cols) { + const float encode_scale = fminf( + reciprocal_approx_ftz(tensor_scale * block_scale_value), + 3.402823466e+38f); + #pragma unroll + for (int pair = 0; pair < 8; ++pair) { + const uint8_t even = encode_fp4_e2m1(values[pair * 2] * encode_scale); + const uint8_t odd = encode_fp4_e2m1(values[pair * 2 + 1] * encode_scale); qdata[row * q_cols + block_col * 8 + pair] = static_cast((even << 4) | odd); } } @@ -277,6 +496,298 @@ __global__ void qkv_to_bshd_vec16_kernel( } } +__device__ __forceinline__ float sage2_warp_sum(float value) { + #pragma unroll + for (int offset = 16; offset > 0; offset >>= 1) { + value += __shfl_down_sync(0xffffffffu, value, offset); + } + return __shfl_sync(0xffffffffu, value, 0); +} + +__device__ __forceinline__ float sage2_warp_max(float value) { + #pragma unroll + for (int mask = 16; mask > 0; mask >>= 1) { + value = fmaxf(value, __shfl_xor_sync(0xffffffffu, value, mask)); + } + return value; +} + +__device__ __forceinline__ int8_t sage2_int8_rn(float value) { + uint32_t result; + asm volatile("cvt.rni.sat.s8.f32 %0, %1;" : "=r"(result) : "f"(value)); + return static_cast(result); +} + +__device__ __forceinline__ nv_bfloat16 sage2_rotate_value( + const nv_bfloat16* __restrict__ normalized, + const nv_bfloat16* __restrict__ rotation, + int dimension, + int rotation_pairs, + int64_t rotation_pair_stride, + int64_t rotation_row_stride, + int64_t rotation_component_stride) { + if (dimension >= rotation_pairs * 2) { + return normalized[dimension]; + } + const int pair = dimension < rotation_pairs ? dimension : dimension - rotation_pairs; + const float x0 = __bfloat162float(normalized[pair]); + const float x1 = __bfloat162float(normalized[pair + rotation_pairs]); + const nv_bfloat16* pair_rotation = rotation + static_cast(pair) * rotation_pair_stride; + const int row = dimension < rotation_pairs ? 0 : 1; + const float f0 = __bfloat162float(pair_rotation[static_cast(row) * rotation_row_stride]); + const float f1 = __bfloat162float(pair_rotation[static_cast(row) * rotation_row_stride + rotation_component_stride]); + return __float2bfloat16_rn(f0 * x0 + f1 * x1); +} + +__global__ __launch_bounds__(1024) void sage2_prepare_qk_bf16_kernel( + const nv_bfloat16* __restrict__ q, + nv_bfloat16* __restrict__ k, + const nv_bfloat16* __restrict__ rotation, + const nv_bfloat16* __restrict__ q_weight, + const nv_bfloat16* __restrict__ k_weight, + int8_t* __restrict__ q_int8, + float* __restrict__ q_scale, + nv_bfloat16* __restrict__ q_prepared, + int64_t sequence, + int64_t heads, + int64_t q_s0, + int64_t q_s1, + int64_t q_s2, + int64_t k_s0, + int64_t k_s1, + int64_t k_s2, + int64_t f_s0, + int64_t f_s1, + int64_t f_s2, + int64_t f_s3, + int64_t f_s4, + int64_t f_s5, + int rotation_pairs, + float epsilon) { + constexpr int kHeadDim = 128; + constexpr int kTokensPerBlock = 32; + __shared__ nv_bfloat16 normalized[kTokensPerBlock][kHeadDim]; + __shared__ float warp_maxima[32]; + __shared__ float block_amax; + + const int lane = threadIdx.x & 31; + const int token_slot = threadIdx.x >> 5; + const int64_t token = static_cast(blockIdx.x) * kTokensPerBlock + token_slot; + const int64_t head = blockIdx.y; + const int64_t batch = blockIdx.z; + const bool valid = token < sequence; + const int64_t q_base = batch * q_s0 + token * q_s1 + head * q_s2; + const int64_t k_base = batch * k_s0 + token * k_s1 + head * k_s2; + const int64_t out_base = (batch * sequence * heads + token * heads + head) * kHeadDim; + const int64_t freq_base = batch * f_s0 + token * f_s1; + + float q_raw[4]; + float q_sum = 0.0f; + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + const float value = valid ? __bfloat162float(q[q_base + dimension]) : 0.0f; + q_raw[item] = value; + q_sum = fmaf(value, value, q_sum); + } + const float q_rrms = rsqrtf(sage2_warp_sum(q_sum) / static_cast(kHeadDim) + epsilon); + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + normalized[token_slot][dimension] = __float2bfloat16_rn( + q_raw[item] * q_rrms * __bfloat162float(q_weight[dimension])); + } + __syncthreads(); + + const nv_bfloat16* token_rotation = rotation + freq_base; + nv_bfloat16 q_values[4]; + float local_amax = 0.0000001f; + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + const nv_bfloat16 value = valid + ? sage2_rotate_value(normalized[token_slot], token_rotation, dimension, rotation_pairs, f_s3, f_s4, f_s5) + : __float2bfloat16_rn(0.0f); + q_values[item] = value; + local_amax = fmaxf(local_amax, fabsf(__bfloat162float(value))); + if (valid && q_prepared != nullptr) { + q_prepared[out_base + dimension] = value; + } + } + + const float warp_amax = sage2_warp_max(local_amax); + if (lane == 0) { + warp_maxima[token_slot] = warp_amax; + } + __syncthreads(); + float block_value = threadIdx.x < 32 ? warp_maxima[lane] : -1e20f; + block_value = sage2_warp_max(block_value); + if (threadIdx.x == 0) { + block_amax = block_value; + q_scale[(batch * heads + head) * gridDim.x + blockIdx.x] = + block_value * reciprocal_approx_ftz(127.0f); + } + __syncthreads(); + const float reciprocal_scale = 127.0f * reciprocal_approx_ftz(block_amax); + if (valid) { + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + q_int8[out_base + dimension] = sage2_int8_rn(__bfloat162float(q_values[item]) * reciprocal_scale); + } + } + __syncthreads(); + + float k_raw[4]; + float k_sum = 0.0f; + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + const float value = valid ? __bfloat162float(k[k_base + dimension]) : 0.0f; + k_raw[item] = value; + k_sum = fmaf(value, value, k_sum); + } + const float k_rrms = rsqrtf(sage2_warp_sum(k_sum) / static_cast(kHeadDim) + epsilon); + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + normalized[token_slot][dimension] = __float2bfloat16_rn( + k_raw[item] * k_rrms * __bfloat162float(k_weight[dimension])); + } + __syncthreads(); + if (valid) { + #pragma unroll + for (int item = 0; item < 4; ++item) { + const int dimension = lane + item * 32; + k[k_base + dimension] = sage2_rotate_value( + normalized[token_slot], token_rotation, dimension, rotation_pairs, f_s3, f_s4, f_s5); + } + } +} + +__global__ void sage2_v_partial_absmax_bf16_kernel( + const nv_bfloat16* __restrict__ v, + float* __restrict__ partials, + int64_t sequence, + int64_t heads, + int64_t tiles, + int64_t v_s0, + int64_t v_s1, + int64_t v_s2) { + constexpr int kHeadDim = 128; + constexpr int kTileRows = 64; + const int dimension = threadIdx.x; + const int64_t tile = blockIdx.x; + const int64_t head = blockIdx.y; + const int64_t batch = blockIdx.z; + const int64_t first_token = tile * kTileRows; + float local_max = 0.0f; + + if (dimension < kHeadDim) { + #pragma unroll + for (int row = 0; row < kTileRows; ++row) { + const int64_t token = first_token + row; + if (token < sequence) { + const int64_t offset = batch * v_s0 + token * v_s1 + head * v_s2 + dimension; + local_max = fmaxf(local_max, fabsf(__bfloat162float(v[offset]))); + } + } + partials[((batch * heads + head) * kHeadDim + dimension) * tiles + tile] = local_max; + } +} + +__global__ void sage2_v_finalize_scale_kernel( + const float* __restrict__ partials, + float* __restrict__ scale, + float* __restrict__ reciprocal_scale, + int64_t heads, + int64_t tiles, + float scale_max) { + constexpr int kHeadDim = 128; + const int64_t head = blockIdx.x; + const int64_t batch = blockIdx.y; + const int64_t dimension = blockIdx.z; + const int64_t partial_base = ((batch * heads + head) * kHeadDim + dimension) * tiles; + float local_max = 0.0f; + for (int64_t tile = threadIdx.x; tile < tiles; tile += blockDim.x) { + local_max = fmaxf(local_max, partials[partial_base + tile]); + } + const float channel_max = block_reduce_max(local_max); + if (threadIdx.x == 0) { + scale[(batch * heads + head) * kHeadDim + dimension] = + channel_max * reciprocal_approx_ftz(scale_max); + reciprocal_scale[(batch * heads + head) * kHeadDim + dimension] = + channel_max == 0.0f ? 0.0f : scale_max * reciprocal_approx_ftz(channel_max); + } +} + +__device__ __forceinline__ uint32_t sage2_e4m3x4(float x0, float x1, float x2, float x3) { + uint32_t packed; + asm volatile( + "{\n" + ".reg .b16 lo;\n" + ".reg .b16 hi;\n" + "cvt.rn.satfinite.e4m3x2.f32 lo, %2, %1;\n" + "cvt.rn.satfinite.e4m3x2.f32 hi, %4, %3;\n" + "mov.b32 %0, {lo, hi};\n" + "}\n" + : "=r"(packed) : "f"(x0), "f"(x1), "f"(x2), "f"(x3)); + return packed; +} + +__global__ __launch_bounds__(1024) void sage2_v_quantize_permute_bf16_kernel( + const nv_bfloat16* __restrict__ v, + const float* __restrict__ reciprocal_scale, + uint8_t* __restrict__ output, + int64_t sequence, + int64_t heads, + int64_t padded_sequence, + int64_t v_s0, + int64_t v_s1, + int64_t v_s2, + float scale_max) { + constexpr int kHeadDim = 128; + constexpr int kTileRows = 64; + __shared__ nv_bfloat16 tile[kTileRows][kHeadDim]; + + const int lane_group = threadIdx.x & 15; + const int source_row = threadIdx.x >> 4; + const int dimension_base = lane_group * 8; + const int64_t tile_index = blockIdx.x; + const int64_t head = blockIdx.y; + const int64_t batch = blockIdx.z; + const int64_t token = tile_index * kTileRows + source_row; + const int row_mod = source_row & 15; + const int permuted_row = (source_row & ~15) + (row_mod / 8) * 2 + ((row_mod / 2) & 3) * 4 + (row_mod & 1); + + #pragma unroll + for (int item = 0; item < 8; ++item) { + const int dimension = dimension_base + item; + tile[permuted_row][dimension] = token < sequence + ? v[batch * v_s0 + token * v_s1 + head * v_s2 + dimension] + : __float2bfloat16_rn(0.0f); + } + __syncthreads(); + + const int output_dimension = threadIdx.x >> 3; + const int output_group = threadIdx.x & 7; + const int output_row = output_group * 8; + const float channel_reciprocal_scale = + reciprocal_scale[(batch * heads + head) * kHeadDim + output_dimension]; + float values[8]; + #pragma unroll + for (int item = 0; item < 8; ++item) { + values[item] = __bfloat162float(tile[output_row + item][output_dimension]) * channel_reciprocal_scale; + } + const uint2 packed = make_uint2( + sage2_e4m3x4(values[0], values[1], values[2], values[3]), + sage2_e4m3x4(values[4], values[5], values[6], values[7])); + const int64_t output_base = + ((batch * kHeadDim + output_dimension) * heads + head) * padded_sequence + + tile_index * kTileRows + output_row; + *reinterpret_cast(output + output_base) = packed; +} + } // namespace torch::Tensor nvfp4_activation_scale_cuda(torch::Tensor input, double divisor) { @@ -327,6 +838,8 @@ torch::Tensor nvfp4_activation_scale_into_cuda(torch::Tensor input, double divis return output; } +void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first); + std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch::Tensor scale, bool pad_16x, int64_t threads) { c10::cuda::CUDAGuard device_guard(input.device()); const int64_t rows = input.size(0); @@ -336,8 +849,21 @@ std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch:: const int64_t q_cols = pad_16x ? roundup(cols, 16) / 2 : cols / 2; const int64_t scale_rows = roundup(q_rows, 128); const int64_t scale_cols = roundup(q_cols / 8, 4); + TORCH_CHECK(scale_cols % 4 == 0, "NVFP4 block-scale width must be divisible by four"); auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); + quantize_nvfp4_bf16_into_cuda(input, scale, qdata, block_scale, threads, true); + return {qdata, block_scale}; +} + +void quantize_nvfp4_bf16_into_cuda(torch::Tensor input, torch::Tensor scale, torch::Tensor qdata, torch::Tensor block_scale, int64_t threads, bool hi_first) { + c10::cuda::CUDAGuard device_guard(input.device()); + const int64_t rows = input.size(0); + const int64_t cols = input.size(1); + const int64_t q_rows = qdata.size(0); + const int64_t q_cols = qdata.size(1); + const int64_t scale_rows = block_scale.size(0); + const int64_t scale_cols = block_scale.size(1); auto stream = at::cuda::getCurrentCUDAStream(); quantize_nvfp4_bf16_kernel<<(threads), 0, stream>>>( reinterpret_cast(input.data_ptr()), @@ -349,11 +875,110 @@ std::vector quantize_nvfp4_bf16_cuda(torch::Tensor input, torch:: q_rows, q_cols, scale_rows, + scale_cols, + hi_first); + C10_CUDA_KERNEL_LAUNCH_CHECK(); +} + +std::vector quantize_nvfp4_modulated_bf16_cuda( + torch::Tensor input, + torch::Tensor shift, + torch::Tensor adaln_scale, + torch::Tensor row_index, + double divisor, + bool pad_16x, + int64_t blocks, + int64_t scale_threads, + int64_t pack_threads) { + c10::cuda::CUDAGuard device_guard(input.device()); + const int64_t rows = input.size(0); + const int64_t cols = input.size(1); + const int64_t q_rows = pad_16x ? roundup(rows, 16) : rows; + const int64_t q_cols = pad_16x ? roundup(cols, 16) / 2 : cols / 2; + const int64_t scale_rows = roundup(q_rows, 128); + const int64_t scale_cols = roundup(q_cols / 8, 4); + auto partials = torch::empty({blocks}, input.options().dtype(torch::kFloat32)); + auto tensor_scale = torch::empty({}, input.options().dtype(torch::kFloat32)); + auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); + auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); + auto stream = at::cuda::getCurrentCUDAStream(); + + AT_DISPATCH_FLOATING_TYPES_AND(at::ScalarType::BFloat16, shift.scalar_type(), "vortex_nvfp4_modulated_bf16", [&] { + partial_absmax_modulated_bf16_kernel<<(blocks), static_cast(scale_threads), 0, stream>>>( + reinterpret_cast(input.data_ptr()), + shift.data_ptr(), + adaln_scale.data_ptr(), + row_index.data_ptr(), + partials.data_ptr(), + rows, + cols); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + final_scale_bf16_compat_kernel<<<1, static_cast(scale_threads), 0, stream>>>( + partials.data_ptr(), tensor_scale.data_ptr(), blocks, static_cast(divisor)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + quantize_nvfp4_modulated_bf16_kernel<<(scale_rows), static_cast(pack_threads), 0, stream>>>( + reinterpret_cast(input.data_ptr()), + shift.data_ptr(), + adaln_scale.data_ptr(), + row_index.data_ptr(), + tensor_scale.data_ptr(), + qdata.data_ptr(), + reinterpret_cast(block_scale.data_ptr()), + rows, + cols, + q_rows, + q_cols, + scale_rows, + scale_cols); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + }); + return {tensor_scale, qdata, block_scale}; +} + +std::vector quantize_nvfp4_swiglu_bf16_cuda( + torch::Tensor input, + double divisor, + bool pad_16x, + int64_t blocks, + int64_t scale_threads, + int64_t pack_threads) { + c10::cuda::CUDAGuard device_guard(input.device()); + const int64_t rows = input.size(0); + const int64_t output_cols = input.size(1) / 2; + const int64_t q_rows = pad_16x ? roundup(rows, 16) : rows; + const int64_t q_cols = pad_16x ? roundup(output_cols, 16) / 2 : output_cols / 2; + const int64_t scale_rows = roundup(q_rows, 128); + const int64_t scale_cols = roundup(q_cols / 8, 4); + auto partials = torch::empty({blocks}, input.options().dtype(torch::kFloat32)); + auto tensor_scale = torch::empty({}, input.options().dtype(torch::kFloat32)); + auto qdata = torch::empty({q_rows, q_cols}, input.options().dtype(torch::kUInt8)); + auto block_scale = torch::zeros({scale_rows, scale_cols}, input.options().dtype(torch::kUInt8)).view(torch::kFloat8_e4m3fn); + auto stream = at::cuda::getCurrentCUDAStream(); + + partial_absmax_swiglu_bf16_kernel<<(blocks), static_cast(scale_threads), 0, stream>>>( + reinterpret_cast(input.data_ptr()), + partials.data_ptr(), + rows, + output_cols); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + final_scale_bf16_compat_kernel<<<1, static_cast(scale_threads), 0, stream>>>( + partials.data_ptr(), tensor_scale.data_ptr(), blocks, static_cast(divisor)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + quantize_nvfp4_swiglu_bf16_kernel<<(scale_rows), static_cast(pack_threads), 0, stream>>>( + reinterpret_cast(input.data_ptr()), + tensor_scale.data_ptr(), + qdata.data_ptr(), + reinterpret_cast(block_scale.data_ptr()), + rows, + output_cols, + q_rows, + q_cols, scale_cols); C10_CUDA_KERNEL_LAUNCH_CHECK(); - return {qdata, block_scale}; + return {tensor_scale, qdata, block_scale}; } + std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, int64_t head_dim) { c10::cuda::CUDAGuard device_guard(qkv.device()); const int64_t sequence = qkv.size(0); @@ -396,3 +1021,97 @@ std::vector qkv_to_bshd_cuda(torch::Tensor qkv, int64_t heads, in C10_CUDA_KERNEL_LAUNCH_CHECK(); return {q, k, v}; } + +std::vector sage2_prepare_qk_cuda( + torch::Tensor q, + torch::Tensor k, + torch::Tensor rotation, + torch::Tensor q_weight, + torch::Tensor k_weight, + double epsilon, + bool materialize_q) { + c10::cuda::CUDAGuard device_guard(q.device()); + const int64_t batch = q.size(0); + const int64_t sequence = q.size(1); + const int64_t heads = q.size(2); + const int64_t groups = ((sequence + 127) / 128) * 4; + auto q_int8 = torch::empty(q.sizes(), q.options().dtype(torch::kInt8)); + auto q_scale = torch::empty({batch, heads, groups}, q.options().dtype(torch::kFloat32)); + auto q_prepared = materialize_q + ? torch::empty(q.sizes(), q.options()) + : torch::empty({0}, q.options()); + auto stream = at::cuda::getCurrentCUDAStream(); + const dim3 grid(static_cast(groups), static_cast(heads), static_cast(batch)); + sage2_prepare_qk_bf16_kernel<<>>( + reinterpret_cast(q.data_ptr()), + reinterpret_cast(k.data_ptr()), + reinterpret_cast(rotation.data_ptr()), + reinterpret_cast(q_weight.data_ptr()), + reinterpret_cast(k_weight.data_ptr()), + q_int8.data_ptr(), + q_scale.data_ptr(), + materialize_q ? reinterpret_cast(q_prepared.data_ptr()) : nullptr, + sequence, + heads, + q.stride(0), q.stride(1), q.stride(2), + k.stride(0), k.stride(1), k.stride(2), + rotation.stride(0), rotation.stride(1), rotation.stride(2), + rotation.stride(3), rotation.stride(4), rotation.stride(5), + static_cast(rotation.size(3)), + static_cast(epsilon)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return {q_int8, q_scale, q_prepared}; +} + +std::vector sage2_prepare_v_cuda(torch::Tensor v, double scale_max) { + c10::cuda::CUDAGuard device_guard(v.device()); + constexpr int64_t kHeadDim = 128; + constexpr int64_t kTileRows = 64; + const int64_t batch = v.size(0); + const int64_t sequence = v.size(1); + const int64_t heads = v.size(2); + const int64_t tiles = (sequence + kTileRows - 1) / kTileRows; + const int64_t padded_sequence = tiles * kTileRows; + auto partials = torch::empty({batch, heads, kHeadDim, tiles}, v.options().dtype(torch::kFloat32)); + auto v_scale = torch::empty({batch, heads, kHeadDim}, v.options().dtype(torch::kFloat32)); + auto reciprocal_scale = torch::empty_like(v_scale); + auto v_fp8 = torch::empty( + {batch, kHeadDim, heads, padded_sequence}, + v.options().dtype(torch::kFloat8_e4m3fn)); + auto stream = at::cuda::getCurrentCUDAStream(); + const dim3 tile_grid( + static_cast(tiles), + static_cast(heads), + static_cast(batch)); + sage2_v_partial_absmax_bf16_kernel<<>>( + reinterpret_cast(v.data_ptr()), + partials.data_ptr(), + sequence, + heads, + tiles, + v.stride(0), v.stride(1), v.stride(2)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + const dim3 scale_grid( + static_cast(heads), + static_cast(batch), + static_cast(kHeadDim)); + sage2_v_finalize_scale_kernel<<>>( + partials.data_ptr(), + v_scale.data_ptr(), + reciprocal_scale.data_ptr(), + heads, + tiles, + static_cast(scale_max)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + sage2_v_quantize_permute_bf16_kernel<<>>( + reinterpret_cast(v.data_ptr()), + reciprocal_scale.data_ptr(), + reinterpret_cast(v_fp8.data_ptr()), + sequence, + heads, + padded_sequence, + v.stride(0), v.stride(1), v.stride(2), + static_cast(scale_max)); + C10_CUDA_KERNEL_LAUNCH_CHECK(); + return {v_fp8, v_scale}; +} diff --git a/src/h3_blackwell_runtime/cute_qkv_ring.py b/src/h3_blackwell_runtime/cute_qkv_ring.py new file mode 100644 index 0000000..5108a4d --- /dev/null +++ b/src/h3_blackwell_runtime/cute_qkv_ring.py @@ -0,0 +1,282 @@ +"""Opt-in CuTe bounded-ring backend for full-width H3 QKV projections.""" + +from __future__ import annotations + +import importlib.util +import os +import sys +import threading +import weakref +from dataclasses import dataclass +from pathlib import Path + +import torch + +from .nvfp4_quant import nvfp4_activation_scale, vortex_native_quantize_nvfp4_into + + +def _enabled(name: str) -> bool: + return os.getenv(name, "").lower() in {"1", "true", "yes", "on"} + + +def _output_tensor(storage: torch.Tensor): + from cutlass.cute.runtime import from_dlpack + + tensor = from_dlpack(storage.unsqueeze(-1), assumed_align=16) + return tensor.mark_compact_shape_dynamic( + mode=1, stride_order=(2, 0, 1), divisibility=1, + ) + + +def _scale_tensor(storage: torch.Tensor): + import cutlass + from cutlass.cute.runtime import from_dlpack + + tensor = from_dlpack(storage.view(torch.uint8).unsqueeze(-1), assumed_align=16) + tensor.element_type = cutlass.Float8E4M3FN + return tensor.mark_layout_dynamic(leading_dim=1) + + +def _weight_tensor(storage: torch.Tensor): + import cutlass + import cutlass.torch as cutlass_torch + + lookup = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, + -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], + device=storage.device, + dtype=torch.float32, + ) + codes = torch.stack((storage >> 4, storage & 0x0F), dim=-1).reshape( + storage.shape[0], -1, + ) + logical = lookup[codes.long()].unsqueeze(-1) + tensor, backing = cutlass_torch.cute_tensor_like( + logical, + cutlass.Float4E2M1FN, + is_dynamic_layout=True, + assumed_align=16, + ) + return tensor, backing + + +def _load_kernel(path: Path): + if not path.is_file(): + raise FileNotFoundError(f"CuTe QKV kernel not found: {path}") + sys.path.insert(0, str(path.parent)) + spec = importlib.util.spec_from_file_location("h3_cute_qkv_ring_kernel", path) + if spec is None or spec.loader is None: + raise ImportError(f"Cannot load CuTe QKV kernel: {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +@dataclass +class _PreparedWeight: + b: object + backing: torch.Tensor + sfb: object + + +@dataclass +class _Workspace: + a: object + a_backing: torch.Tensor + qdata: torch.Tensor + block_scale: torch.Tensor + sfa: object + alpha: torch.Tensor + alpha_argument: object + outputs: dict[int, tuple[torch.Tensor, list[object]]] + + +class _QkvRingBackend: + def __init__(self) -> None: + self.capacity = int(os.getenv("H3_CUTE_QKV_RING_CAPACITY", "2048")) + if self.capacity <= 0 or self.capacity % 128: + raise ValueError("H3_CUTE_QKV_RING_CAPACITY must be a positive multiple of 128") + default_path = Path("/opt/h3-blackwell-runtime/tools/dense_blockscaled_gemm_persistent_cooperative_vortex_alpha.py") + self.kernel_path = Path(os.getenv("H3_CUTE_QKV_KERNEL", str(default_path))) + self.strict = _enabled("H3_CUTE_QKV_RING_STRICT") + self._lock = threading.Lock() + self._module = None + self._workspace: dict[int, _Workspace] = {} + self._weights: weakref.WeakKeyDictionary = weakref.WeakKeyDictionary() + self._compiled: dict[int, tuple[object, object]] = {} + self.disabled_reason: str | None = None + + def _ineligible(self, message: str): + if self.strict: + raise RuntimeError(message) + return None + + def _eligible(self, linear, x: torch.Tensor) -> str | None: + if linear.role != "h3_attn_qkv": + return "projection role is not H3 QKV" + if linear.in_features != 5376 or linear.out_features != 21504: + return "QKV projection is sharded or has an unsupported shape" + if linear.full_precision_matrix_mult or linear.pre_quant_scale is not None: + return "QKV projection uses an unsupported quantization policy" + if linear.bias is not None or linear.output_dtype != torch.bfloat16: + return "QKV bias/output dtype is unsupported" + if not x.is_cuda or x.dtype != torch.bfloat16 or x.dim() != 2 or not x.is_contiguous(): + return "QKV activation must be contiguous 2D CUDA BF16" + if x.requires_grad or torch.is_grad_enabled(): + return "QKV ring is inference-only" + major, _ = torch.cuda.get_device_capability(x.device) + if major != 12: + return "QKV ring is currently validated only on SM12x" + return None + + def _prepare_workspace(self, device: torch.device) -> _Workspace: + import cutlass + import cutlass.torch as cutlass_torch + from cutlass.cute.runtime import from_dlpack + + index = device.index or 0 + workspace = self._workspace.get(index) + if workspace is not None: + return workspace + a, a_backing = cutlass_torch.cute_tensor_like( + torch.zeros( + self.capacity, 5376, 1, device=device, dtype=torch.float32, + ), + cutlass.Float4E2M1FN, + is_dynamic_layout=True, + assumed_align=16, + ) + qdata = a_backing.view(torch.uint8).flatten()[ + : self.capacity * 5376 // 2 + ].reshape(self.capacity, 5376 // 2) + block_scale = torch.empty( + self.capacity, 5376 // 16, device=device, dtype=torch.float8_e4m3fn, + ) + alpha = torch.empty(1, device=device, dtype=torch.float32) + workspace = _Workspace( + a=a, + a_backing=a_backing, + qdata=qdata, + block_scale=block_scale, + sfa=_scale_tensor(block_scale), + alpha=alpha, + alpha_argument=from_dlpack(alpha, assumed_align=4), + outputs={}, + ) + self._workspace[index] = workspace + return workspace + + def _prepare_weight(self, linear) -> _PreparedWeight: + prepared = self._weights.get(linear) + if prepared is not None: + return prepared + b, backing = _weight_tensor(linear.weight) + prepared = _PreparedWeight( + b=b, + backing=backing, + sfb=_scale_tensor(linear.weight_scale), + ) + self._weights[linear] = prepared + return prepared + + def _output(self, workspace: _Workspace, rows: int): + padded_rows = ((rows + self.capacity - 1) // self.capacity) * self.capacity + cached = workspace.outputs.get(padded_rows) + if cached is not None: + return cached + output = torch.empty( + padded_rows, + 21504, + device=workspace.qdata.device, + dtype=torch.bfloat16, + ) + chunks = [ + _output_tensor(output[start : start + self.capacity]) + for start in range(0, padded_rows, self.capacity) + ] + cached = (output, chunks) + workspace.outputs[padded_rows] = cached + return cached + + def _compile(self, device: torch.device, workspace: _Workspace, weight: _PreparedWeight, c): + import cutlass + import cutlass.cute as cute + import cutlass.torch as cutlass_torch + + index = device.index or 0 + cached = self._compiled.get(index) + if cached is not None: + return cached + if self._module is None: + self._module = _load_kernel(self.kernel_path) + gemm = self._module.Sm120BlockScaledGemmKernel( + cutlass.Float32, 16, (128, 128, 128), (128, 128), + ) + stream = cutlass_torch.default_stream() + max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) + compiled = cute.compile( + gemm, + workspace.a, + weight.b, + workspace.sfa, + weight.sfb, + c, + workspace.alpha_argument, + max_active_clusters, + stream, + ) + cached = (compiled, stream) + self._compiled[index] = cached + return cached + + def __call__(self, linear, x: torch.Tensor): + reason = self._eligible(linear, x) + if reason is not None: + return self._ineligible(reason) + try: + with self._lock: + workspace = self._prepare_workspace(x.device) + weight = self._prepare_weight(linear) + output, c_chunks = self._output(workspace, x.shape[0]) + compiled, stream = self._compile( + x.device, workspace, weight, c_chunks[0], + ) + scale = nvfp4_activation_scale(x).float() + workspace.alpha.copy_(scale * linear.weight_scale_2.float()) + for index, start in enumerate(range(0, x.shape[0], self.capacity)): + stop = min(start + self.capacity, x.shape[0]) + vortex_native_quantize_nvfp4_into( + x[start:stop], + scale, + workspace.qdata, + workspace.block_scale, + hi_first=False, + ) + compiled( + workspace.a, + weight.b, + workspace.sfa, + weight.sfb, + c_chunks[index], + workspace.alpha_argument, + stream, + ) + return output[: x.shape[0], : linear.out_features] + except (ImportError, FileNotFoundError, RuntimeError) as error: + self.disabled_reason = str(error) + return self._ineligible(f"QKV ring initialization failed: {error}") + + +_BACKEND: _QkvRingBackend | None = None + + +def qkv_ring_linear(linear, x: torch.Tensor): + global _BACKEND + if _BACKEND is None: + try: + _BACKEND = _QkvRingBackend() + except (ImportError, ValueError) as error: + if _enabled("H3_CUTE_QKV_RING_STRICT"): + raise + return None + return _BACKEND(linear, x) diff --git a/src/h3_blackwell_runtime/h3_fusion.py b/src/h3_blackwell_runtime/h3_fusion.py new file mode 100644 index 0000000..0ee54db --- /dev/null +++ b/src/h3_blackwell_runtime/h3_fusion.py @@ -0,0 +1,206 @@ +"""Triton fusion for H3's segmented BF16 modulation and residual gates.""" + +from __future__ import annotations + +import torch +import triton +import triton.language as tl + + +_TILE = 1024 +_segment_cache_key = None +_segment_cache_value = None + + +@triton.jit +def _round_bf16_fp32(value): + """Apply round-to-nearest-even BF16 precision while retaining FP32.""" + bits = value.to(tl.int32, bitcast=True) + bits = bits + 0x7FFF + ((bits >> 16) & 1) + return (bits & -65536).to(tl.float32, bitcast=True) + + +@triton.jit +def _modulate_kernel( + x_ptr, + scale_ptr, + shift_ptr, + row_index_ptr, + tokens, + width, + sx_t, + sx_d, + ss_t, + ss_d, + sh_t, + sh_d, + BLOCK: tl.constexpr, +): + row = tl.program_id(0) + columns = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) + valid = (row < tokens) & (columns < width) + table_row = tl.load(row_index_ptr + row) + x = tl.load(x_ptr + row * sx_t + columns * sx_d, mask=valid).to(tl.float32) + scale = tl.load( + scale_ptr + table_row * ss_t + columns * ss_d, mask=valid, + ).to(tl.float32) + shift = tl.load( + shift_ptr + table_row * sh_t + columns * sh_d, mask=valid, + ).to(tl.float32) + + scale = _round_bf16_fp32(scale) + shift = _round_bf16_fp32(shift) + multiplied = _round_bf16_fp32(x * _round_bf16_fp32(1.0 + scale)) + result = _round_bf16_fp32(multiplied + shift) + tl.store(x_ptr + row * sx_t + columns * sx_d, result.to(tl.bfloat16), mask=valid) + + +@triton.jit +def _gate_add_kernel( + residual_ptr, + gate_ptr, + update_ptr, + row_index_ptr, + tokens, + width, + sr_t, + sr_d, + sg_t, + sg_d, + su_t, + su_d, + BLOCK: tl.constexpr, +): + row = tl.program_id(0) + columns = tl.program_id(1) * BLOCK + tl.arange(0, BLOCK) + valid = (row < tokens) & (columns < width) + table_row = tl.load(row_index_ptr + row) + residual = tl.load( + residual_ptr + row * sr_t + columns * sr_d, mask=valid, + ).to(tl.float32) + update = tl.load( + update_ptr + row * su_t + columns * su_d, mask=valid, + ).to(tl.float32) + gate = tl.load( + gate_ptr + table_row * sg_t + columns * sg_d, mask=valid, + ).to(tl.float32) + + result = residual + update * _round_bf16_fp32(gate) + tl.store( + residual_ptr + row * sr_t + columns * sr_d, + result.to(tl.bfloat16), + mask=valid, + ) + + +def segment_index( + tokens: int, + segments: list[tuple[int, int, int]], + device: torch.device, +) -> torch.Tensor: + """Return one cached row-to-modulation-table lookup for a packed layout.""" + global _segment_cache_key, _segment_cache_value + normalized = tuple((int(start), int(stop), int(row)) for start, stop, row in segments) + key = (str(device), int(tokens), normalized) + if key == _segment_cache_key: + return _segment_cache_value + host = torch.empty(tokens, dtype=torch.int32) + cursor = 0 + for start, stop, row in normalized: + if start != cursor or stop < start or stop > tokens or row < 0: + raise ValueError("segments must be ordered, contiguous, and in range") + host[start:stop] = row + cursor = stop + if cursor != tokens: + raise ValueError("segments must cover every packed token") + _segment_cache_key = key + _segment_cache_value = host.to(device=device) + return _segment_cache_value + + +def _validate_activation(tensor: torch.Tensor, name: str) -> None: + if tensor.ndim != 2: + raise ValueError(f"{name} must have shape [tokens, hidden]") + if tensor.device.type != "cuda" or tensor.dtype != torch.bfloat16: + raise TypeError(f"{name} must be a CUDA BF16 tensor") + if tensor.stride(1) != 1: + raise ValueError(f"{name}'s hidden dimension must be contiguous") + + +def _validate_table(table: torch.Tensor, activation: torch.Tensor, name: str) -> None: + if table.ndim != 2 or table.shape[1] != activation.shape[1]: + raise ValueError(f"{name} must have shape [rows, {activation.shape[1]}]") + if table.device != activation.device or table.dtype not in {torch.bfloat16, torch.float32}: + raise TypeError(f"{name} must be CUDA BF16/FP32 on the activation device") + if table.stride(1) != 1: + raise ValueError(f"{name}'s hidden dimension must be contiguous") + + +def fused_modulate_( + activation: torch.Tensor, + shift: torch.Tensor, + scale: torch.Tensor, + row_index: torch.Tensor, +) -> torch.Tensor: + """Apply segmented scale/shift in place with eager-equivalent BF16 rounding.""" + _validate_activation(activation, "activation") + _validate_table(shift, activation, "shift") + _validate_table(scale, activation, "scale") + if row_index.shape != (activation.shape[0],) or row_index.dtype != torch.int32: + raise TypeError("row_index must be int32 with one entry per token") + if row_index.device != activation.device or not row_index.is_contiguous(): + raise TypeError("row_index must be contiguous on the activation device") + grid = (activation.shape[0], triton.cdiv(activation.shape[1], _TILE)) + _modulate_kernel[grid]( + activation, + scale, + shift, + row_index, + activation.shape[0], + activation.shape[1], + activation.stride(0), + activation.stride(1), + scale.stride(0), + scale.stride(1), + shift.stride(0), + shift.stride(1), + BLOCK=_TILE, + num_warps=4, + ) + return activation + + +def fused_gate_add_( + residual: torch.Tensor, + update: torch.Tensor, + gate: torch.Tensor, + row_index: torch.Tensor, +) -> torch.Tensor: + """Apply the segmented residual gate in place with addcmul-equivalent math.""" + _validate_activation(residual, "residual") + _validate_activation(update, "update") + if residual.shape != update.shape or residual.device != update.device: + raise ValueError("residual and update must share shape and device") + _validate_table(gate, residual, "gate") + if row_index.shape != (residual.shape[0],) or row_index.dtype != torch.int32: + raise TypeError("row_index must be int32 with one entry per token") + if row_index.device != residual.device or not row_index.is_contiguous(): + raise TypeError("row_index must be contiguous on the residual device") + grid = (residual.shape[0], triton.cdiv(residual.shape[1], _TILE)) + _gate_add_kernel[grid]( + residual, + gate, + update, + row_index, + residual.shape[0], + residual.shape[1], + residual.stride(0), + residual.stride(1), + gate.stride(0), + gate.stride(1), + update.stride(0), + update.stride(1), + BLOCK=_TILE, + num_warps=4, + ) + return residual diff --git a/src/h3_blackwell_runtime/nvfp4.py b/src/h3_blackwell_runtime/nvfp4.py index a479c87..2c2d0ef 100644 --- a/src/h3_blackwell_runtime/nvfp4.py +++ b/src/h3_blackwell_runtime/nvfp4.py @@ -1,6 +1,7 @@ """Standalone Blackwell NVFP4 linear adapter for Comfy-format checkpoints.""" import json +import os from dataclasses import dataclass import torch @@ -33,7 +34,7 @@ def parse_quant_sidecar(sidecar: torch.Tensor) -> dict: class Nvfp4Linear(DynamicLoraMixin, nn.Module): """Execute a packed Comfy NVFP4 linear with Comfy Kitchen's CUDA 13 kernel.""" - def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16): + def __init__(self, tensors: Nvfp4LinearTensors, output_dtype=torch.bfloat16, *, role: str = "other"): super().__init__() if tensors.weight.dtype != torch.uint8 or tensors.weight.ndim != 2: raise ValueError("NVFP4 weights must be a rank-2 packed uint8 tensor.") @@ -45,6 +46,7 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module): self.in_features = tensors.in_features self.out_features = tensors.out_features self.output_dtype = output_dtype + self.role = role self.full_precision_matrix_mult = tensors.full_precision_matrix_mult self.register_buffer("weight", tensors.weight.contiguous(), persistent=False) self.register_buffer("weight_scale", tensors.weight_scale.view(torch.float8_e4m3fn).contiguous(), persistent=False) @@ -88,13 +90,91 @@ class Nvfp4Linear(DynamicLoraMixin, nn.Module): else: if x.dtype == torch.float32: raise ValueError("Quantized NVFP4 activation GEMM requires FP16 or BF16 activations.") - packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout") - output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] + ring_output = None + if self.role == "h3_attn_qkv" and os.getenv("H3_CUTE_QKV_RING", "").lower() in { + "1", "true", "yes", "on", + }: + from .cute_qkv_ring import qkv_ring_linear + + ring_output = qkv_ring_linear(self, flat_x) + if ring_output is not None: + output = ring_output + elif os.getenv("H3_NVFP4_SCALE_BACKEND", "torch").lower() == "torch": + packed_x = QuantizedTensor.from_float(flat_x, "TensorCoreNVFP4Layout") + output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] + else: + from .nvfp4_quant import vortex_quantize_nvfp4 + + packed_x = vortex_quantize_nvfp4(flat_x) + output = functional.linear(packed_x, packed_weight, bias)[:flat_x.shape[0], :self.out_features] base = output.reshape(*original_shape, self.out_features) return self._apply_lora(original_x, base) + def forward_modulated( + self, + x: torch.Tensor, + shift: torch.Tensor, + scale: torch.Tensor, + row_index: torch.Tensor, + ) -> torch.Tensor: + """Fuse exact H3 modulation into activation packing for QKV or FC1.""" + if self.role not in {"h3_attn_qkv", "h3_mlp_fc1"}: + raise ValueError("modulated NVFP4 dispatch is restricted to H3 QKV and FC1") + if ( + self.full_precision_matrix_mult + or self.pre_quant_scale is not None + or self.active_lora is not None + or self.lora_strength != 0.0 + or torch.is_grad_enabled() + or x.requires_grad + ): + from .h3_fusion import fused_modulate_ -def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16) -> Nvfp4Linear: + return self(fused_modulate_(x, shift, scale, row_index)) + if x.shape != (row_index.numel(), self.in_features): + raise ValueError("modulated NVFP4 input and row index have incompatible shapes") + if x.dtype != torch.bfloat16 or not x.is_cuda or not x.is_contiguous(): + raise ValueError("modulated NVFP4 dispatch requires contiguous CUDA BF16 input") + + from .nvfp4_quant import vortex_quantize_modulated_nvfp4 + + packed_x = vortex_quantize_modulated_nvfp4(x, shift, scale, row_index) + bias = self.bias.to(x) if self.bias is not None else None + output = functional.linear(packed_x, self._packed_weight(), bias)[ + : x.shape[0], : self.out_features + ] + return output + + def forward_swiglu(self, gate_up: torch.Tensor) -> torch.Tensor: + """Fuse exact BF16 SwiGLU into activation packing for H3 FC2.""" + if self.role != "h3_mlp_fc2": + raise ValueError("SwiGLU NVFP4 dispatch is restricted to H3 FC2") + if gate_up.shape[-1] != self.in_features * 2: + raise ValueError("SwiGLU input width must be twice the FC2 input width") + if ( + self.full_precision_matrix_mult + or self.pre_quant_scale is not None + or self.active_lora is not None + or self.lora_strength != 0.0 + or torch.is_grad_enabled() + or gate_up.requires_grad + ): + gate, up = gate_up.chunk(2, dim=-1) + return self(torch.nn.functional.silu(gate).mul_(up)) + if gate_up.dtype != torch.bfloat16 or not gate_up.is_cuda or not gate_up.is_contiguous(): + raise ValueError("SwiGLU NVFP4 dispatch requires contiguous CUDA BF16 input") + + from .nvfp4_quant import vortex_quantize_swiglu_nvfp4 + + packed_x = vortex_quantize_swiglu_nvfp4(gate_up) + bias = self.bias.to(gate_up) if self.bias is not None else None + output = functional.linear(packed_x, self._packed_weight(), bias)[ + : gate_up.shape[0], : self.out_features + ] + return output + + +def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_dtype=torch.bfloat16, role: str = "other") -> Nvfp4Linear: """Load one Comfy-format NVFP4 linear from a safetensors tensor mapping.""" sidecar_key = f"{prefix}.comfy_quant" metadata = parse_quant_sidecar(tensors[sidecar_key]) @@ -110,4 +190,4 @@ def load_nvfp4_linear(tensors: dict[str, torch.Tensor], prefix: str, *, output_d in_features=in_features, out_features=weight.shape[0], ) - return Nvfp4Linear(packed, output_dtype=output_dtype) + return Nvfp4Linear(packed, output_dtype=output_dtype, role=role) diff --git a/src/h3_blackwell_runtime/nvfp4_quant.py b/src/h3_blackwell_runtime/nvfp4_quant.py index 97ea031..b964944 100644 --- a/src/h3_blackwell_runtime/nvfp4_quant.py +++ b/src/h3_blackwell_runtime/nvfp4_quant.py @@ -214,3 +214,131 @@ def vortex_native_quantize_nvfp4( ), ), ) + + +def vortex_native_quantize_nvfp4_into( + tensor: torch.Tensor, + scale: torch.Tensor, + qdata: torch.Tensor, + block_scale: torch.Tensor, + *, + hi_first: bool = True, +) -> None: + """Pack BF16 activation into caller-owned NVFP4 ring buffers.""" + if tensor.dim() != 2 or tensor.dtype != torch.bfloat16: + raise ValueError("vortex_native_quantize_nvfp4_into expects a 2D BF16 tensor") + if not tensor.is_cuda or not tensor.is_contiguous(): + raise ValueError("vortex_native_quantize_nvfp4_into expects contiguous CUDA input") + _vortex_scale_extension().quantize_nvfp4_bf16_into( + tensor, + torch.as_tensor(scale, device=tensor.device, dtype=torch.float32), + qdata, + block_scale, + _env_int("H3_NVFP4_PACK_THREADS", 256), + hi_first, + ) + + +def vortex_native_quantize_modulated_nvfp4( + tensor: torch.Tensor, + shift: torch.Tensor, + scale: torch.Tensor, + row_index: torch.Tensor, +): + """Produce exact Comfy-layout NVFP4 bytes without materializing modulation.""" + if tensor.dim() != 2 or tensor.dtype != torch.bfloat16: + raise ValueError("vortex_native_quantize_modulated_nvfp4 expects a 2D BF16 tensor") + if not tensor.is_cuda or not tensor.is_contiguous(): + raise ValueError("vortex_native_quantize_modulated_nvfp4 expects contiguous CUDA input") + if shift.shape != scale.shape or shift.dim() != 2 or shift.shape[1] != tensor.shape[1]: + raise ValueError("shift and scale must have matching [table_rows, hidden] shapes") + if shift.device != tensor.device or scale.device != tensor.device: + raise ValueError("modulation tables must be on the input device") + if shift.dtype != scale.dtype or shift.dtype not in {torch.bfloat16, torch.float32}: + raise ValueError("modulation tables must share BF16 or FP32 dtype") + shift = shift.contiguous() + scale = scale.contiguous() + if row_index.shape != (tensor.shape[0],) or row_index.dtype != torch.int32: + raise ValueError("row_index must be int32 with one entry per input row") + if row_index.device != tensor.device or not row_index.is_contiguous(): + raise ValueError("row_index must be contiguous on the input device") + + from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + blocks, scale_threads = _vortex_scale_geometry(tensor.numel()) + return _vortex_scale_extension().quantize_nvfp4_modulated_bf16( + tensor, + shift, + scale, + row_index, + float(F8_E4M3_MAX * F4_E2M1_MAX), + TensorCoreNVFP4Layout.get_padded_shape(tuple(tensor.shape)) != tuple(tensor.shape), + blocks, + scale_threads, + _env_int("H3_NVFP4_PACK_THREADS", 256), + ) + + +def vortex_quantize_modulated_nvfp4( + tensor: torch.Tensor, + shift: torch.Tensor, + scale: torch.Tensor, + row_index: torch.Tensor, +): + """Wrap fused modulation output as a Comfy TensorCoreNVFP4Layout tensor.""" + from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout + + tensor_scale, qdata, block_scale = vortex_native_quantize_modulated_nvfp4( + tensor, shift, scale, row_index, + ) + return QuantizedTensor( + qdata, + "TensorCoreNVFP4Layout", + TensorCoreNVFP4Layout.Params( + scale=tensor_scale, + orig_dtype=tensor.dtype, + orig_shape=tuple(tensor.shape), + block_scale=block_scale, + ), + ) + + +def vortex_native_quantize_swiglu_nvfp4(tensor: torch.Tensor): + """Produce exact Comfy-layout NVFP4 bytes without materializing SwiGLU.""" + if tensor.dim() != 2 or tensor.dtype != torch.bfloat16 or tensor.shape[1] % 2: + raise ValueError("vortex_native_quantize_swiglu_nvfp4 expects even-width 2D BF16 input") + if not tensor.is_cuda or not tensor.is_contiguous(): + raise ValueError("vortex_native_quantize_swiglu_nvfp4 expects contiguous CUDA input") + + from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + output_shape = (tensor.shape[0], tensor.shape[1] // 2) + blocks, scale_threads = _vortex_scale_geometry(tensor.numel() // 2) + return _vortex_scale_extension().quantize_nvfp4_swiglu_bf16( + tensor, + float(F8_E4M3_MAX * F4_E2M1_MAX), + TensorCoreNVFP4Layout.get_padded_shape(output_shape) != output_shape, + blocks, + scale_threads, + _env_int("H3_NVFP4_PACK_THREADS", 256), + ) + + +def vortex_quantize_swiglu_nvfp4(tensor: torch.Tensor): + """Wrap fused SwiGLU output as a Comfy TensorCoreNVFP4Layout tensor.""" + from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout + + tensor_scale, qdata, block_scale = vortex_native_quantize_swiglu_nvfp4(tensor) + output_shape = (tensor.shape[0], tensor.shape[1] // 2) + return QuantizedTensor( + qdata, + "TensorCoreNVFP4Layout", + TensorCoreNVFP4Layout.Params( + scale=tensor_scale, + orig_dtype=tensor.dtype, + orig_shape=output_shape, + block_scale=block_scale, + ), + ) diff --git a/src/h3_blackwell_runtime/sage2_entry.py b/src/h3_blackwell_runtime/sage2_entry.py new file mode 100644 index 0000000..215d4d1 --- /dev/null +++ b/src/h3_blackwell_runtime/sage2_entry.py @@ -0,0 +1,81 @@ +"""Exact fused entry preparation for the pinned SageAttention 2.2.0 path.""" + +from __future__ import annotations + +import math + +import torch + + +def prepare_qk( + q: torch.Tensor, + k: torch.Tensor, + rotation: torch.Tensor, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + epsilon: float, + *, + materialize_q: bool = False, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + """Prepare Q/K in place and emit Sage2's exact per-warp Q representation.""" + from .nvfp4_quant import _vortex_scale_extension + + return tuple( + _vortex_scale_extension().sage2_prepare_qk( + q, k, rotation, q_weight, k_weight, epsilon, materialize_q, + ) + ) + + +def prepare_v(v: torch.Tensor, *, scale_max: float = 2.25) -> tuple[torch.Tensor, torch.Tensor]: + """Emit Sage2's FP8 V tensor and per-channel scale without a BF16 transpose tensor.""" + from .nvfp4_quant import _vortex_scale_extension + + return tuple(_vortex_scale_extension().sage2_prepare_v(v, scale_max)) + + +def attention_nhd( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + rotation: torch.Tensor, + q_weight: torch.Tensor, + k_weight: torch.Tensor, + epsilon: float, +) -> torch.Tensor: + """Run the unchanged Sage2 mainloop after fused strided-NHD entry preparation.""" + import sageattention.core as sage_core + import sageattention.quant as sage_quant + + q_int8, q_scale, _ = prepare_qk( + q, k, rotation, q_weight, k_weight, epsilon, materialize_q=False, + ) + k_mean = k.mean(dim=1, keepdim=True) + k_int8 = torch.empty(k.shape, dtype=torch.int8, device=k.device) + k_scale = torch.empty( + (k.shape[0], k.shape[2], math.ceil(k.shape[1] / 64)), + dtype=torch.float32, + device=k.device, + ) + sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( + k, k_mean.squeeze(1), k_int8, k_scale, 64, 0, + ) + v_fp8, v_scale, _ = sage_core.per_channel_fp8( + v, tensor_layout="NHD", scale_max=2.25, smooth_v=False, + ) + output = torch.empty(q.shape, dtype=q.dtype, device=q.device) + sage_core.sm89_compile.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( + q_int8, + k_int8, + v_fp8, + output, + q_scale, + k_scale, + v_scale, + 0, + 0, + 2, + output.shape[-1] ** -0.5, + 0, + ) + return output diff --git a/src/h3_blackwell_runtime/sampler.py b/src/h3_blackwell_runtime/sampler.py index 713e2bd..d33a6ab 100644 --- a/src/h3_blackwell_runtime/sampler.py +++ b/src/h3_blackwell_runtime/sampler.py @@ -80,6 +80,17 @@ def _cache_sample(x: torch.Tensor, factor: int) -> torch.Tensor: return x[..., ::factor] +def _profiled(trace: dict[str, float] | None, name: str, device: torch.device, fn): + if trace is None: + return fn() + torch.cuda.synchronize(device) + started = time.perf_counter() + result = fn() + torch.cuda.synchronize(device) + trace[name] = time.perf_counter() - started + return result + + @torch.inference_mode() def sample_video_res_multistep( model, @@ -104,6 +115,7 @@ def sample_video_res_multistep( cache_subsample_factor: int = 2, cache_stats: dict | None = None, audio_step_trace: list[dict] | None = None, + sampling_stage_trace: list[dict] | None = None, ) -> torch.Tensor: """Direct H3 beta/RES sampling with Comfy-equivalent joint AV carry semantics.""" sigmas = beta_sigmas(steps, device=video.device) @@ -127,6 +139,11 @@ def sample_video_res_multistep( } for index, sigma in enumerate(sigmas[:-1], start=1): step_started = time.perf_counter() + stage_row = None + if sampling_stage_trace is not None: + torch.cuda.synchronize(video.device) + profiled_step_started = time.perf_counter() + stage_row = {"step": index} previous_index = index - 1 sigma_down = sigmas[index] audio_before = audio_carried if audio_step_trace is None else audio_carried.detach().cpu() @@ -149,37 +166,64 @@ def sample_video_res_multistep( if cache_stats is not None: cache_stats["rates"].append({"step": previous_index, "rate": rate, "skipped": skipped}) if skipped: - video_denoised = video + cache["video_diff"] - audio_denoised = audio_carried + cache["audio_diff"] + video_denoised, audio_denoised = _profiled( + stage_row, + "cache_reuse", + video.device, + lambda: (video + cache["video_diff"], audio_carried + cache["audio_diff"]), + ) if cache_stats is not None: cache_stats["skipped_steps"] += 1 else: - sigma_audio = _audio_sigma(sigma) - carry = sigma_audio / sigma - native_audio = audio_carried.to(torch.bfloat16) * carry - step_timesteps = None if model_timesteps is None else model_timesteps[previous_index] - hidden, times, segments, positions, video_segment, audio_segment = packer( - text, - video, - native_audio, - _model_sigma(sigma), - step_timesteps, - text_token_tags=text_token_tags, - cond_latents=cond_latents, - cond_frame_indices=cond_frame_indices, - frame_count=frame_count, - seed=seed, + def prepare_audio(): + sigma_audio = _audio_sigma(sigma) + carry = sigma_audio / sigma + native_audio = audio_carried.to(torch.bfloat16) * carry + step_timesteps = None if model_timesteps is None else model_timesteps[previous_index] + return sigma_audio, carry, native_audio, step_timesteps + + sigma_audio, carry, native_audio, step_timesteps = _profiled( + stage_row, "audio_input", video.device, prepare_audio, + ) + hidden, times, segments, positions, video_segment, audio_segment = _profiled( + stage_row, + "packing", + video.device, + lambda: packer( + text, + video, + native_audio, + _model_sigma(sigma), + step_timesteps, + text_token_tags=text_token_tags, + cond_latents=cond_latents, + cond_frame_indices=cond_frame_indices, + frame_count=frame_count, + seed=seed, + ), + ) + raw_video, raw_audio = _profiled( + stage_row, + "denoiser", + video.device, + lambda: model(hidden, times, positions, segments, video_segment, audio_segment), + ) + + def transform_output(): + converted_video = raw_video.to(torch.bfloat16).float() + converted_audio = raw_audio.to(torch.bfloat16) + velocity_video = -unpatchify_video( + converted_video, video.shape[2], video.shape[-2], video.shape[-1], + ) + velocity_audio = ( + (1.0 - 4.0) * (audio_carried.to(torch.bfloat16) * carry.to(torch.bfloat16)) + + (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(converted_audio)) + ).float() + return video - sigma * velocity_video, audio_carried - sigma * velocity_audio + + video_denoised, audio_denoised = _profiled( + stage_row, "output_transform", video.device, transform_output, ) - raw_video, raw_audio = model(hidden, times, positions, segments, video_segment, audio_segment) - raw_video = raw_video.to(torch.bfloat16).float() - raw_audio = raw_audio.to(torch.bfloat16) - velocity_video = -unpatchify_video(raw_video, video.shape[2], video.shape[-2], video.shape[-1]) - velocity_audio = ( - (1.0 - 4.0) * (audio_carried.to(torch.bfloat16) * carry.to(torch.bfloat16)) - + (1.0 + 3.0 * sigma_audio).to(torch.bfloat16) * (-_unpack_audio(raw_audio)) - ).float() - video_denoised = video - sigma * velocity_video - audio_denoised = audio_carried - sigma * velocity_audio if cache_mode is not None: cache["video_diff"] = (video_denoised - video).detach() cache["audio_diff"] = (audio_denoised - audio_carried).detach() @@ -187,8 +231,19 @@ def sample_video_res_multistep( cache["audio_prev"] = _cache_sample(audio_carried, cache_subsample_factor).detach().clone() cache["prev_norm"] = video.flatten().abs().mean() + audio_carried.flatten().abs().mean() previous_sigma = sigmas[previous_index - 1] if previous_index else None - video = res_multistep_update(video, video_denoised, sigma, sigma_down, video_history, video_history_sigma, previous_sigma) - audio_carried = res_multistep_update(audio_carried, audio_denoised, sigma, sigma_down, audio_history, audio_history_sigma, previous_sigma) + + def update_state(): + updated_video = res_multistep_update( + video, video_denoised, sigma, sigma_down, + video_history, video_history_sigma, previous_sigma, + ) + updated_audio = res_multistep_update( + audio_carried, audio_denoised, sigma, sigma_down, + audio_history, audio_history_sigma, previous_sigma, + ) + return updated_video, updated_audio + + video, audio_carried = _profiled(stage_row, "sampler_update", video.device, update_state) if audio_step_trace is not None: audio_step_trace.append({ "step": index, @@ -201,6 +256,10 @@ def sample_video_res_multistep( }) video_history, audio_history = video_denoised, audio_denoised video_history_sigma = audio_history_sigma = sigma_down + if stage_row is not None: + torch.cuda.synchronize(video.device) + stage_row["step_total"] = time.perf_counter() - profiled_step_started + sampling_stage_trace.append(stage_row) if progress: elapsed = time.perf_counter() - started eta = elapsed / index * (total_steps - index) diff --git a/src/h3_blackwell_runtime/tensor_parallel.py b/src/h3_blackwell_runtime/tensor_parallel.py index 3b7868a..68d7bdf 100644 --- a/src/h3_blackwell_runtime/tensor_parallel.py +++ b/src/h3_blackwell_runtime/tensor_parallel.py @@ -39,7 +39,7 @@ def select_nvfp4_outputs(linear: Nvfp4Linear, ranges: tuple[tuple[int, int], ... in_features=linear.in_features, out_features=indices.numel(), ) - return Nvfp4Linear(tensors, output_dtype=linear.output_dtype) + return Nvfp4Linear(tensors, output_dtype=linear.output_dtype, role=linear.role) def slice_nvfp4_inputs(linear: Nvfp4Linear, start: int, stop: int) -> tuple[Nvfp4Linear, torch.Tensor | None]: @@ -63,7 +63,7 @@ def slice_nvfp4_inputs(linear: Nvfp4Linear, start: int, stop: int) -> tuple[Nvfp in_features=stop - start, out_features=linear.out_features, ) - return Nvfp4Linear(tensors, output_dtype=linear.output_dtype), bias + return Nvfp4Linear(tensors, output_dtype=linear.output_dtype, role=linear.role), bias def configure_h3_tensor_parallel(model, context: SequenceParallelContext) -> None: diff --git a/tests/test_fl2va_contracts.py b/tests/test_fl2va_contracts.py index a608fef..73c0390 100644 --- a/tests/test_fl2va_contracts.py +++ b/tests/test_fl2va_contracts.py @@ -9,7 +9,7 @@ from torch import nn from torch.nn import functional as F from h3_blackwell_runtime.packing import FRAME_RESCALE, H3PromptPacker, _video_t_spans -from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, run_attention +from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, run_attention, run_sage_attention_nhd from h3_blackwell_runtime.qwen3vl_vision import ( TEXT_HEAD_DIM, TEXT_ROPE_DIMS, @@ -65,6 +65,22 @@ class AttentionBackendContracts(unittest.TestCase): def test_hot_backends_include_benchmark_candidates(self): self.assertTrue({"sage2", "cudnn_sdpa", "ck_int8", "flash4"}.issubset(AVAILABLE_BACKENDS)) + def test_sage_nhd_preserves_projection_strides(self): + qkv = torch.randn(1, 3, 6, 4) + q, k, v = qkv.split(2, dim=2) + expected = torch.randn(1, 3, 2, 4) + sage = unittest.mock.MagicMock(return_value=expected) + with patch.dict(sys.modules, {"sageattention": SimpleNamespace(sageattn=sage)}): + actual = run_sage_attention_nhd(q, k, v) + self.assertIs(actual, expected) + called_q, called_k, called_v = sage.call_args.args + self.assertIs(called_q, q) + self.assertIs(called_k, k) + self.assertIs(called_v, v) + self.assertFalse(called_q.is_contiguous()) + self.assertEqual(sage.call_args.kwargs["tensor_layout"], "NHD") + self.assertFalse(sage.call_args.kwargs["smooth_k"]) + def test_flash4_dispatches_bshd_and_restores_hnd(self): q = torch.randn(1, 2, 3, 128, dtype=torch.bfloat16) expected_bshd = torch.randn(1, 3, 2, 128, dtype=torch.bfloat16) diff --git a/tests/test_h3_fusion.py b/tests/test_h3_fusion.py new file mode 100644 index 0000000..6cc9434 --- /dev/null +++ b/tests/test_h3_fusion.py @@ -0,0 +1,66 @@ +import unittest + +import torch + +from h3_blackwell_runtime.block import gate_segments, modulate_segments + + +class SegmentIndexContracts(unittest.TestCase): + def test_segment_index_covers_rows_and_reuses_layout(self): + try: + from h3_blackwell_runtime.h3_fusion import segment_index + except ImportError: + self.skipTest("Triton is not installed") + segments = [(0, 2, 1), (2, 5, 4), (5, 8, 0)] + first = segment_index(8, segments, torch.device("cpu")) + second = segment_index(8, segments, torch.device("cpu")) + self.assertIs(first, second) + torch.testing.assert_close(first, torch.tensor([1, 1, 4, 4, 4, 0, 0, 0], dtype=torch.int32)) + + def test_segment_index_rejects_gaps(self): + try: + from h3_blackwell_runtime.h3_fusion import segment_index + except ImportError: + self.skipTest("Triton is not installed") + with self.assertRaisesRegex(ValueError, "ordered, contiguous"): + segment_index(4, [(0, 2, 0), (3, 4, 1)], torch.device("cpu")) + + +@unittest.skipUnless(torch.cuda.is_available(), "requires CUDA") +class FusedElementwiseParity(unittest.TestCase): + @classmethod + def setUpClass(cls): + try: + from h3_blackwell_runtime.h3_fusion import ( + fused_gate_add_, fused_modulate_, segment_index, + ) + except ImportError as error: + raise unittest.SkipTest("Triton is not installed") from error + cls.fused_gate_add = staticmethod(fused_gate_add_) + cls.fused_modulate = staticmethod(fused_modulate_) + cls.segment_index = staticmethod(segment_index) + + def setUp(self): + torch.manual_seed(440420) + self.segments = [(0, 19, 1), (19, 100, 4), (100, 201, 8), (201, 259, 2)] + self.row_index = self.segment_index(259, self.segments, torch.device("cuda")) + + def test_modulation_is_bit_exact(self): + x = torch.randn(259, 1024, device="cuda", dtype=torch.bfloat16) + shift = torch.randn(9, 1024, device="cuda", dtype=torch.float32) + scale = torch.randn(9, 1024, device="cuda", dtype=torch.float32) + expected = modulate_segments(x, shift, scale, self.segments) + actual = self.fused_modulate(x.clone(), shift, scale, self.row_index) + self.assertTrue(torch.equal(actual, expected)) + + def test_gate_add_is_bit_exact(self): + residual = torch.randn(259, 1024, device="cuda", dtype=torch.bfloat16) + update = torch.randn_like(residual) + gate = torch.randn(9, 1024, device="cuda", dtype=torch.float32) + expected = gate_segments(residual, update, gate, self.segments) + actual = self.fused_gate_add(residual.clone(), update, gate, self.row_index) + self.assertTrue(torch.equal(actual, expected)) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_nvfp4_quant.py b/tests/test_nvfp4_quant.py new file mode 100644 index 0000000..522a3ff --- /dev/null +++ b/tests/test_nvfp4_quant.py @@ -0,0 +1,123 @@ +import unittest + +import torch + + +@unittest.skipUnless(torch.cuda.is_available(), "requires CUDA") +class Nvfp4NativePackParity(unittest.TestCase): + def test_h3_projection_widths_match_comfy_layout(self): + try: + import comfy_kitchen as ck + from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX + from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension + except ImportError as error: + raise unittest.SkipTest("Comfy Kitchen is not installed") from error + + torch.manual_seed(440420) + extension = _vortex_scale_extension() + for features in (5376, 7168, 14336): + with self.subTest(features=features): + value = torch.randn(129, features, device="cuda", dtype=torch.bfloat16) + scale = (value.abs().amax() / float(F8_E4M3_MAX * F4_E2M1_MAX)).float() + expected_qdata, expected_block_scale = ck.quantize_nvfp4(value, scale, pad_16x=True) + actual_qdata, actual_block_scale = extension.quantize_nvfp4_bf16(value, scale, True, 512) + self.assertTrue(torch.equal(actual_qdata, expected_qdata)) + self.assertTrue(torch.equal(actual_block_scale.view(torch.uint8), expected_block_scale.view(torch.uint8))) + into_qdata = torch.empty_like(actual_qdata) + into_block_scale = torch.empty_like(actual_block_scale) + extension.quantize_nvfp4_bf16_into( + value, scale, into_qdata, into_block_scale, 512, + ) + self.assertTrue(torch.equal(into_qdata, expected_qdata)) + self.assertTrue( + torch.equal( + into_block_scale.view(torch.uint8), + expected_block_scale.view(torch.uint8), + ) + ) + low_first_qdata = torch.empty_like(actual_qdata) + extension.quantize_nvfp4_bf16_into( + value, + scale, + low_first_qdata, + into_block_scale, + 512, + False, + ) + expected_low_first = ( + ((expected_qdata & 0x0F) << 4) + | ((expected_qdata & 0xF0) >> 4) + ) + self.assertTrue(torch.equal(low_first_qdata, expected_low_first)) + + def test_modulated_producer_matches_materialized_bf16(self): + try: + import comfy_kitchen as ck + from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX + from h3_blackwell_runtime.h3_fusion import fused_modulate_ + from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension + except ImportError as error: + raise unittest.SkipTest("Comfy Kitchen or Triton is not installed") from error + + torch.manual_seed(440421) + value = torch.randn(129, 128, device="cuda", dtype=torch.bfloat16) + row_index = torch.arange(129, device="cuda", dtype=torch.int32) % 3 + divisor = float(F8_E4M3_MAX * F4_E2M1_MAX) + extension = _vortex_scale_extension() + for table_dtype in (torch.bfloat16, torch.float32): + with self.subTest(table_dtype=table_dtype): + shift = torch.randn(3, 128, device="cuda", dtype=table_dtype) + scale = torch.randn(3, 128, device="cuda", dtype=table_dtype) + materialized = fused_modulate_( + value.clone(), shift, scale, row_index, + ) + expected_scale = (materialized.abs().amax() / divisor).to( + torch.bfloat16, + ).float() + expected_qdata, expected_sfa = ck.quantize_nvfp4( + materialized, expected_scale, pad_16x=True, + ) + actual_scale, actual_qdata, actual_sfa = ( + extension.quantize_nvfp4_modulated_bf16( + value, shift, scale, row_index, divisor, True, 32, 128, 128, + ) + ) + self.assertTrue(torch.equal(actual_scale, expected_scale)) + self.assertTrue(torch.equal(actual_qdata, expected_qdata)) + self.assertTrue( + torch.equal(actual_sfa.view(torch.uint8), expected_sfa.view(torch.uint8)) + ) + + def test_swiglu_producer_matches_materialized_bf16(self): + try: + import comfy_kitchen as ck + from comfy_kitchen.float_utils import F4_E2M1_MAX, F8_E4M3_MAX + from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension + except ImportError as error: + raise unittest.SkipTest("Comfy Kitchen is not installed") from error + + torch.manual_seed(440422) + gate_up = torch.randn(129, 256, device="cuda", dtype=torch.bfloat16) + gate, up = gate_up.chunk(2, dim=-1) + materialized = torch.nn.functional.silu(gate).mul_(up) + divisor = float(F8_E4M3_MAX * F4_E2M1_MAX) + expected_scale = (materialized.abs().amax() / divisor).to( + torch.bfloat16, + ).float() + expected_qdata, expected_sfa = ck.quantize_nvfp4( + materialized, expected_scale, pad_16x=True, + ) + actual_scale, actual_qdata, actual_sfa = ( + _vortex_scale_extension().quantize_nvfp4_swiglu_bf16( + gate_up, divisor, True, 32, 128, 128, + ) + ) + self.assertTrue(torch.equal(actual_scale, expected_scale)) + self.assertTrue(torch.equal(actual_qdata, expected_qdata)) + self.assertTrue( + torch.equal(actual_sfa.view(torch.uint8), expected_sfa.view(torch.uint8)) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/Run-RunPodUlyssesBenchmark.ps1 b/tools/Run-RunPodUlyssesBenchmark.ps1 new file mode 100644 index 0000000..fc85b3b --- /dev/null +++ b/tools/Run-RunPodUlyssesBenchmark.ps1 @@ -0,0 +1,51 @@ +param( + [Parameter(Mandatory = $true)] + [string]$SshTarget, + + [int]$SshPort = 22, + [int]$GpuCount = 4, + [int]$Runs = 2, + [string]$TopologyClass = "mixed", + [string]$SshKey = (Join-Path $HOME ".ssh\inceptal-dev-envs") +) + +$ErrorActionPreference = "Stop" + +$repo = "/workspace/h3-runtime" +$python = "/runpod-volume/h3-runtime/h3-venv/bin/python" +$benchmark = "benchmarks/t2va-dialogue-quoted-1344x768-243f-base12-sdpa-seed440420.json" +$model = "/runpod-volume/ComfyUI/models/diffusion_models/minimax_h3_fl2va_pruned_nvfp4.safetensors" +$textEncoder = "/runpod-volume/ComfyUI/models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors" +$reportRoot = "/runpod-volume/h3-benchmarks" +$reportStem = "rtxpro6000-server-${GpuCount}gpu-${TopologyClass}-direct-ulysses-sdpa-1344x768-243f-base12-seed440420" + +$common = @( + "PYTHONPATH=$repo/src" + $python + "-m torch.distributed.run" + "--standalone" + "--nnodes=1" + "--nproc-per-node=$GpuCount" + "tools/distributed_t2va.py" + "--attention sdpa" + "--mode ulysses" + "--model $model" + "--text-encoder $textEncoder" +) -join " " + +$commands = @( + "stty -echo" + "cd $repo" + "unset H3_DISABLE_MMAP H3_FAST_SAFETENSORS" +) + +for ($run = 1; $run -le $Runs; $run++) { + $report = "$reportRoot/$reportStem-run$run.json" + $commands += "$common --benchmark $benchmark --report $report" +} + +$commands += "exit" +$commandsText = ($commands -join "`n") + "`n" + +$commandsText | & ssh -tt -p $SshPort -i $SshKey $SshTarget +exit $LASTEXITCODE diff --git a/tools/benchmark_sage2_entry_blocks.py b/tools/benchmark_sage2_entry_blocks.py new file mode 100644 index 0000000..7b6096c --- /dev/null +++ b/tools/benchmark_sage2_entry_blocks.py @@ -0,0 +1,289 @@ +"""Gate the fused Sage2 entry path on captured complete H3 blocks.""" + +from __future__ import annotations + +import argparse +import json +import os +import time +import types +from pathlib import Path + +import torch + +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.denoiser import H3PackedDenoiser +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents +from profile_h3_block import profile_module_kernels, summarize + + +def compare(actual: torch.Tensor, expected: torch.Tensor) -> dict[str, float | int | bool]: + different = torch.count_nonzero(actual != expected).item() + delta = (actual.float() - expected.float()).abs() + return { + "equal": different == 0, + "different_elements": different, + "max_abs": float(delta.max()), + "mean_abs": float(delta.mean()), + } + + +def timed(fn) -> tuple[float, torch.Tensor]: + torch.cuda.synchronize() + started = time.perf_counter() + output = fn() + torch.cuda.synchronize() + return time.perf_counter() - started, output + + +def memory(fn) -> dict[str, int]: + torch.cuda.synchronize() + torch.cuda.reset_peak_memory_stats() + allocated = torch.cuda.memory_allocated() + reserved = torch.cuda.memory_reserved() + output = fn() + torch.cuda.synchronize() + result = { + "allocated_bytes_before": allocated, + "reserved_bytes_before": reserved, + "peak_allocated_bytes": torch.cuda.max_memory_allocated(), + "peak_reserved_bytes": torch.cuda.max_memory_reserved(), + "peak_allocated_delta_bytes": torch.cuda.max_memory_allocated() - allocated, + "peak_reserved_delta_bytes": torch.cuda.max_memory_reserved() - reserved, + } + del output + return result + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--capture", type=Path) + parser.add_argument("--model", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--blocks", type=int, nargs="+", default=[0, 24, 49]) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--warmup", type=int, default=2) + parser.add_argument("--rounds", type=int, default=8) + parser.add_argument("--cuda-profiler-capture", action="store_true") + parser.add_argument("--profile-mode", choices=("baseline", "candidate"), default="baseline") + parser.add_argument( + "--output", + type=Path, + default=Path("/output/h3-blackwell-runtime/benchmarks/gb10-sage2-p1-block-gate.json"), + ) + args = parser.parse_args() + + os.environ["H3_SAGE_QKV_LAYOUT"] = "strided_nhd" + os.environ["H3_SAGE_ENTRY_FUSION"] = "0" + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model) + model = H3PackedDenoiser.from_checkpoint(checkpoint, attention_backend="sage2").eval() + if args.capture is not None: + captured = torch.load(args.capture / "input.pt", map_location="cuda", weights_only=False) + hidden = captured["hidden"].to("cuda") + timesteps = captured["timesteps"].to("cuda") + positions = captured["position_ids"].to("cuda") + segments = captured["segments"] + workload = {"capture": str(args.capture)} + else: + packer = H3PromptPacker(checkpoint) + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device="cuda", + ) + sigma = beta_sigmas(args.steps, device="cuda")[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn(1, args.text_tokens, 5376, device="cuda", dtype=torch.bfloat16) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + workload = { + "resolution": [args.width, args.height], + "frames": aligned_frames, + "steps": args.steps, + "sampler_step": args.sampler_step, + "seed": args.seed, + "text_tokens": args.text_tokens, + } + rotation = h3_rope_rotation(positions.to("cuda"), model.backbone.inv_freq, torch.bfloat16) + workload["tokens"] = hidden.shape[0] + workload["hidden_shape"] = list(hidden.shape) + workload["rotation_shape"] = list(rotation.shape) + print(workload, flush=True) + block_inputs = {} + block_outputs = {} + rows = [] + + with torch.inference_mode(): + for index, (block, adaln) in enumerate(zip(model.backbone.blocks, model.backbone.adaln, strict=True)): + if index in args.blocks: + block_inputs[index] = hidden + hidden = block(hidden, rotation, *adaln(timesteps), segments) + if index in args.blocks: + block_outputs[index] = hidden + if index >= max(args.blocks): + break + + baseline_forwards = {index: model.backbone.blocks[index].attention.forward for index in args.blocks} + + def fused_attention_forward( + attention, + x, + rope_rotation, + sequence_parallel=None, + tensor_parallel=None, + modulation=None, + ): + from h3_blackwell_runtime.sage2_entry import attention_nhd + + if sequence_parallel is not None or tensor_parallel is not None: + raise ValueError("the Sage2 entry-fusion benchmark is single-GPU only") + sequence = x.shape[0] + inner = attention.heads * attention.head_dim + qkv = attention.qkv_proj.forward_modulated(x, *modulation) if modulation is not None else attention.qkv_proj(x) + q, k, v = qkv.split(inner, dim=-1) + q = q.view(1, sequence, attention.heads, attention.head_dim) + k = k.view(1, sequence, attention.heads, attention.head_dim) + v = v.view(1, sequence, attention.heads, attention.head_dim) + output = attention_nhd( + q, + k, + v, + rope_rotation, + attention.q_norm_weight, + attention.k_norm_weight, + attention.eps, + ) + return attention.out_proj(output.reshape(sequence, inner)) + + candidate_forwards = { + index: types.MethodType(fused_attention_forward, model.backbone.blocks[index].attention) + for index in args.blocks + } + + def set_fused(index: int, fused: bool) -> None: + model.backbone.blocks[index].attention.forward = ( + candidate_forwards[index] if fused else baseline_forwards[index] + ) + + if args.cuda_profiler_capture: + checksums = {} + fused = args.profile_mode == "candidate" + os.environ["H3_SAGE_ENTRY_FUSION"] = "1" if fused else "0" + for index in args.blocks: + block = model.backbone.blocks[index] + set_fused(index, fused) + adaln_values = model.backbone.adaln[index](timesteps) + for _ in range(args.warmup): + block(block_inputs[index], rotation, *adaln_values, segments) + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStart() + output = block(block_inputs[index], rotation, *adaln_values, segments) + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStop() + checksums[str(index)] = float(output.float().sum()) + result = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "workload": workload, + "profile_mode": args.profile_mode, + "blocks": args.blocks, + "checksums": checksums, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(result, indent=2) + "\n") + print(json.dumps(result, indent=2), flush=True) + return + + for index in args.blocks: + hidden = block_inputs[index] + expected = block_outputs[index] + block = model.backbone.blocks[index] + adaln_values = model.backbone.adaln[index](timesteps) + + def run(fused: bool): + set_fused(index, fused) + return block(hidden, rotation, *adaln_values, segments) + + for _ in range(args.warmup): + run(False) + run(True) + torch.cuda.synchronize() + + baseline = run(False) + candidate = run(True) + torch.cuda.synchronize() + parity = compare(candidate, baseline) + baseline_capture = compare(baseline, expected) + candidate_capture = compare(candidate, expected) + del baseline, candidate + + samples = {"baseline": [], "candidate": []} + for round_index in range(args.rounds): + order = (False, True) if round_index % 2 == 0 else (True, False) + for fused in order: + elapsed, output = timed(lambda fused=fused: run(fused)) + samples["candidate" if fused else "baseline"].append(elapsed) + del output + + os.environ["H3_SAGE_ENTRY_FUSION"] = "0" + baseline_memory = memory(lambda: run(False)) + os.environ["H3_SAGE_ENTRY_FUSION"] = "1" + candidate_memory = memory(lambda: run(True)) + + os.environ["H3_SAGE_ENTRY_FUSION"] = "0" + baseline_events, baseline_launches = profile_module_kernels(lambda: run(False), warmup=1, iterations=1, row_limit=1000) + os.environ["H3_SAGE_ENTRY_FUSION"] = "1" + candidate_events, candidate_launches = profile_module_kernels(lambda: run(True), warmup=1, iterations=1, row_limit=1000) + baseline_launches["device_kernel_launches"] = sum( + row["count"] for row in baseline_events if row["device_type"] == "DeviceType.CUDA" + ) + candidate_launches["device_kernel_launches"] = sum( + row["count"] for row in candidate_events if row["device_type"] == "DeviceType.CUDA" + ) + + baseline_timing = summarize(samples["baseline"]) + candidate_timing = summarize(samples["candidate"]) + rows.append({ + "block": index, + "candidate_vs_baseline": parity, + "baseline_vs_capture": baseline_capture, + "candidate_vs_capture": candidate_capture, + "baseline": { + "timing": baseline_timing, + "memory": baseline_memory, + "launches": baseline_launches, + }, + "candidate": { + "timing": candidate_timing, + "memory": candidate_memory, + "launches": candidate_launches, + }, + "p50_improvement_percent": + (baseline_timing["p50_s"] - candidate_timing["p50_s"]) + / baseline_timing["p50_s"] + * 100.0, + }) + del adaln_values + + result = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "workload": workload, + "rounds": args.rounds, + "blocks": rows, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(result, indent=2) + "\n") + print(json.dumps(result, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/prepare_runpod_region.py b/tools/prepare_runpod_region.py new file mode 100644 index 0000000..1fae0ac --- /dev/null +++ b/tools/prepare_runpod_region.py @@ -0,0 +1,254 @@ +"""Create and populate a regional RunPod network volume for H3 benchmarks.""" + +import argparse +import json +import os +import subprocess +import sys +import tempfile +import time +import urllib.error +import urllib.request +from datetime import datetime, timezone +from pathlib import Path + + +API_V1 = "https://rest.runpod.io/v1" +API_V2 = "https://api.runpod.io/v2" +CPU_IMAGE = "runpod/base:0.7.0-ubuntu2004" +MODEL_REPO = "coolthor/MiniMax-H3-pruned-NVFP4" +MODEL_REVISION = "fb06c2af47899f086a33bd599e91084b9f95bc54" +MODEL_REPO_PATH = "diffusion_models/minimax_h3_fl2va_pruned_nvfp4.safetensors" +MODEL_VOLUME_PATH = f"ComfyUI/models/{MODEL_REPO_PATH}" +MODEL_SHA256 = "9d49beb65ddc373a0df523b8ce715b61b73f88127bf3c8d3ffda76c5dda02bd4" +TEXT_REPO = "Comfy-Org/MiniMax-H3" +TEXT_REVISION = "0f7fb980293fcc4d55c1158cbda920806682ed5d" +TEXT_REPO_PATH = "text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors" +TEXT_VOLUME_PATH = f"ComfyUI/models/{TEXT_REPO_PATH}" +TEXT_SHA256 = "35a88d51044231fe332301d7a62aa81e3f2cba62febeb446e2c1e3e0ef76f2c6" + + +def request(api_key: str, method: str, url: str, body=None): + data = None if body is None else json.dumps(body).encode("utf-8") + call = urllib.request.Request(url, data=data, method=method) + call.add_header("User-Agent", "h3-blackwell-runtime/0.1") + call.add_header("Authorization", f"Bearer {api_key}") + call.add_header("Accept", "application/json") + if data is not None: + call.add_header("Content-Type", "application/json") + try: + with urllib.request.urlopen(call, timeout=60) as response: + payload = response.read() + return json.loads(payload) if payload else {"status": response.status} + except urllib.error.HTTPError as error: + detail = error.read().decode("utf-8", errors="replace") + raise RuntimeError(f"RunPod API returned HTTP {error.code}: {detail}") from error + + +def run(command: list[str], *, cwd: Path | None = None, input_text: str | None = None) -> None: + if input_text is None: + subprocess.run(command, cwd=cwd, text=True, check=True) + else: + # Preserve Unix LF endings when a Windows client streams a script to bash. + subprocess.run(command, cwd=cwd, input=input_text.encode("utf-8"), check=True) + + +def capture(command: list[str], *, cwd: Path) -> str: + return subprocess.check_output(command, cwd=cwd, text=True).strip() + + +def wait_for_ssh(api_key: str, pod_id: str, timeout_seconds: int) -> dict: + deadline = time.monotonic() + timeout_seconds + last_status = None + while time.monotonic() < deadline: + pod = request(api_key, "GET", f"{API_V2}/pods/{pod_id}") + status = pod.get("status") + if status != last_status: + print(f"pod {pod_id}: {status}", flush=True) + last_status = status + direct = (pod.get("ssh") or {}).get("direct") + if status == "RUNNING" and direct: + return direct + if status in {"ERROR", "TERMINATED"}: + raise RuntimeError(f"pod {pod_id} entered terminal status {status}") + time.sleep(5) + raise TimeoutError(f"pod {pod_id} did not expose direct SSH within {timeout_seconds}s") + + +def ssh_base(endpoint: dict, key: Path) -> list[str]: + return [ + "ssh", + "-o", "BatchMode=yes", + "-o", "StrictHostKeyChecking=accept-new", + "-o", "ServerAliveInterval=15", + "-o", "ServerAliveCountMax=20", + "-p", str(endpoint["port"]), + "-i", str(key), + f'{endpoint["username"]}@{endpoint["host"]}', + ] + + +def remote_download_script(hf_token: str) -> str: + def download(repo: str, revision: str, repo_path: str, volume_path: str, checksum: str) -> str: + url = f"https://huggingface.co/{repo}/resolve/{revision}/{repo_path}?download=true" + target = f"/runpod-volume/{volume_path}" + directory, filename = target.rsplit("/", 1) + return f""" +mkdir -p '{directory}' +if echo '{checksum} {target}' | sha256sum --check --status; then + echo 'READY {volume_path}' +else + if [[ -f '{target}' && ! -f '{target}.aria2' ]]; then rm -f '{target}'; fi + echo 'DOWNLOAD {volume_path}' + aria2c --header="Authorization: Bearer $HF_TOKEN" \ + --dir='{directory}' --out='{filename}' --continue=true \ + --allow-overwrite=true --auto-file-renaming=false \ + --max-connection-per-server=16 --split=16 --min-split-size=16M \ + --file-allocation=none --max-tries=20 --retry-wait=5 --timeout=60 \ + --console-log-level=warn --summary-interval=10 '{url}' + echo '{checksum} {target}' | sha256sum --check +fi +""" + + return f"""#!/usr/bin/env bash +set -Eeuo pipefail +export HF_TOKEN={json.dumps(hf_token)} +echo "$(date -u +%FT%TZ) BOOTSTRAP_START" +apt-get update -qq +apt-get install -y -qq aria2 ca-certificates +{download(MODEL_REPO, MODEL_REVISION, MODEL_REPO_PATH, MODEL_VOLUME_PATH, MODEL_SHA256)} +{download(TEXT_REPO, TEXT_REVISION, TEXT_REPO_PATH, TEXT_VOLUME_PATH, TEXT_SHA256)} +mkdir -p /runpod-volume/h3-runtime +touch /runpod-volume/h3-runtime/artifacts-ready +echo "$(date -u +%FT%TZ) BOOTSTRAP_COMPLETE" +""" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--region", required=True, help="RunPod data center ID, for example US-MO-2") + parser.add_argument("--volume-id", help="Reuse and repair an existing volume instead of creating one") + parser.add_argument("--volume-name", default="h3-vortex-benchmark") + parser.add_argument("--volume-size", type=int, default=100) + parser.add_argument("--cpu-flavor", default="cpu3g") + parser.add_argument("--vcpu", type=int, default=2) + parser.add_argument("--ssh-key", type=Path, default=Path.home() / ".ssh" / "inceptal-dev-envs") + parser.add_argument("--ssh-timeout", type=int, default=300) + parser.add_argument("--keep-pod", action="store_true", help="Do not terminate the CPU pod after bootstrapping") + parser.add_argument("--dry-run", action="store_true", help="Print the plan without creating billable resources") + parser.add_argument("--yes", action="store_true", help="Confirm network-volume and CPU-pod charges") + args = parser.parse_args() + + repo_root = Path(__file__).resolve().parents[1] + commit = capture(["git", "rev-parse", "HEAD"], cwd=repo_root) + dirty = bool(capture(["git", "status", "--porcelain"], cwd=repo_root)) + archive_name = f"h3-blackwell-runtime-{commit[:12]}.tar.gz" + plan = { + "region": args.region, + "volume_id": args.volume_id, + "volume_name": args.volume_name, + "volume_size_gb": args.volume_size, + "cpu": {"id": args.cpu_flavor, "vcpu_count": args.vcpu}, + "cpu_image": CPU_IMAGE, + "git_commit": commit, + "git_worktree_dirty": dirty, + "archive": f"h3-runtime/{archive_name}", + "artifacts": { + MODEL_VOLUME_PATH: MODEL_SHA256, + TEXT_VOLUME_PATH: TEXT_SHA256, + }, + } + if args.dry_run: + print(json.dumps(plan, indent=2)) + return 0 + if not args.yes: + parser.error("this creates billable storage and CPU compute; repeat with --yes after reviewing --dry-run") + + api_key = os.environ.get("RUNPOD_API_KEY") + hf_token = os.environ.get("HF_TOKEN") + if not api_key: + parser.error("RUNPOD_API_KEY is required") + if not hf_token: + parser.error("HF_TOKEN is required for the gated model repository") + if not args.ssh_key.is_file(): + parser.error(f"SSH private key not found: {args.ssh_key}") + + volume_id = args.volume_id + if volume_id: + volume = request(api_key, "GET", f"{API_V1}/networkvolumes/{volume_id}") + if volume["dataCenterId"] != args.region: + raise RuntimeError( + f"volume {volume_id} is in {volume['dataCenterId']}, not requested region {args.region}" + ) + print(f"reusing volume {volume_id} in {args.region}", flush=True) + else: + volume = request(api_key, "POST", f"{API_V1}/networkvolumes", { + "name": args.volume_name, + "size": args.volume_size, + "dataCenterId": args.region, + }) + volume_id = volume["id"] + print(f"created volume {volume_id} in {args.region}", flush=True) + + pod_id = None + try: + pod = request(api_key, "POST", f"{API_V2}/pods", { + "name": f"h3-volume-bootstrap-{args.region.lower()}", + "image": CPU_IMAGE, + "cpu": {"id": args.cpu_flavor, "vcpuCount": args.vcpu}, + "cloud": "SECURE", + "dataCenterIds": [args.region], + "disk": 5, + "ports": ["22/tcp"], + "mounts": {"network": [{"volumeId": volume_id, "path": "/runpod-volume"}]}, + "startSsh": True, + }) + pod_id = pod["id"] + print(f"created CPU pod {pod_id} at ${pod['cost']:.2f}/hour", flush=True) + endpoint = wait_for_ssh(api_key, pod_id, args.ssh_timeout) + ssh = ssh_base(endpoint, args.ssh_key) + + with tempfile.TemporaryDirectory() as temporary_directory: + temporary = Path(temporary_directory) + archive = temporary / archive_name + run(["git", "archive", "--format=tar.gz", f"--output={archive}", "HEAD"], cwd=repo_root) + run(ssh + ["mkdir", "-p", "/runpod-volume/h3-runtime"]) + scp = [ + "scp", "-q", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=accept-new", + "-P", str(endpoint["port"]), "-i", str(args.ssh_key), str(archive), + f'{endpoint["username"]}@{endpoint["host"]}:/runpod-volume/h3-runtime/{archive_name}', + ] + run(scp) + print(f"uploaded application archive {archive_name}", flush=True) + + print("starting pinned model downloads; progress follows", flush=True) + run(ssh + ["bash", "-s"], input_text=remote_download_script(hf_token)) + + manifest = dict(plan) + manifest.update({ + "volume_id": volume_id, + "prepared_at": datetime.now(timezone.utc).isoformat(), + "cpu_pod_id": pod_id, + }) + manifest_path = temporary / "regional-bootstrap-manifest.json" + manifest_path.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8") + scp[-2] = str(manifest_path) + scp[-1] = ( + f'{endpoint["username"]}@{endpoint["host"]}:' + "/runpod-volume/h3-runtime/regional-bootstrap-manifest.json" + ) + run(scp) + + print(json.dumps({"status": "ready", "region": args.region, "volume_id": volume_id}, indent=2)) + return 0 + finally: + if pod_id and not args.keep_pod: + try: + request(api_key, "POST", f"{API_V2}/pods/{pod_id}/action", {"action": "terminate"}) + print(f"terminated CPU pod {pod_id}", flush=True) + except Exception as error: + print(f"WARNING: failed to terminate CPU pod {pod_id}: {error}", file=sys.stderr) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/profile_attention_components.py b/tools/profile_attention_components.py new file mode 100644 index 0000000..7b941a1 --- /dev/null +++ b/tools/profile_attention_components.py @@ -0,0 +1,216 @@ +"""Attribute Sage2 and Sol-Attn latency on representative real H3 tensors.""" + +from __future__ import annotations + +import argparse +import json +import time +from contextlib import contextmanager +from pathlib import Path + +import torch + +from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize, sync + + +class StageHooks: + def __init__(self): + self.current: dict[str, float] = {} + self.originals: list[tuple[object, str, object]] = [] + + def patch(self, owner: object, attribute: str, stage: str) -> None: + original = getattr(owner, attribute) + + def wrapped(*args, **kwargs): + sync() + started = time.perf_counter() + result = original(*args, **kwargs) + sync() + self.current[stage] = self.current.get(stage, 0.0) + time.perf_counter() - started + return result + + self.originals.append((owner, attribute, original)) + setattr(owner, attribute, wrapped) + + def restore(self) -> None: + for owner, attribute, original in reversed(self.originals): + setattr(owner, attribute, original) + self.originals.clear() + + +@contextmanager +def installed_hooks(configure): + hooks = StageHooks() + try: + configure(hooks) + yield hooks + finally: + hooks.restore() + + +def measure(fn, configure_hooks, *, warmup: int, iterations: int) -> tuple[dict, torch.Tensor]: + with installed_hooks(configure_hooks) as hooks, torch.inference_mode(): + for _ in range(warmup): + hooks.current = {} + fn() + rows = [] + output = None + for _ in range(iterations): + hooks.current = {} + sync() + started = time.perf_counter() + output = fn() + sync() + row = dict(hooks.current) + row["total"] = time.perf_counter() - started + rows.append(row) + names = sorted({name for row in rows for name in row}) + return {name: summarize([row.get(name, 0.0) for row in rows]) for name in names}, output + + +def add_residual(timings: dict, total: str, children: tuple[str, ...], name: str) -> None: + values = [] + count = timings[total]["count"] + # The summary alone cannot reconstruct paired iterations. This residual is + # therefore an explicitly labeled median estimate, not a distribution. + estimate = timings[total]["p50_s"] - sum(timings[child]["p50_s"] for child in children if child in timings) + values.extend([max(estimate, 0.0)] * count) + timings[name] = summarize(values) + + +def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict[str, float]: + delta = actual.float() - expected.float() + return { + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "relative_l2": (delta.norm() / expected.float().norm().clamp_min(1e-12)).item(), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--sol-tau", type=float, default=1.3) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--attention", default="sage2", choices=("sage2",)) + parser.add_argument("--sage-only", action="store_true") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + block, hidden, rotation, segments, metadata = representative_attention_inputs(args) + q, k, v, _ = prepare_qkv(block, hidden, rotation, None) + sequence = q.shape[1] + conditioning_stop = segments[-1][0] + sink_blocks = (0, (conditioning_stop + 63) // 64) + + import sageattention.core as sage_core + + def configure_sage(hooks: StageHooks) -> None: + hooks.patch(sage_core, "per_warp_int8_cuda", "qk_quantize") + hooks.patch(sage_core, "per_channel_fp8", "v_quantize") + hooks.patch( + sage_core.sm89_compile, + "qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf", + "attention_forward", + ) + + sage_timings, sage_output = measure( + lambda: sage_core.sageattn( + q, k, v, tensor_layout="NHD", is_causal=False, smooth_k=False, + ), + configure_sage, + warmup=args.warmup, + iterations=args.iterations, + ) + add_residual( + sage_timings, + "total", + ("qk_quantize", "v_quantize", "attention_forward"), + "framework_and_k_smoothing_estimate", + ) + sol_results = [] + if not args.sage_only: + import sol_kernel.fwd as sol_fwd + import sol_kernel.preprocess as sol_preprocess + import sol_kernel.quant as sol_quant + + def configure_sol(hooks: StageHooks) -> None: + hooks.patch(sol_preprocess, "reduce_quantize_k", "k_summary_and_optional_quantize") + hooks.patch(sol_preprocess, "_reduce_v", "v_summary") + hooks.patch(sol_preprocess, "_compute_diag_threshold", "routing_threshold") + hooks.patch(sol_preprocess, "quantize_q_with_threshold", "q_quantize_and_threshold") + hooks.patch(sol_quant, "quantize_v_per_channel", "v_int8_quantize") + hooks.patch(sol_fwd, "prepare", "prepare_total") + + for int8_qk, int8_pv in ((False, False), (True, False), (True, True)): + sol_timings, sol_output = measure( + lambda int8_qk=int8_qk, int8_pv=int8_pv: sol_fwd.sol_attn( + q, + k, + v, + tau=args.sol_tau, + thresh_type="diag", + int8_qk=int8_qk, + int8_pv=int8_pv, + sink_blocks=sink_blocks, + ), + configure_sol, + warmup=args.warmup, + iterations=args.iterations, + ) + add_residual(sol_timings, "total", ("prepare_total",), "routed_forward_estimate") + prepare_children = ( + "k_summary_and_optional_quantize", + "v_summary", + "routing_threshold", + "q_quantize_and_threshold", + "v_int8_quantize", + ) + add_residual(sol_timings, "prepare_total", prepare_children, "prepare_other_estimate") + sol_results.append({ + "int8_qk": int8_qk, + "int8_pv": int8_pv, + "tau": args.sol_tau, + "sink_blocks": list(sink_blocks), + "timings": sol_timings, + "difference_vs_sage2": difference(sol_output, sage_output), + }) + + report = { + "metadata": metadata, + "sequence": sequence, + "conditioning_stop": conditioning_stop, + "warmup": args.warmup, + "iterations": args.iterations, + "measurement_policy": { + "component_timings": "CUDA-synchronized nested probes; attribution only", + "residual_timings": "difference of component p50 values; labeled estimates", + "end_to_end": "must be measured separately without component synchronization", + }, + "sage2": { + "version": "2.2.0", + "timings": sage_timings, + "checksum": sage_output.float().sum().item(), + }, + "sol": sol_results, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/profile_attention_path.py b/tools/profile_attention_path.py index 055071d..ab74148 100644 --- a/tools/profile_attention_path.py +++ b/tools/profile_attention_path.py @@ -143,11 +143,38 @@ def run_sol_fused_path(qkv: torch.Tensor, heads: int, head_dim: int, stats: dict return timed(stats, "output_reshape", lambda: out.reshape(sequence, inner).contiguous()) if stats is not None else out.reshape(sequence, inner).contiguous() +def run_sage_nhd_path( + q_src: torch.Tensor, + k_src: torch.Tensor, + v_src: torch.Tensor, + *, + materialize: bool, + stats: dict[str, list[float]] | None = None, +): + from sageattention import sageattn + + sequence = q_src.shape[1] + inner = q_src.shape[2] * q_src.shape[3] + if materialize: + q = timed(stats, "q_nhd_contiguous", q_src.contiguous) if stats is not None else q_src.contiguous() + k = timed(stats, "k_nhd_contiguous", k_src.contiguous) if stats is not None else k_src.contiguous() + v = timed(stats, "v_nhd_contiguous", v_src.contiguous) if stats is not None else v_src.contiguous() + else: + q, k, v = q_src, k_src, v_src + run = lambda: sageattn(q, k, v, tensor_layout="NHD", is_causal=False, smooth_k=False) + out = timed(stats, "attention_kernel", run) if stats is not None else run() + return timed(stats, "output_reshape", lambda: out.reshape(sequence, inner)) if stats is not None else out.reshape(sequence, inner) + + def layout_timing_names(layout_mode: str) -> tuple[str, ...]: if layout_mode == "sol_fused": return ("qkv_to_bshd", "attention_kernel", "output_reshape") if layout_mode == "sol_native": return ("q_bshd_contiguous", "k_bshd_contiguous", "v_bshd_contiguous", "attention_kernel", "output_reshape") + if layout_mode == "sage_nhd": + return ("q_nhd_contiguous", "k_nhd_contiguous", "v_nhd_contiguous", "attention_kernel", "output_reshape") + if layout_mode == "sage_strided_nhd": + return ("attention_kernel", "output_reshape") return ("q_transpose_contiguous", "k_transpose_contiguous", "v_transpose_contiguous", "attention_kernel", "output_reshape") @@ -166,6 +193,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sage2", help="Backend used only while building representative upstream tensors.") parser.add_argument("--backends", nargs="+", choices=AVAILABLE_BACKENDS, default=("sol_attn", "sage2", "sage3", "sage3_mean", "kj_sage_fp8", "kj_sage_fp8pp", "sdpa")) parser.add_argument("--sol-layout", choices=("hnd", "native", "fused", "both", "all"), default="hnd", help="Compare generic HND Sol path with direct BSHD and fused QKV layout paths.") + parser.add_argument("--sage-layout", choices=("hnd", "nhd", "strided_nhd", "all"), default="hnd", help="Compare Sage HND with contiguous or projection-strided NHD inputs.") parser.add_argument("--segments", nargs="+", choices=("all", "text", "secondary", "video"), default=("all",)) parser.add_argument("--warmup", type=int, default=20) parser.add_argument("--iterations", type=int, default=100) @@ -193,6 +221,12 @@ def main() -> None: "both": ["hnd", "sol_native"], "all": ["hnd", "sol_native", "sol_fused"], }[args.sol_layout] + elif backend == "sage2" and args.sage_layout != "hnd": + layout_modes = { + "nhd": ["sage_nhd"], + "strided_nhd": ["sage_strided_nhd"], + "all": ["hnd", "sage_nhd", "sage_strided_nhd"], + }[args.sage_layout] for layout_mode in layout_modes: try: if layout_mode == "sol_fused" and qkv_src is None: @@ -202,6 +236,10 @@ def main() -> None: run_sol_fused_path(qkv_src, block.attention.heads, block.attention.head_dim) elif layout_mode == "sol_native": run_sol_native_path(q_src, k_src, v_src) + elif layout_mode == "sage_nhd": + run_sage_nhd_path(q_src, k_src, v_src, materialize=True) + elif layout_mode == "sage_strided_nhd": + run_sage_nhd_path(q_src, k_src, v_src, materialize=False) else: run_path(q_src, k_src, v_src, backend) stats: dict[str, list[float]] = {} @@ -211,6 +249,10 @@ def main() -> None: output = run_sol_fused_path(qkv_src, block.attention.heads, block.attention.head_dim, stats) elif layout_mode == "sol_native": output = run_sol_native_path(q_src, k_src, v_src, stats) + elif layout_mode == "sage_nhd": + output = run_sage_nhd_path(q_src, k_src, v_src, materialize=True, stats=stats) + elif layout_mode == "sage_strided_nhd": + output = run_sage_nhd_path(q_src, k_src, v_src, materialize=False, stats=stats) else: output = run_path(q_src, k_src, v_src, backend, stats) if reference is None: diff --git a/tools/profile_h3_block.py b/tools/profile_h3_block.py index f004e79..ee5994e 100644 --- a/tools/profile_h3_block.py +++ b/tools/profile_h3_block.py @@ -4,6 +4,7 @@ from __future__ import annotations import argparse import json +import os import time import warnings from pathlib import Path @@ -12,9 +13,10 @@ warnings.filterwarnings("ignore", message="Found GPU0 NVIDIA GB10 which is of cu import torch import torch.nn.functional as functional +from torch.profiler import ProfilerActivity, profile from h3_blackwell_runtime.adaln import H3CurveAdaLN -from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, rms_rope_split_half_, rms_norm, run_attention, run_sol_attention_bshd +from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS, rms_rope_split_half_, rms_norm, run_attention, run_sage_attention_nhd, run_sol_attention_bshd from h3_blackwell_runtime.block import H3DiTBlock, gate_segments, modulate_segments from h3_blackwell_runtime.checkpoint import H3Checkpoint from h3_blackwell_runtime.nvfp4 import Nvfp4Linear @@ -71,6 +73,41 @@ def summarize(values: list[float]) -> dict[str, float]: } +def profile_module_kernels(fn, *, warmup: int, iterations: int, row_limit: int) -> tuple[list[dict], dict]: + with torch.inference_mode(): + for _ in range(warmup): + fn() + sync() + activities = [ProfilerActivity.CPU] + if torch.cuda.is_available(): + activities.append(ProfilerActivity.CUDA) + with profile(activities=activities, record_shapes=True, profile_memory=True) as prof: + for _ in range(iterations): + fn() + sync() + + rows = [] + for event in prof.key_averages(group_by_input_shape=True): + rows.append({ + "key": event.key, + "count": int(event.count), + "device_type": str(getattr(event, "device_type", "")), + "cpu_time_total_us": float(getattr(event, "cpu_time_total", 0.0) or 0.0), + "device_time_total_us": float(getattr(event, "device_time_total", 0.0) or 0.0), + "self_device_time_total_us": float(getattr(event, "self_device_time_total", 0.0) or 0.0), + "self_device_memory_usage_bytes": int(getattr(event, "self_device_memory_usage", 0) or 0), + "input_shapes": str(getattr(event, "input_shapes", "")), + }) + rows.sort(key=lambda row: (row["device_time_total_us"], row["cpu_time_total_us"]), reverse=True) + launches = sum(row["count"] for row in rows if row["key"] in {"cudaLaunchKernel", "cuLaunchKernelEx"}) + return rows[:row_limit], { + "profiled_iterations": iterations, + "runtime_kernel_launches": launches, + "runtime_kernel_launches_per_block": launches / iterations, + "positive_self_device_allocated_bytes": sum(max(row["self_device_memory_usage_bytes"], 0) for row in rows), + } + + def profiled_nvfp4_linear(stats: dict[str, list[float]], prefix: str, module: Nvfp4Linear, x: torch.Tensor) -> torch.Tensor: from comfy_kitchen.tensor import QuantizedTensor, TensorCoreNVFP4Layout @@ -145,7 +182,15 @@ def profile_block( qkv = timed(stats, "attn_qkv_proj", lambda: profiled_nvfp4_linear(stats, "linear.attn_qkv_proj", attention.qkv_proj, h)) q, k, v = timed(stats, "attn_qkv_split_view", lambda: tuple(t.view(1, sequence, attention.heads, attention.head_dim) for t in qkv.split(inner, dim=-1))) q, k = timed(stats, "attn_qk_rms_rope", lambda: rms_rope_split_half_(q, k, rotation, attention.q_norm_weight, attention.k_norm_weight, attention.eps)) - if attention.backend == "sol_attn": + if ( + attention.backend == "sage2" + and os.getenv("H3_SAGE_QKV_LAYOUT", "hnd").lower() == "strided_nhd" + ): + attn_out = timed(stats, "attention_kernel", lambda: run_sage_attention_nhd(q, k, v)) + if not attn_out.is_contiguous(): + raise RuntimeError("Sage2 NHD output must be contiguous for zero-copy output projection") + attn_rows = timed(stats, "attn_output_reshape", lambda: attn_out.reshape(sequence, inner)) + elif attention.backend == "sol_attn": q = timed(stats, "attn_q_bshd_contiguous", lambda: q.contiguous()) k = timed(stats, "attn_k_bshd_contiguous", lambda: k.contiguous()) v = timed(stats, "attn_v_bshd_contiguous", lambda: v.contiguous()) @@ -185,10 +230,14 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--seed", type=int, default=440407) parser.add_argument("--text-tokens", type=int, default=93, help="Synthetic refined-text token count; avoids Qwen/refiner load.") parser.add_argument("--block-index", type=int, default=24) - parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sol_attn") + parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, default="sage2") parser.add_argument("--warmup", type=int, default=20) parser.add_argument("--iterations", type=int, default=50) + parser.add_argument("--profiler-iterations", type=int, default=2) + parser.add_argument("--profiler-row-limit", type=int, default=100) + parser.add_argument("--cuda-profiler-capture", action="store_true", help="Capture exactly one warmed module forward with cudaProfilerStart/Stop.") parser.add_argument("--device", default="cuda") + parser.add_argument("--quiet", action="store_true") return parser.parse_args() @@ -215,9 +264,49 @@ def main() -> None: adaln_values = tuple(value.detach() for value in adaln(timesteps)) with torch.inference_mode(): + if args.cuda_profiler_capture: + for _ in range(args.warmup): + block(hidden, rotation, *adaln_values, segments) + sync() + torch.cuda.cudart().cudaProfilerStart() + module_output = block(hidden, rotation, *adaln_values, segments) + sync() + torch.cuda.cudart().cudaProfilerStop() + result = { + "block_index": args.block_index, + "hidden_shape": list(hidden.shape), + "segments": segments, + "fused_elementwise": block.fused_elementwise, + "fused_nvfp4_modulation": block.fused_nvfp4_modulation, + "fused_nvfp4_swiglu": block.mlp.fused_nvfp4_swiglu, + "nvfp4_scale_backend": os.getenv("H3_NVFP4_SCALE_BACKEND"), + "sage_qkv_layout": os.getenv("H3_SAGE_QKV_LAYOUT"), + "module_forward_checksum": module_output.float().sum().item(), + "capture": "one warmed block between cudaProfilerStart/Stop", + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(result, indent=2), encoding="utf-8") + if not args.quiet: + print(json.dumps(result, indent=2), flush=True) + return for _ in range(args.warmup): profile_block(block, hidden, rotation, adaln_values, segments, iterations=1) stats, output = profile_block(block, hidden, rotation, adaln_values, segments, iterations=args.iterations) + for _ in range(args.warmup): + block(hidden, rotation, *adaln_values, segments) + module_forward = [] + for _ in range(args.iterations): + sync() + started = time.perf_counter() + module_output = block(hidden, rotation, *adaln_values, segments) + sync() + module_forward.append(time.perf_counter() - started) + profiler_events, profiler_summary = profile_module_kernels( + lambda: block(hidden, rotation, *adaln_values, segments), + warmup=args.warmup, + iterations=args.profiler_iterations, + row_limit=args.profiler_row_limit, + ) result = { "prompt": CAT_PROMPT, @@ -237,10 +326,16 @@ def main() -> None: "output_shape": list(output.shape), "segments": segments, "timings": stats, + "module_forward": summarize(module_forward), + "module_forward_checksum": module_output.float().sum().item(), + "fused_elementwise": block.fused_elementwise, + "profiler_summary": profiler_summary, + "profiler_top_events": profiler_events, } args.output.parent.mkdir(parents=True, exist_ok=True) args.output.write_text(json.dumps(result, indent=2), encoding="utf-8") - print(json.dumps(result, indent=2), flush=True) + if not args.quiet: + print(json.dumps(result, indent=2), flush=True) if __name__ == "__main__": diff --git a/tools/profile_hybrid_attention.py b/tools/profile_hybrid_attention.py new file mode 100644 index 0000000..e895f33 --- /dev/null +++ b/tools/profile_hybrid_attention.py @@ -0,0 +1,160 @@ +"""Profile H3 Sage2 against Sol routing with exact conditioning sinks.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.attention import run_attention +from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize, sync + + +def timed_iterations(fn, warmup: int, iterations: int) -> tuple[dict[str, float], torch.Tensor]: + with torch.inference_mode(): + for _ in range(warmup): + fn() + values = [] + output = None + for _ in range(iterations): + sync() + started = time.perf_counter() + output = fn() + sync() + values.append(time.perf_counter() - started) + return summarize(values), output + + +def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict[str, float]: + delta = actual.float() - expected.float() + expected_float = expected.float() + return { + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "relative_l2": (delta.norm() / expected_float.norm().clamp_min(1e-12)).item(), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--attention", default="sage2", choices=("sage2",)) + parser.add_argument("--taus", nargs="+", type=float, default=(0.0, 0.4, 0.8, 1.0, 1.3)) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=5) + parser.add_argument("--device", default="cuda") + parser.add_argument("--int8-qk", action="store_true") + parser.add_argument("--int8-pv", action="store_true") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + if args.int8_pv and not args.int8_qk: + raise ValueError("--int8-pv requires --int8-qk") + block, hidden, rotation, segments, metadata = representative_attention_inputs(args) + q, k, v, _ = prepare_qkv(block, hidden, rotation, None) + sequence = q.shape[1] + + sage_timing, sage_output_hnd = timed_iterations( + lambda: run_attention( + q.transpose(1, 2).contiguous(), + k.transpose(1, 2).contiguous(), + v.transpose(1, 2).contiguous(), + backend="sage2", + is_causal=False, + ), + args.warmup, + args.iterations, + ) + sage_output = sage_output_hnd.transpose(1, 2).contiguous() + + try: + from sol_kernel import sol_attn + except ImportError as error: + raise RuntimeError("Sol-Attn must be installed to profile the hybrid policy") from error + + conditioning_stop = segments[-1][0] + conditioning_blocks = (conditioning_stop + 63) // 64 + sink_modes = { + "off": ((0, 0), (0, 0)), + "exact_kv": ((0, conditioning_blocks), (0, 0)), + "exact_kv_and_rows": ((0, conditioning_blocks), (0, conditioning_blocks)), + } + spans = { + "conditioning": (0, conditioning_stop), + "video": (conditioning_stop, sequence), + "all": (0, sequence), + } + + results = [] + for sink_name, (sink_blocks, sink_q) in sink_modes.items(): + for tau in args.taus: + timing, output = timed_iterations( + lambda tau=tau, sink_blocks=sink_blocks, sink_q=sink_q: sol_attn( + q, + k, + v, + tau=tau, + thresh_type="diag", + int8_qk=args.int8_qk, + int8_pv=args.int8_pv, + sink_blocks=sink_blocks, + sink_q=sink_q, + ), + args.warmup, + args.iterations, + ) + results.append({ + "tau": tau, + "sink": sink_name, + "sink_blocks": list(sink_blocks), + "sink_q": list(sink_q), + "timing": timing, + "speedup_vs_sage2_p50": sage_timing["p50_s"] / timing["p50_s"], + "difference_vs_sage2": { + name: difference(output[:, start:stop], sage_output[:, start:stop]) + for name, (start, stop) in spans.items() + }, + }) + print( + sink_name, + f"tau={tau:.2f}", + f"p50_ms={timing['p50_s'] * 1000:.3f}", + f"speedup={sage_timing['p50_s'] / timing['p50_s']:.3f}x", + f"rel_l2={results[-1]['difference_vs_sage2']['all']['relative_l2']:.6f}", + flush=True, + ) + + report = { + "metadata": metadata, + "conditioning_stop": conditioning_stop, + "conditioning_blocks": conditioning_blocks, + "sage2": { + "timing": sage_timing, + "checksum": sage_output.float().sum().item(), + }, + "warmup": args.warmup, + "iterations": args.iterations, + "int8_qk": args.int8_qk, + "int8_pv": args.int8_pv, + "results": results, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/tools/profile_nvfp4_linear.py b/tools/profile_nvfp4_linear.py index 94a6a8f..2c28bfd 100644 --- a/tools/profile_nvfp4_linear.py +++ b/tools/profile_nvfp4_linear.py @@ -118,7 +118,16 @@ def run_linear_with_quantizer(module: Nvfp4Linear, x: torch.Tensor, quantizer: s return output[:flat_x.shape[0], :module.out_features].reshape(*original_shape, module.out_features) -def profile_cuda_kernels(module: Nvfp4Linear, x: torch.Tensor, *, warmup: int, iterations: int, row_limit: int, quantizer: str) -> list[dict]: +def profile_cuda_kernels( + module: Nvfp4Linear, + x: torch.Tensor, + *, + warmup: int, + iterations: int, + row_limit: int, + quantizer: str, + trace_path: Path | None, +) -> tuple[list[dict], dict]: with torch.inference_mode(): precomputed_scale = nvfp4_activation_scale(x.reshape(-1, module.in_features).contiguous()) if quantizer == "vortex_precomputed_scale" else None for _ in range(warmup): @@ -127,25 +136,81 @@ def profile_cuda_kernels(module: Nvfp4Linear, x: torch.Tensor, *, warmup: int, i activities = [ProfilerActivity.CPU] if torch.cuda.is_available(): activities.append(ProfilerActivity.CUDA) - with profile(activities=activities, record_shapes=True) as prof: + with profile(activities=activities, record_shapes=True, profile_memory=True, with_flops=True) as prof: for _ in range(iterations): run_linear_with_quantizer(module, x, quantizer, precomputed_scale) sync() + if trace_path is not None: + trace_path.parent.mkdir(parents=True, exist_ok=True) + prof.export_chrome_trace(str(trace_path)) rows = [] for event in prof.key_averages(group_by_input_shape=True): cpu_us = float(getattr(event, "cpu_time_total", 0.0) or 0.0) - cuda_us = float(getattr(event, "cuda_time_total", 0.0) or 0.0) + device_us = float(getattr(event, "device_time_total", 0.0) or 0.0) rows.append( { "key": event.key, "count": int(event.count), + "device_type": str(getattr(event, "device_type", "")), "cpu_time_total_us": cpu_us, - "cuda_time_total_us": cuda_us, + "self_cpu_time_total_us": float(getattr(event, "self_cpu_time_total", 0.0) or 0.0), + "device_time_total_us": device_us, + "self_device_time_total_us": float(getattr(event, "self_device_time_total", 0.0) or 0.0), + "cpu_memory_usage_bytes": int(getattr(event, "cpu_memory_usage", 0) or 0), + "self_cpu_memory_usage_bytes": int(getattr(event, "self_cpu_memory_usage", 0) or 0), + "device_memory_usage_bytes": int(getattr(event, "device_memory_usage", 0) or 0), + "self_device_memory_usage_bytes": int(getattr(event, "self_device_memory_usage", 0) or 0), + "flops": int(getattr(event, "flops", 0) or 0), "input_shapes": str(getattr(event, "input_shapes", "")), } ) - rows.sort(key=lambda item: (item["cuda_time_total_us"], item["cpu_time_total_us"]), reverse=True) - return rows[:row_limit] + rows.sort(key=lambda item: (item["device_time_total_us"], item["cpu_time_total_us"]), reverse=True) + device_rows = [row for row in rows if "CUDA" in row["device_type"]] + launch_rows = [row for row in rows if row["key"] in {"cudaLaunchKernel", "cuLaunchKernelEx"}] + summary = { + "device_kernel_launches": sum(row["count"] for row in device_rows), + "runtime_kernel_launches": sum(row["count"] for row in launch_rows), + "unique_device_kernels": len(device_rows), + "positive_self_device_allocated_bytes": sum( + max(row["self_device_memory_usage_bytes"], 0) for row in rows + ), + "negative_self_device_allocated_bytes": sum( + min(row["self_device_memory_usage_bytes"], 0) for row in rows + ), + "profiled_iterations": iterations, + "trace_path": str(trace_path) if trace_path is not None else None, + "note": "Profiler memory values are aggregate event attribution, not peak-memory measurements.", + } + return rows[:row_limit], summary + + +def intermediate_traffic(name: str, module: Nvfp4Linear, x: torch.Tensor) -> dict: + """Report mandatory and currently materialized tensor sizes for one linear.""" + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + rows = x.numel() // module.in_features + padded_rows, padded_features = TensorCoreNVFP4Layout.get_padded_shape((rows, module.in_features)) + input_bytes = rows * module.in_features * x.element_size() + qdata_bytes = padded_rows * padded_features // 2 + # One E4M3 block scale covers 16 FP4 values; the physical layout may tile it. + scale_rows = ((padded_rows + 127) // 128) * 128 + block_scale_bytes = scale_rows * padded_features // 16 + output_bytes = rows * module.out_features * torch.empty((), dtype=module.output_dtype).element_size() + return { + "logical_rows": rows, + "padded_activation_shape": [padded_rows, padded_features], + "input_bytes": input_bytes, + "packed_qdata_bytes": qdata_bytes, + "block_scale_bytes": block_scale_bytes, + "output_bytes": output_bytes, + "separate_materialization_bytes": input_bytes + qdata_bytes + block_scale_bytes + output_bytes, + "epilogue_candidate": { + "attn_qkv_proj": "QKV views/layout and Q/K normalization/RoPE", + "attn_out_proj": "segmented BF16 residual gate/add", + "mlp_fc1": "BF16 split and SwiGLU input", + "mlp_fc2": "segmented BF16 residual gate/add", + }[name], + } def representative_inputs(args: argparse.Namespace) -> tuple[H3DiTBlock, dict[str, torch.Tensor], dict]: @@ -252,6 +317,7 @@ def parse_args() -> argparse.Namespace: parser.add_argument("--iterations", type=int, default=5) parser.add_argument("--profiler-iterations", type=int, default=2) parser.add_argument("--profiler-row-limit", type=int, default=30) + parser.add_argument("--trace-dir", type=Path, help="Optional directory for one Chrome trace per linear and quantizer.") parser.add_argument("--device", default="cuda") return parser.parse_args() @@ -272,7 +338,16 @@ def main() -> None: stage_timings = profile_linear_stages(module, x, iterations=args.iterations, quantizer=quantizer) candidate = run_linear_with_quantizer(module, x, quantizer, precomputed_scale) diff = (candidate.float() - reference.float()).abs() - kernels = profile_cuda_kernels(module, x, warmup=args.warmup, iterations=args.profiler_iterations, row_limit=args.profiler_row_limit, quantizer=quantizer) + trace_path = args.trace_dir / f"{name}-{quantizer}.json" if args.trace_dir is not None else None + kernels, profiler_summary = profile_cuda_kernels( + module, + x, + warmup=args.warmup, + iterations=args.profiler_iterations, + row_limit=args.profiler_row_limit, + quantizer=quantizer, + trace_path=trace_path, + ) results.append( { "name": name, @@ -280,6 +355,8 @@ def main() -> None: "module": module_info(module, x), "stage_timings": stage_timings, "reference_diff": {"max": diff.max().item(), "mean": diff.mean().item()}, + "intermediate_traffic": intermediate_traffic(name, module, x), + "profiler_summary": profiler_summary, "profiler_top_events": kernels, } ) diff --git a/tools/profile_sage2_p0_counters.sh b/tools/profile_sage2_p0_counters.sh new file mode 100644 index 0000000..fffc05e --- /dev/null +++ b/tools/profile_sage2_p0_counters.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +set -euo pipefail + +root=/home/daniel/aeon-spark-test/h3/h3-blackwell-runtime +builds=/home/daniel/aeon-spark-test/h3/sageattention-p0-builds +output=/home/daniel/StoryStudioAssets/H3-output/h3-blackwell-runtime/benchmarks +extension=_qattn_sm89.cpython-312-aarch64-linux-gnu.so +variants=(baseline cap240 cap224 cap208 cap192 cap176 cap170 cap168 scopes pad-v-smem) + +for variant in "${variants[@]}"; do + report="$output/gb10-sage2-p0-$variant-counters.ncu-rep" + if [[ -f "$report" ]]; then + continue + fi + docker compose -f "$root/compose.spark.yml" run --rm --no-deps \ + --cap-add SYS_ADMIN --cap-add SYS_PTRACE \ + -v "$root/tools/profile_sage2_scheduler.py:/opt/h3-blackwell-runtime/tools/profile_sage2_scheduler.py:ro" \ + -v "$builds/$variant/lib/sageattention/$extension:/opt/venv/lib/python3.12/site-packages/sageattention/$extension:ro" \ + h3-blackwell-runtime \ + /opt/nvidia/nsight-compute/2025.3.1/ncu \ + --target-processes all \ + --profile-from-start off \ + --section LaunchStats \ + --section Occupancy \ + --section SchedulerStats \ + --section WarpStateStats \ + --section MemoryWorkloadAnalysis \ + --section SpeedOfLight \ + --kernel-name regex:qk_int_sv_f8_attn_kernel \ + --launch-count 1 \ + --force-overwrite \ + -o "/output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-$variant-counters" \ + python tools/profile_sage2_scheduler.py \ + --output "/output/h3-blackwell-runtime/benchmarks/gb10-sage2-p0-$variant-capture.json" \ + --cuda-profiler-capture \ + --skip-tail-study +done diff --git a/tools/profile_sage2_register_variants.py b/tools/profile_sage2_register_variants.py new file mode 100644 index 0000000..23419c6 --- /dev/null +++ b/tools/profile_sage2_register_variants.py @@ -0,0 +1,193 @@ +"""Benchmark isolated Sage2 SM89 extension variants on one real block-24 input.""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.util +import json +import math +from pathlib import Path + +import torch + +from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize +from profile_sage2_scheduler import prepare_quantized, run_mainloop + + +EXPECTED_OUTPUT_SHA256 = "4c666c20f5f8f651158a2ced33ccff08f3bada07665c595b99008d171db30574" + + +def parse_variant(value: str) -> tuple[str, Path]: + try: + name, path = value.split("=", 1) + except ValueError as exc: + raise argparse.ArgumentTypeError("variant must be NAME=PATH") from exc + return name, Path(path) + + +def tensor_sha256(value: torch.Tensor) -> str: + host_bytes = value.detach().contiguous().view(torch.uint8).cpu().numpy() + return hashlib.sha256(memoryview(host_bytes)).hexdigest() + + +def load_variant(path: Path): + spec = importlib.util.spec_from_file_location("sageattention._qattn_sm89", path) + if spec is None or spec.loader is None: + raise RuntimeError(f"cannot load extension from {path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def measure(fn, *, warmup: int, iterations: int) -> list[float]: + for _ in range(warmup): + fn() + torch.cuda.synchronize() + values = [] + for _ in range(iterations): + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + fn() + end.record() + end.synchronize() + values.append(start.elapsed_time(end) / 1000.0) + return values + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--variant", type=parse_variant, action="append", required=True) + parser.add_argument("--rounds", type=int, default=3) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--attention", default="sage2", choices=("sage2",)) + parser.add_argument("--device", default="cuda") + args = parser.parse_args() + + block, hidden, rotation, _segments, metadata = representative_attention_inputs(args) + with torch.inference_mode(): + q, k, v, _ = prepare_qkv(block, hidden, rotation, None) + quantized = prepare_quantized(q, k, v) + _, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, reference_output = quantized + run_mainloop( + q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, reference_output, + ) + torch.cuda.synchronize() + reference_sha256 = tensor_sha256(reference_output) + if reference_sha256 != EXPECTED_OUTPUT_SHA256: + raise RuntimeError( + f"deployed reference mismatch: expected {EXPECTED_OUTPUT_SHA256}, got {reference_sha256}" + ) + + variants = [] + for name, path in args.variant: + extension = load_variant(path) + output = torch.empty_like(reference_output) + + def launch(extension=extension, output=output): + extension.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( + q_int8, + k_int8, + v_fp8, + output, + q_scale, + k_scale, + v_scale, + 0, + 0, + 2, + output.shape[-1] ** -0.5, + 0, + ) + return output + + variants.append({ + "name": name, + "path": str(path), + "output": output, + "launch": launch, + "samples": [], + "rounds": [], + }) + + for round_index in range(args.rounds): + round_samples = {variant["name"]: [] for variant in variants} + for variant in variants: + for _ in range(args.warmup): + variant["launch"]() + torch.cuda.synchronize() + for iteration in range(args.iterations): + offset = (round_index + iteration) % len(variants) + ordered = variants[offset:] + variants[:offset] + for variant in ordered: + samples = measure(variant["launch"], warmup=0, iterations=1) + variant["samples"].extend(samples) + round_samples[variant["name"]].extend(samples) + for variant in variants: + variant["rounds"].append(summarize(round_samples[variant["name"]])) + + results = [] + baseline_p50 = None + for variant in variants: + variant["launch"]() + torch.cuda.synchronize() + output_sha256 = tensor_sha256(variant["output"]) + timing = summarize(variant["samples"]) + if variant["name"] == "baseline": + baseline_p50 = timing["p50_s"] + results.append({ + "name": variant["name"], + "path": variant["path"], + "output_sha256": output_sha256, + "byte_exact": output_sha256 == reference_sha256, + "timing": timing, + "rounds": variant["rounds"], + }) + + if baseline_p50 is None: + raise RuntimeError("a variant named 'baseline' is required") + for result in results: + result["speedup_vs_baseline_percent"] = ( + baseline_p50 / result["timing"]["p50_s"] - 1.0 + ) * 100.0 + + report = { + "metadata": metadata, + "source_commit": "d1a57a546c3d395b1ffcbeecc66d81db76f3b4b5", + "reference_output_sha256": reference_sha256, + "q_shape": list(q.shape), + "scheduler": { + "cta_q": 128, + "cta_k": 64, + "warp_q": 32, + "warp_k": 64, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "grid_ctas": math.ceil(q.shape[1] / 128) * q.shape[2], + }, + "rounds": args.rounds, + "warmup": args.warmup, + "iterations_per_round": args.iterations, + "variants": results, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if not all(result["byte_exact"] for result in results): + raise RuntimeError("one or more variants failed byte-exact output parity") + + +if __name__ == "__main__": + main() diff --git a/tools/profile_sage2_scheduler.py b/tools/profile_sage2_scheduler.py new file mode 100644 index 0000000..03130e1 --- /dev/null +++ b/tools/profile_sage2_scheduler.py @@ -0,0 +1,407 @@ +"""Profile exact SageAttention2 preparation, mainloop, and tail scheduling on real H3 tensors.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +from pathlib import Path + +import torch + +from h3_blackwell_runtime.sage2_entry import prepare_v +from profile_attention_path import prepare_qkv, representative_attention_inputs, summarize + + +CTA_Q = 128 +CTA_K = 64 +WARP_Q = 32 +WARP_K = 64 +V_SCALE_MAX = 2.25 + + +def tensor_sha256(value: torch.Tensor) -> str: + host_bytes = value.detach().contiguous().view(torch.uint8).cpu().numpy() + return hashlib.sha256(memoryview(host_bytes)).hexdigest() + + +def event_measure(fn, *, warmup: int, iterations: int): + for _ in range(warmup): + fn() + torch.cuda.synchronize() + values = [] + result = None + for _ in range(iterations): + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + result = fn() + finished.record() + finished.synchronize() + values.append(started.elapsed_time(finished) / 1000.0) + return summarize(values), result + + +def quantize_qk(q: torch.Tensor, k: torch.Tensor, km: torch.Tensor): + import sageattention.core as sage_core + + return sage_core.per_warp_int8_cuda( + q, + k, + km, + BLKQ=CTA_Q, + WARPQ=WARP_Q, + BLKK=CTA_K, + tensor_layout="NHD", + ) + + +def quantize_v(v: torch.Tensor): + import sageattention.core as sage_core + + return sage_core.per_channel_fp8( + v, + tensor_layout="NHD", + scale_max=V_SCALE_MAX, + smooth_v=False, + ) + + +def run_mainloop( + q_int8: torch.Tensor, + k_int8: torch.Tensor, + v_fp8: torch.Tensor, + q_scale: torch.Tensor, + k_scale: torch.Tensor, + v_scale: torch.Tensor, + output: torch.Tensor, +) -> torch.Tensor: + import sageattention.core as sage_core + + sage_core.sm89_compile.qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf( + q_int8, + k_int8, + v_fp8, + output, + q_scale, + k_scale, + v_scale, + 0, + 0, + 2, + output.shape[-1] ** -0.5, + 0, + ) + return output + + +def prepare_quantized(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor): + km = k.mean(dim=1, keepdim=True) + q_int8, q_scale, k_int8, k_scale = quantize_qk(q, k, km) + v_fp8, v_scale, _ = quantize_v(v) + output = torch.empty(q.shape, dtype=q.dtype, device=q.device) + return km, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output + + +def tail_row( + q: torch.Tensor, + k: torch.Tensor, + v: torch.Tensor, + *, + warmup: int, + iterations: int, +) -> dict: + quantized = prepare_quantized(q, k, v) + _, q_int8, q_scale, k_int8, k_scale, v_fp8, v_scale, output = quantized + timing, result = event_measure( + lambda: run_mainloop( + q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, + ), + warmup=warmup, + iterations=iterations, + ) + q_len = q.shape[1] + kv_len = k.shape[1] + q_ctas = math.ceil(q_len / CTA_Q) + k_iterations = math.ceil(kv_len / CTA_K) + return { + "q_len": q_len, + "kv_len": kv_len, + "q_ctas_per_head": q_ctas, + "k_iterations_per_cta": k_iterations, + "q_tail_rows": q_len % CTA_Q, + "k_tail_rows": kv_len % CTA_K, + "scheduled_q_rows": q_ctas * CTA_Q, + "q_row_efficiency": q_len / (q_ctas * CTA_Q), + "mainloop": timing, + "checksum": result.float().sum().item(), + } + + +def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: + delta = actual.float() - expected.float() + return { + "equal": torch.equal(actual, expected), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--attention", default="sage2", choices=("sage2",)) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--tail-iterations", type=int, default=5) + parser.add_argument("--skip-tail-study", action="store_true") + parser.add_argument("--cuda-profiler-capture", action="store_true") + parser.add_argument("--expected-output-sha256") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + block, hidden, rotation, _segments, metadata = representative_attention_inputs(args) + with torch.inference_mode(): + q, k, v, _ = prepare_qkv(block, hidden, rotation, None) + km_timing, km = event_measure( + lambda: k.mean(dim=1, keepdim=True), + warmup=args.warmup, + iterations=args.iterations, + ) + qk_timing, qk = event_measure( + lambda: quantize_qk(q, k, km), + warmup=args.warmup, + iterations=args.iterations, + ) + q_int8, q_scale, k_int8, k_scale = qk + import sageattention.core as sage_core + import sageattention.quant as sage_quant + + q_int8_probe = torch.empty(q.shape, dtype=torch.int8, device=q.device) + q_scale_probe = torch.empty_like(q_scale) + q_quant_timing, _ = event_measure( + lambda: sage_quant._fused.quant_per_warp_int8_cuda( + q, q_int8_probe, q_scale_probe, CTA_Q, WARP_Q, 0, + ), + warmup=args.warmup, + iterations=args.iterations, + ) + k_int8_probe = torch.empty(k.shape, dtype=torch.int8, device=k.device) + k_scale_probe = torch.empty_like(k_scale) + k_quant_timing, _ = event_measure( + lambda: sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( + k, km.squeeze(1), k_int8_probe, k_scale_probe, CTA_K, 0, + ), + warmup=args.warmup, + iterations=args.iterations, + ) + padded_k = math.ceil(v.shape[1] / CTA_K) * CTA_K + v_transposed = torch.empty( + (v.shape[0], v.shape[3], v.shape[2], padded_k), + dtype=v.dtype, + device=v.device, + ) + v_transpose_timing, _ = event_measure( + lambda: sage_quant._fused.transpose_pad_permute_cuda(v, v_transposed, 0), + warmup=args.warmup, + iterations=args.iterations, + ) + v_fp8_probe = torch.empty_like(v_transposed, dtype=torch.float8_e4m3fn) + v_scale_probe = torch.empty( + (v.shape[0], v.shape[2], v.shape[3]), + dtype=torch.float32, + device=v.device, + ) + v_scale_quant_timing, _ = event_measure( + lambda: sage_quant._fused.scale_fuse_quant_cuda( + v_transposed, v_fp8_probe, v_scale_probe, v.shape[1], V_SCALE_MAX, 0, + ), + warmup=args.warmup, + iterations=args.iterations, + ) + v_timing, vq = event_measure( + lambda: quantize_v(v), + warmup=args.warmup, + iterations=args.iterations, + ) + v_fp8, v_scale, _ = vq + candidate_v_timing, candidate_vq = event_measure( + lambda: prepare_v(v, scale_max=V_SCALE_MAX), + warmup=args.warmup, + iterations=args.iterations, + ) + candidate_v_fp8, candidate_v_scale = candidate_vq + output = torch.empty(q.shape, dtype=q.dtype, device=q.device) + mainloop_fn = lambda: run_mainloop( + q_int8, k_int8, v_fp8, q_scale, k_scale, v_scale, output, + ) + + if args.cuda_profiler_capture: + for _ in range(args.warmup): + mainloop_fn() + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStart() + captured = mainloop_fn() + torch.cuda.synchronize() + torch.cuda.cudart().cudaProfilerStop() + report = { + "metadata": metadata, + "capture": "one unchanged prequantized Sage2 mainloop", + "q_shape": list(q.shape), + "k_shape": list(k.shape), + "v_shape": list(v.shape), + "checksum": captured.float().sum().item(), + "scheduler": { + "cta_q": CTA_Q, + "cta_k": CTA_K, + "warp_q": WARP_Q, + "warp_k": WARP_K, + "warps_per_cta": 4, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), + "heads": q.shape[2], + "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], + "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), + "explicit_pipeline_stages": 2, + }, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + return + + mainloop_timing, manual_output = event_measure( + mainloop_fn, + warmup=args.warmup, + iterations=args.iterations, + ) + candidate_output = torch.empty_like(output) + run_mainloop( + q_int8, + k_int8, + candidate_v_fp8, + q_scale, + k_scale, + candidate_v_scale, + candidate_output, + ) + torch.cuda.synchronize() + reference = __import__("sageattention").sageattn( + q, k, v, tensor_layout="NHD", is_causal=False, smooth_k=False, + ) + torch.cuda.synchronize() + output_sha256 = tensor_sha256(manual_output) + expected_output_matches = ( + args.expected_output_sha256 is None + or output_sha256 == args.expected_output_sha256 + ) + + tail_study = [] + if not args.skip_tail_study: + q_lengths = sorted({ + (q.shape[1] // CTA_Q) * CTA_Q, + (q.shape[1] // CTA_Q) * CTA_Q + 1, + q.shape[1], + }) + kv_lengths = sorted({ + (k.shape[1] // CTA_K) * CTA_K, + (k.shape[1] // CTA_K) * CTA_K + 1, + k.shape[1], + }) + for q_len in q_lengths: + tail_study.append({ + "sweep": "q_tail_fixed_kv", + **tail_row( + q[:, :q_len], k, v, + warmup=args.warmup, + iterations=args.tail_iterations, + ), + }) + for kv_len in kv_lengths: + tail_study.append({ + "sweep": "kv_tail_fixed_q", + **tail_row( + q, k[:, :kv_len], v[:, :kv_len], + warmup=args.warmup, + iterations=args.tail_iterations, + ), + }) + + report = { + "metadata": metadata, + "q_shape": list(q.shape), + "k_shape": list(k.shape), + "v_shape": list(v.shape), + "warmup": args.warmup, + "iterations": args.iterations, + "phase_timings": { + "k_mean_and_smoothing_preparation": km_timing, + "qk_int8_quantization": qk_timing, + "q_int8_quantization": q_quant_timing, + "k_int8_subtract_mean_quantization": k_quant_timing, + "v_fp8_transpose_scale_quantization": v_timing, + "v_transpose_pad_permute": v_transpose_timing, + "v_scale_fp8_quantization": v_scale_quant_timing, + "vortex_direct_v_fp8_preparation": candidate_v_timing, + "fused_mainloop": mainloop_timing, + }, + "fused_mainloop_phases": { + "int8_qk": "fused inside qk_int_sv_f8_attn_kernel", + "scale_application": "fused inside qk_int_sv_f8_attn_kernel", + "online_softmax": "fused inside qk_int_sv_f8_attn_kernel", + "pv_accumulation": "fused inside qk_int_sv_f8_attn_kernel", + "final_normalization_and_output": "fused inside qk_int_sv_f8_attn_kernel", + "timing_policy": "Do not assign independent wall time without changing the exact kernel schedule; use source-correlated hardware counters.", + }, + "scheduler": { + "cta_q": CTA_Q, + "cta_k": CTA_K, + "warp_q": WARP_Q, + "warp_k": WARP_K, + "warps_per_cta": 4, + "threads_per_cta": 128, + "dynamic_shared_memory_bytes": 32768, + "q_ctas_per_head": math.ceil(q.shape[1] / CTA_Q), + "heads": q.shape[2], + "grid_ctas": math.ceil(q.shape[1] / CTA_Q) * q.shape[2] * q.shape[0], + "k_iterations_per_cta": math.ceil(k.shape[1] / CTA_K), + "q_tail_rows": q.shape[1] % CTA_Q, + "k_tail_rows": k.shape[1] % CTA_K, + "explicit_pipeline_stages": 2, + }, + "manual_decomposition_vs_public_sage2": difference(manual_output, reference), + "vortex_v_fp8_vs_sage2": difference(candidate_v_fp8, v_fp8), + "vortex_v_scale_vs_sage2": difference(candidate_v_scale, v_scale), + "vortex_v_mainloop_vs_sage2": difference(candidate_output, manual_output), + "manual_checksum": manual_output.float().sum().item(), + "reference_checksum": reference.float().sum().item(), + "output_sha256": output_sha256, + "expected_output_sha256": args.expected_output_sha256, + "expected_output_matches": expected_output_matches, + "tail_study": tail_study, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if not expected_output_matches: + raise RuntimeError( + f"output SHA256 mismatch: expected {args.expected_output_sha256}, got {output_sha256}" + ) + + +if __name__ == "__main__": + main() diff --git a/tools/profile_sampling_stages.py b/tools/profile_sampling_stages.py new file mode 100644 index 0000000..64a17a3 --- /dev/null +++ b/tools/profile_sampling_stages.py @@ -0,0 +1,130 @@ +"""Profile top-level H3 sampling stages with synthetic refined text.""" + +from __future__ import annotations + +import argparse +import inspect +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.attention import AVAILABLE_BACKENDS +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.denoiser import H3PackedDenoiser +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.sampler import sample_video_res_multistep +from h3_blackwell_runtime.t2v import random_av_latents + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--attention", choices=AVAILABLE_BACKENDS, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=2) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--device", default="cuda") + parser.add_argument("--uninstrumented", action="store_true") + parser.add_argument("--warmup-runs", type=int, default=0) + parser.add_argument( + "--cuda-profiler-capture", + action="store_true", + help="Capture the measured sampling run between cudaProfilerStart/Stop.", + ) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + model = H3PackedDenoiser.from_checkpoint( + checkpoint, output_dtype=torch.bfloat16, attention_backend=args.attention, + ).eval() + packer = H3PromptPacker(checkpoint) + if hasattr(checkpoint, "release_cache"): + checkpoint.release_cache() + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + with torch.inference_mode(): + for _ in range(args.warmup_runs): + sample_video_res_multistep( + model, + packer, + text, + video.clone(), + audio.clone(), + steps=args.steps, + seed=args.seed, + return_audio=True, + progress=False, + ) + torch.cuda.synchronize() + trace = [] + torch.cuda.reset_peak_memory_stats() + torch.cuda.synchronize() + if args.cuda_profiler_capture: + torch.cuda.cudart().cudaProfilerStart() + started = time.perf_counter() + supports_stage_trace = "sampling_stage_trace" in inspect.signature( + sample_video_res_multistep, + ).parameters + sample_kwargs = {} + if supports_stage_trace: + sample_kwargs["sampling_stage_trace"] = None if args.uninstrumented else trace + with torch.cuda.nvtx.range("complete_sampling_run"): + sampled_video, sampled_audio = sample_video_res_multistep( + model, + packer, + text, + video.clone(), + audio.clone(), + steps=args.steps, + seed=args.seed, + return_audio=True, + progress=True, + **sample_kwargs, + ) + torch.cuda.synchronize() + elapsed = time.perf_counter() - started + if args.cuda_profiler_capture: + torch.cuda.cudart().cudaProfilerStop() + report = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "attention": args.attention, + "resolution": [args.width, args.height], + "frames": aligned_frames, + "steps": args.steps, + "seed": args.seed, + "text_tokens": args.text_tokens, + "warmup_runs": args.warmup_runs, + "cuda_profiler_capture": args.cuda_profiler_capture, + "elapsed_seconds": elapsed, + "stage_trace": trace, + "checksums": [sampled_video.float().sum().item(), sampled_audio.float().sum().item()], + "peak_allocated_bytes": torch.cuda.max_memory_allocated(), + "peak_reserved_bytes": torch.cuda.max_memory_reserved(), + "measurement_policy": ( + "uninstrumented sampling wall time" + if args.uninstrumented or not supports_stage_trace + else "CUDA-synchronized stage attribution; compare uninstrumented sampling separately" + ), + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_conversion.py b/tools/validate_cute_nvfp4_conversion.py new file mode 100644 index 0000000..6172960 --- /dev/null +++ b/tools/validate_cute_nvfp4_conversion.py @@ -0,0 +1,114 @@ +"""Validate CuTe DSL E2M1/E4M3 conversion boundaries used by the streamed producer.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import cutlass +import cutlass.cute as cute +import cutlass.torch as cutlass_torch +import torch +from cutlass.cute.runtime import from_dlpack + + +VALUES = ( + -6.1, -6.0, -5.0, -4.999, -3.5, -3.499, -2.5, -2.499, + -1.75, -1.749, -1.25, -1.249, -0.75, -0.749, -0.25, -0.249, + -0.0, 0.0, 0.249, 0.25, 0.251, 0.749, 0.75, 0.751, + 1.249, 1.25, 1.251, 1.749, 1.75, 1.751, 2.499, 2.5, + 2.501, 3.499, 3.5, 3.501, 4.999, 5.0, 5.001, 6.0, 6.1, +) +PADDED_COUNT = ((len(VALUES) + 15) // 16) * 16 + + +@cute.kernel +def conversion_kernel(source: cute.Tensor, fp4: cute.Tensor, fp8: cute.Tensor): + if cute.arch.thread_idx()[0] == 0: + fp4_tiles = cute.zipped_divide(fp4, (8,)) + fp8_tiles = cute.zipped_divide(fp8, (8,)) + fp4_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN) + fp8_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float8E4M3FN) + source_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) + fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) + fp8_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) + for fragment_index in cutlass.range_constexpr(PADDED_COUNT // 8): + offset = fragment_index * 8 + for element in cutlass.range_constexpr(8): + source_fragment[element] = source[offset + element] + values = source_fragment.load() + fp4_fragment.store(values.to(cutlass.Float4E2M1FN)) + fp8_fragment.store(values.to(cutlass.Float8E4M3FN)) + cute.copy(fp4_store, fp4_fragment, fp4_tiles[(None, fragment_index)]) + cute.copy(fp8_store, fp8_fragment, fp8_tiles[(None, fragment_index)]) + + +@cute.jit +def convert(source: cute.Tensor, fp4: cute.Tensor, fp8: cute.Tensor): + conversion_kernel(source, fp4, fp8).launch(grid=(1, 1, 1), block=(1, 1, 1)) + + +def fp4_code(value: float) -> int: + negative = torch.signbit(torch.tensor(value)).item() + magnitude = abs(value) + if magnitude > 5.0: + code = 7 + elif magnitude >= 3.5: + code = 6 + elif magnitude > 2.5: + code = 5 + elif magnitude >= 1.75: + code = 4 + elif magnitude > 1.25: + code = 3 + elif magnitude >= 0.75: + code = 2 + elif magnitude > 0.25: + code = 1 + else: + code = 0 + return code | (8 if negative else 0) + + +def main() -> None: + padded_values = VALUES + (0.0,) * (PADDED_COUNT - len(VALUES)) + source_torch = torch.tensor(padded_values, device="cuda", dtype=torch.float32) + source = from_dlpack(source_torch, assumed_align=4).mark_layout_dynamic() + fp4, fp4_torch = cutlass_torch.cute_tensor_like( + torch.zeros_like(source_torch), cutlass.Float4E2M1FN, is_dynamic_layout=True, assumed_align=16, + ) + fp8, fp8_torch = cutlass_torch.cute_tensor_like( + torch.zeros_like(source_torch), cutlass.Float8E4M3FN, is_dynamic_layout=True, assumed_align=16, + ) + compiled = cute.compile(convert, source, fp4, fp8) + compiled(source, fp4, fp8) + torch.cuda.synchronize() + + fp4_bytes = fp4_torch.view(torch.uint8).flatten().cpu().tolist() + expected_codes = [fp4_code(value) for value in padded_values] + expected_low_first = [ + expected_codes[index] | ((expected_codes[index + 1] if index + 1 < len(expected_codes) else 0) << 4) + for index in range(0, len(expected_codes), 2) + ] + fp8_bytes = fp8_torch.view(torch.uint8).flatten().cpu() + expected_fp8 = source_torch.to(torch.float8_e4m3fn).view(torch.uint8).cpu() + report = { + "device": torch.cuda.get_device_name(), + "cutlass_dsl": "4.6.2", + "value_count": len(VALUES), + "padded_value_count": len(padded_values), + "fp4_torch_shape": list(fp4_torch.shape), + "fp4_bytes": fp4_bytes, + "expected_low_first_bytes": expected_low_first, + "fp4_prefix_equal": fp4_bytes[:len(expected_low_first)] == expected_low_first, + "fp8_equal": torch.equal(fp8_bytes[:len(padded_values)], expected_fp8), + "fp8_difference_count": int((fp8_bytes[:len(padded_values)] != expected_fp8).sum().item()), + } + output = Path("/output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-conversion-contract.json") + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_h3.py b/tools/validate_cute_nvfp4_h3.py new file mode 100644 index 0000000..f7a3aa3 --- /dev/null +++ b/tools/validate_cute_nvfp4_h3.py @@ -0,0 +1,537 @@ +"""Compare the CUTLASS DSL SM121 block-scaled GEMM with real H3 NVFP4 tensors.""" + +from __future__ import annotations + +import argparse +import importlib.util +import json +import sys +from pathlib import Path + +import torch +import torch.nn.functional as functional + +from h3_blackwell_runtime.nvfp4_quant import vortex_quantize_nvfp4 +from profile_nvfp4_linear import module_for_name, representative_inputs + + +def _replace_once(source: str, old: str, new: str, name: str) -> str: + if source.count(old) != 1: + raise RuntimeError( + f"CUTLASS {name} patch expected one occurrence, found {source.count(old)}: {old!r}" + ) + return source.replace(old, new) + + +def _patch_streaming_a(source: str) -> str: + source = _replace_once( + source, + "import cutlass\nimport cutlass.cute as cute", + """import cutlass +import cutlass.cute as cute +from cutlass import Float32 +from cutlass.cutlass_dsl import dsl_user_op +from cutlass._mlir import ir +from cutlass._mlir.dialects import llvm + + +@dsl_user_op +def vortex_rcp_approx_ftz_f32( + x: Float32, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, +) -> Float32: + result = llvm.inline_asm( + Float32.mlir_type, + [x.ir_value(loc=loc, ip=ip)], + "rcp.approx.ftz.f32 $0, $1;", + "=f,f", + has_side_effects=False, + asm_dialect=0, + loc=loc, + ip=ip, + ) + return Float32(result)""", + "streaming-A reciprocal", + ) + replacements = ( + ( + " self.a_dtype = a.element_type\n self.b_dtype = b.element_type\n self.c_dtype = c.element_type\n self.sf_dtype = sfa.element_type", + " self.a_source_dtype = a.element_type\n self.a_dtype = cutlass.Float4E2M1FN\n self.b_dtype = b.element_type\n self.c_dtype = c.element_type\n self.sf_dtype = cutlass.Float8E4M3FN", + ), + ( + " self.sfa_layout = blockscaled_utils.tile_atom_to_shape_SF(\n a.shape, self.sf_vec_size\n )\n sfa_tensor = cute.make_tensor(sfa.iterator, self.sfa_layout)\n", + "", + ), + ( + " tma_atom_a, tma_tensor_a = self._make_tma_atoms_and_tensors(\n a,\n self.a_smem_layout_staged,\n (self.tile_shape_mnk[0], self.tile_shape_mnk[2]),\n 1,\n internal_type=self.tma_internal_a_dtype,\n )\n\n", + "", + ), + ( + " tma_atom_sfa, tma_tensor_sfa = self._make_tma_atoms_and_tensors(\n sfa_tensor,\n self.sfa_smem_layout_staged,\n (self.tile_shape_mnk[0], self.tile_shape_mnk[2]),\n 1,\n internal_type=cutlass.Int16,\n )\n\n", + "", + ), + ( + " self.kernel(\n tma_atom_a,\n tma_tensor_a,\n tma_atom_b,\n tma_tensor_b,\n tma_atom_sfa,\n tma_tensor_sfa,\n tma_atom_sfb,", + " self.kernel(\n a,\n sfa,\n tma_atom_b,\n tma_tensor_b,\n tma_atom_sfb,", + ), + ( + " tma_atom_a: cute.CopyAtom,\n mA_mkl: cute.Tensor,\n tma_atom_b: cute.CopyAtom,\n mB_nkl: cute.Tensor,\n tma_atom_sfa: cute.CopyAtom,\n mSFA_mkl: cute.Tensor,\n tma_atom_sfb: cute.CopyAtom,", + " mA_mkl: cute.Tensor,\n tensor_scale_a: cute.Tensor,\n tma_atom_b: cute.CopyAtom,\n mB_nkl: cute.Tensor,\n tma_atom_sfb: cute.CopyAtom,", + ), + ( + " cpasync.prefetch_descriptor(tma_atom_a)\n cpasync.prefetch_descriptor(tma_atom_b)\n cpasync.prefetch_descriptor(tma_atom_sfa)\n cpasync.prefetch_descriptor(tma_atom_sfb)", + " cpasync.prefetch_descriptor(tma_atom_b)\n cpasync.prefetch_descriptor(tma_atom_sfb)", + ), + ( + " tma_copy_bytes = (\n cute.size_in_bytes(self.a_dtype, a_smem_layout)\n + cute.size_in_bytes(self.b_dtype, b_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfa_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfb_smem_layout)\n )", + " tma_copy_bytes = (\n cute.size_in_bytes(self.b_dtype, b_smem_layout)\n + cute.size_in_bytes(self.sf_dtype, sfb_smem_layout)\n )", + ), + ( + " # (tM, tK, loopM, loopK, loopL)\n gSFA_mkl = cute.local_tile(\n mSFA_mkl,\n cute.slice_(self.tile_shape_mnk, (None, 0, None)),\n (None, None, None),\n )\n", + "", + ), + ( + " # TMA load A partition_S/D\n a_cta_layout = cute.make_layout(cute.slice_(cta_layout_mnk, (0, None, 0)).shape)\n a_cta_crd = cluster_coord_mnk[1]\n tAsA, tAgA = cpasync.tma_partition(\n tma_atom_a,\n a_cta_crd,\n a_cta_layout,\n cute.group_modes(sA, 0, 2),\n cute.group_modes(gA_mkl, 0, 2),\n )\n\n", + "", + ), + ( + " tAsSFA, tAgSFA = cpasync.tma_partition(\n tma_atom_sfa,\n a_cta_crd,\n a_cta_layout,\n cute.group_modes(sSFA, 0, 2),\n cute.group_modes(gSFA_mkl, 0, 2),\n )\n tAsSFA = cute.filter_zeros(tAsSFA)\n tAgSFA = cute.filter_zeros(tAgSFA)\n\n", + "", + ), + ( + " cute.arch.setmaxregister_decrease(self.load_register_requirement)\n\n while work_tile.is_valid_tile:", + """ cute.arch.setmaxregister_decrease(self.load_register_requirement) + producer_lane = tidx - self.tma_load_warp_id * self.num_threads_per_warp + fp4_store = cute.make_copy_atom( + cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN + ) + source_fragment = cute.make_rmem_tensor((16,), cutlass.Float32) + normalized_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) + fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) + scale_source = cute.make_rmem_tensor((8,), cutlass.Float32) + scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) + decoded_scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) + while work_tile.is_valid_tile:""", + ), + ( + " tAgA_mkl = tAgA[(None, tile_coord_mnl[0], None, tile_coord_mnl[2])]\n tBgB_nkl = tBgB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]\n tAgSFA_mkl = tAgSFA[(None, tile_coord_mnl[0], None, tile_coord_mnl[2])]\n tBgSFB_nkl = tBgSFB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]", + " tBgB_nkl = tBgB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]\n tBgSFB_nkl = tBgSFB[(None, tile_coord_mnl[1], None, tile_coord_mnl[2])]", + ), + ( + " tAgA_k = tAgA_mkl[(None, mainloop_producer_state.count)]\n tAsA_pipe = tAsA[(None, mainloop_producer_state.index)]\n\n tBgB_k = tBgB_nkl[(None, mainloop_producer_state.count)]\n tBsB_pipe = tBsB[(None, mainloop_producer_state.index)]\n\n tAgSFA_k = tAgSFA_mkl[(None, mainloop_producer_state.count)]\n tAsSFA_pipe = tAsSFA[(None, mainloop_producer_state.index)]\n\n tBgSFB_k = tBgSFB_nkl[(None, mainloop_producer_state.count)]", + " tBgB_k = tBgB_nkl[(None, mainloop_producer_state.count)]\n tBsB_pipe = tBsB[(None, mainloop_producer_state.index)]\n\n tBgSFB_k = tBgSFB_nkl[(None, mainloop_producer_state.count)]", + ), + ( + """ cute.copy( + tma_atom_a, + tAgA_k, + tAsA_pipe, + tma_bar_ptr=mainloop_pipeline.producer_get_barrier( + mainloop_producer_state + ), + ) +""", + """ scale = tensor_scale_a[0] + stage = mainloop_producer_state.index + source_k_base = mainloop_producer_state.count * self.tile_shape_mnk[2] + source_m_base = tile_coord_mnl[0] * self.tile_shape_mnk[0] + for row_group in cutlass.range_constexpr(4): + row = producer_lane + row_group * self.num_threads_per_warp + source_row = source_m_base + row + sA_row = sA[(row, None, stage)] + sA_tiles = cute.zipped_divide(sA_row, (8,)) + for block_column in cutlass.range_constexpr(8): + source_column = source_k_base + block_column * 16 + maximum = cutlass.Float32(0.0) + for element in cutlass.range_constexpr(16): + value = mA_mkl[ + source_row, + source_column + element, + tile_coord_mnl[2], + ] + source_fragment[element] = value + maximum = cutlass.max( + cutlass.max(value, -value), maximum + ) + + raw_block_scale = ( + maximum / cutlass.Float32(6.0) + ) / scale + for element in cutlass.range_constexpr(8): + scale_source[element] = raw_block_scale + scale_values = scale_source.load() + scale_values = cute.where( + scale_values <= cutlass.Float32(448.0), + scale_values, + cutlass.Float32(448.0), + ) + scale_fragment.store( + scale_values.to(cutlass.Float8E4M3FN) + ) + sSFA[row, block_column * 16, stage] = scale_fragment[0] + decoded_scale_fragment.store( + scale_fragment.load().to(cutlass.Float32) + ) + raw_encode_scale = vortex_rcp_approx_ftz_f32( + decoded_scale_fragment[0] * scale + ) + for element in cutlass.range_constexpr(8): + scale_source[element] = raw_encode_scale + encode_scale_values = scale_source.load() + encode_scale_values = cute.where( + encode_scale_values + <= cutlass.Float32(3.402823466e38), + encode_scale_values, + cutlass.Float32(3.402823466e38), + ) + scale_source.store(encode_scale_values) + encode_scale = scale_source[0] + + for half in cutlass.range_constexpr(2): + for element in cutlass.range_constexpr(8): + normalized_fragment[element] = source_fragment[ + half * 8 + element + ] * encode_scale + fp4_fragment.store( + normalized_fragment.load().to( + cutlass.Float4E2M1FN + ) + ) + cute.copy( + fp4_store, + fp4_fragment, + sA_tiles[(None, block_column * 2 + half)], + ) + cute.arch.fence_proxy("async.shared", space="cta") +""", + ), + ( + """ cute.copy( + tma_atom_sfa, + tAgSFA_k, + tAsSFA_pipe, + tma_bar_ptr=mainloop_pipeline.producer_get_barrier( + mainloop_producer_state + ), + ) +""", + "", + ), + ) + for index, (old, new) in enumerate(replacements): + source = _replace_once(source, old, new, f"streaming-A[{index}]") + return source + + +def load_cutlass_example(path: Path, *, fuse_alpha: bool, stream_a: bool = False): + if not path.is_file(): + raise FileNotFoundError(f"CUTLASS DSL example not found: {path}") + sys.path.insert(0, str(path.parent)) + source = path.read_text(encoding="utf-8") + if stream_a: + source = _patch_streaming_a(source) + if fuse_alpha: + replacements = ( + (" c: cute.Tensor,\n max_active_clusters: cutlass.Constexpr,", " c: cute.Tensor,\n alpha: cute.Tensor,\n max_active_clusters: cutlass.Constexpr,"), + (" mC_mnl: cute.Tensor,\n tiled_mma: cute.TiledMma,", " mC_mnl: cute.Tensor,\n alpha: cute.Tensor,\n tiled_mma: cute.TiledMma,"), + (" tma_tensor_c,\n self.tiled_mma,", " tma_tensor_c,\n alpha,\n self.tiled_mma,"), + (" tRS_rD_out.store(acc_vec.to(self.c_dtype))", " tRS_rD_out.store((acc_vec * alpha[0]).to(self.c_dtype))"), + ) + for old, new in replacements: + source = _replace_once(source, old, new, "alpha") + if stream_a or fuse_alpha: + suffix = "_vortex_stream_a" if stream_a else "_vortex" + suffix += "_alpha" if fuse_alpha else "" + load_path = path.with_name(f"{path.stem}{suffix}.py") + load_path.write_text(source, encoding="utf-8") + else: + load_path = path + spec = importlib.util.spec_from_file_location("vortex_cutlass_blockscaled", load_path) + if spec is None or spec.loader is None: + raise ImportError(f"Cannot load CUTLASS DSL example: {load_path}") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def decode_comfy_fp4(storage: torch.Tensor) -> torch.Tensor: + lookup = torch.tensor( + [0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, -0.0, -0.5, -1.0, -1.5, -2.0, -3.0, -4.0, -6.0], + device=storage.device, + dtype=torch.float32, + ) + codes = torch.stack((storage >> 4, storage & 0x0F), dim=-1).reshape(storage.shape[0], -1) + return lookup[codes.long()] + + +def fp4_tensor(storage: torch.Tensor, *, swap_nibbles: bool, reencode: bool): + import cutlass + import cutlass.torch as cutlass_torch + from cutlass.cute.runtime import from_dlpack + + rows, packed_columns = storage.shape + if reencode: + logical = decode_comfy_fp4(storage).unsqueeze(-1) + return cutlass_torch.cute_tensor_like( + logical, cutlass.Float4E2M1FN, is_dynamic_layout=True, assumed_align=16, + ) + if swap_nibbles: + storage = ((storage & 0x0F) << 4) | ((storage & 0xF0) >> 4) + # DLPack cannot export Torch's packed FP4 dtype yet. Build the desired + # logical layout from an oversized uint8 allocation, then reinterpret its + # iterator as FP4 and populate only the packed storage that the layout uses. + backing = torch.empty( + (rows, packed_columns * 2, 1), device=storage.device, dtype=torch.uint8, + ) + backing.zero_() + backing[:, :packed_columns, 0].copy_(storage) + tensor = from_dlpack(backing, assumed_align=16) + tensor.element_type = cutlass.Float4E2M1FN + tensor = tensor.mark_layout_dynamic(leading_dim=1) + return tensor, backing + + +def output_tensor(storage: torch.Tensor): + from cutlass.cute.runtime import from_dlpack + + tensor = from_dlpack(storage.unsqueeze(-1), assumed_align=16) + tensor = tensor.mark_compact_shape_dynamic(mode=1, stride_order=(2, 0, 1), divisibility=1) + return tensor + + +def scale_tensor(storage: torch.Tensor): + import cutlass + from cutlass.cute.runtime import from_dlpack + + tensor = from_dlpack(storage.view(torch.uint8).unsqueeze(-1), assumed_align=16) + tensor.element_type = cutlass.Float8E4M3FN + return tensor.mark_layout_dynamic(leading_dim=1) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cutlass-example", type=Path, required=True) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--linear", choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1", "mlp_fc2"), default="attn_qkv_proj") + parser.add_argument("--rows", type=int, default=128) + parser.add_argument("--tile-k", type=int, choices=(128, 256), default=128) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--device", default="cuda") + parser.add_argument("--swap-nibbles", action="store_true") + parser.add_argument("--reencode-fp4", action="store_true") + parser.add_argument("--zero-a", action="store_true") + parser.add_argument("--fuse-alpha", action="store_true") + parser.add_argument("--stream-a", action="store_true") + parser.add_argument("--warmup", type=int, default=0) + parser.add_argument("--iterations", type=int, default=0) + parser.add_argument("--benchmark-reference", action="store_true") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + if args.rows <= 0 or args.rows % 128: + raise ValueError("--rows must be a positive multiple of 128") + if args.stream_a and args.linear == "mlp_fc2": + raise ValueError("--stream-a deliberately excludes mlp_fc2; retain the reference cuBLAS path") + if args.warmup < 0 or args.iterations < 0: + raise ValueError("--warmup and --iterations must be non-negative") + + import cutlass + import cutlass.cute as cute + import cutlass.torch as cutlass_torch + import comfy_kitchen as ck + from cutlass.cute.runtime import from_dlpack + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + example = load_cutlass_example( + args.cutlass_example, fuse_alpha=args.fuse_alpha, stream_a=args.stream_a, + ) + block, inputs, metadata = representative_inputs(args) + linear = module_for_name(block, args.linear) + activation = inputs[args.linear].reshape(-1, linear.in_features)[:args.rows].contiguous() + + with torch.inference_mode(): + packed_activation = vortex_quantize_nvfp4(activation) + packed_weight = linear._packed_weight() + a_qdata, a_tensor_scale, a_block_scale = TensorCoreNVFP4Layout.get_plain_tensors(packed_activation) + b_qdata, b_tensor_scale, b_block_scale = TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) + reference = functional.linear(packed_activation, packed_weight, None)[:args.rows, :linear.out_features] + raw_reference = ck.scaled_mm_nvfp4( + a_qdata, + b_qdata, + tensor_scale_a=a_tensor_scale, + tensor_scale_b=b_tensor_scale, + block_scale_a=a_block_scale, + block_scale_b=b_block_scale, + out_dtype=torch.bfloat16, + alpha=torch.ones(1, device=activation.device, dtype=torch.float32), + )[:args.rows, :linear.out_features] + if args.zero_a: + a_qdata = torch.zeros_like(a_qdata) + + output_bf16 = torch.zeros((a_qdata.shape[0], b_qdata.shape[0]), device=activation.device, dtype=torch.bfloat16) + if args.stream_a: + if args.rows != 128 or activation.shape[1] % 128: + raise ValueError("--stream-a currently requires exactly 128 rows and K divisible by 128") + a = output_tensor(activation) + a_backing = activation + else: + a, a_backing = fp4_tensor(a_qdata, swap_nibbles=args.swap_nibbles, reencode=args.reencode_fp4) + b, b_backing = fp4_tensor(b_qdata, swap_nibbles=args.swap_nibbles, reencode=args.reencode_fp4) + sfa = ( + from_dlpack(a_tensor_scale.float().reshape(1).contiguous(), assumed_align=4) + if args.stream_a + else scale_tensor(a_block_scale) + ) + sfb = scale_tensor(b_block_scale) + c = output_tensor(output_bf16) + + gemm = example.Sm120BlockScaledGemmKernel( + cutlass.Float32, + 16, + (128, 128, args.tile_k), + (128, 128), + ) + hardware_info = cutlass.utils.HardwareInfo() + max_active_clusters = hardware_info.get_max_active_clusters(1) + stream = cutlass_torch.default_stream() + alpha = a_tensor_scale.float() * b_tensor_scale.float() + alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) + if args.fuse_alpha: + compiled = cute.compile(gemm, a, b, sfa, sfb, c, alpha_argument, max_active_clusters, stream) + arguments = (a, b, sfa, sfb, c, alpha_argument, stream) + else: + compiled = cute.compile(gemm, a, b, sfa, sfb, c, max_active_clusters, stream) + arguments = (a, b, sfa, sfb, c, stream) + compiled(*arguments) + torch.cuda.synchronize() + + timing = None + if args.iterations: + for _ in range(args.warmup): + compiled(*arguments) + torch.cuda.synchronize() + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + for _ in range(args.iterations): + compiled(*arguments) + finished.record() + finished.synchronize() + total_ms = started.elapsed_time(finished) + timing = { + "warmup": args.warmup, + "iterations": args.iterations, + "total_ms": total_ms, + "mean_ms": total_ms / args.iterations, + } + + reference_timing = None + if args.benchmark_reference: + if not args.iterations: + raise ValueError("--benchmark-reference requires --iterations") + + def measure_cuda(fn): + result = None + for _ in range(args.warmup): + result = fn() + torch.cuda.synchronize() + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + for _ in range(args.iterations): + result = fn() + finished.record() + finished.synchronize() + total = started.elapsed_time(finished) + return result, total / args.iterations + + _, quantize_ms = measure_cuda(lambda: vortex_quantize_nvfp4(activation)) + _, complete_ms = measure_cuda( + lambda: functional.linear( + vortex_quantize_nvfp4(activation), packed_weight, None, + ) + ) + reference_timing = { + "backend": "vortex_scale_plus_comfy_pack_gemm", + "activation_quantize_mean_ms": quantize_ms, + "complete_projection_mean_ms": complete_ms, + } + + candidate = ( + output_bf16[:args.rows, :linear.out_features] + if args.fuse_alpha + else (output_bf16[:args.rows, :linear.out_features].float() * alpha).to(reference.dtype) + ) + raw_candidate = output_bf16[:args.rows, :linear.out_features] + raw_delta = raw_candidate.float() - raw_reference.float() + delta = candidate.float() - reference.float() + report = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "cutlass_dsl": "4.6.2", + "metadata": metadata, + "linear": args.linear, + "rows": args.rows, + "mnk": [args.rows, linear.out_features, linear.in_features], + "tile_shape_mnk": [128, 128, args.tile_k], + "swap_nibbles": args.swap_nibbles, + "reencode_fp4": args.reencode_fp4, + "zero_a": args.zero_a, + "fuse_alpha": args.fuse_alpha, + "stream_a": args.stream_a, + "streamed_activation_materialization": ( + {"global_qdata": False, "global_sfa": False} + if args.stream_a + else None + ), + "timing": timing, + "reference_timing": reference_timing, + "cute_shapes": {"a": str(a.shape), "b": str(b.shape), "sfa": str(sfa.shape), "sfb": str(sfb.shape), "c": str(c.shape)}, + "tensor_scales": {"a": a_tensor_scale.float().item(), "b": b_tensor_scale.float().item(), "alpha": alpha.item()}, + "raw_output": { + "dtype": str(output_bf16.dtype), + "checksum": output_bf16.float().sum().item(), + "max_abs": output_bf16.float().abs().max().item(), + "nonzero": int(torch.count_nonzero(output_bf16).item()), + "finite": bool(torch.isfinite(output_bf16).all().item()), + }, + "raw_blockscaled_parity": { + "applicable": not args.fuse_alpha, + "reference_checksum": raw_reference.float().sum().item(), + "candidate_checksum": raw_candidate.float().sum().item(), + "equal": torch.equal(raw_candidate, raw_reference), + "max_abs": raw_delta.abs().max().item(), + "mean_abs": raw_delta.abs().mean().item(), + }, + "reference_checksum": reference.float().sum().item(), + "candidate_checksum": candidate.float().sum().item(), + "equal": torch.equal(candidate, reference), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "relative_l2": (delta.norm() / reference.float().norm().clamp_min(1e-12)).item(), + "numerical_note": ( + "The experimental epilogue applies the FP32 global-scale product before BF16 conversion." + if args.fuse_alpha + else "The stock SM121 kernel rounds before the external global-scale product; exact H3 integration requires the fused-alpha epilogue." + ), + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_real_tiles.py b/tools/validate_cute_nvfp4_real_tiles.py new file mode 100644 index 0000000..be72bd2 --- /dev/null +++ b/tools/validate_cute_nvfp4_real_tiles.py @@ -0,0 +1,154 @@ +"""Compare the CuTe tile producer with every tile of a real H3 activation.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import cutlass +import cutlass.cute as cute +import cutlass.torch as cutlass_torch +import torch +from cutlass.cute.runtime import from_dlpack + +from h3_blackwell_runtime.nvfp4_quant import vortex_quantize_nvfp4 +from profile_nvfp4_linear import module_for_name, representative_inputs +from validate_cute_nvfp4_tile_producer import ( + BLOCKS_PER_ROW, + JOBS, + TILE, + produce_tile, + unswizzle_scales, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--linear", + choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), + required=True, + ) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + args = parse_args() + block, inputs, metadata = representative_inputs(args) + linear = module_for_name(block, args.linear) + activation = inputs[args.linear].reshape(-1, linear.in_features)[:TILE].contiguous() + if activation.shape != (TILE, linear.in_features) or activation.shape[1] % TILE: + raise ValueError(f"Expected a 128-row activation with K divisible by 128, got {activation.shape}") + + packed = vortex_quantize_nvfp4(activation) + expected_qdata, tensor_scale, expected_physical_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(packed) + ) + expected_fp4 = ((expected_qdata & 0x0F) << 4) | ((expected_qdata & 0xF0) >> 4) + expected_scales = unswizzle_scales(expected_physical_scales.view(torch.uint8)) + + source_torch = torch.empty(TILE, TILE, device="cuda", dtype=torch.bfloat16) + source = from_dlpack(source_torch, assumed_align=16).mark_layout_dynamic(leading_dim=1) + scale_torch = tensor_scale.float().reshape(1).contiguous() + scale = from_dlpack(scale_torch, assumed_align=4).mark_layout_dynamic() + fp4, fp4_torch = cutlass_torch.cute_tensor_like( + torch.zeros_like(source_torch, dtype=torch.float32), + cutlass.Float4E2M1FN, + is_dynamic_layout=True, + assumed_align=16, + ) + block_scales_torch = torch.zeros(JOBS, 8, device="cuda", dtype=torch.uint8) + block_scales = from_dlpack(block_scales_torch.flatten(), assumed_align=16) + block_scales.element_type = cutlass.Float8E4M3FN + block_scales = block_scales.mark_layout_dynamic() + scalar_scales_torch = torch.zeros(JOBS, device="cuda", dtype=torch.uint8) + scalar_scales = from_dlpack(scalar_scales_torch, assumed_align=16) + scalar_scales.element_type = cutlass.Float8E4M3FN + scalar_scales = scalar_scales.mark_layout_dynamic() + + compiled = cute.compile(produce_tile, source, scale, fp4, block_scales, scalar_scales) + tile_reports = [] + examples = [] + total_fp4_differences = 0 + total_scale_differences = 0 + for k_start in range(0, activation.shape[1], TILE): + source_torch.copy_(activation[:, k_start : k_start + TILE]) + compiled(source, scale, fp4, block_scales, scalar_scales) + torch.cuda.synchronize() + actual_fp4 = fp4_torch.view(torch.uint8).flatten()[: TILE * TILE // 2].reshape(TILE, TILE // 2) + actual_scales = scalar_scales_torch.reshape(TILE, BLOCKS_PER_ROW) + fp4_reference = expected_fp4[:, k_start // 2 : (k_start + TILE) // 2] + scale_reference = expected_scales[:, k_start // 16 : (k_start + TILE) // 16] + fp4_differences = int((actual_fp4 != fp4_reference).sum().item()) + scale_differences = int((actual_scales != scale_reference).sum().item()) + total_fp4_differences += fp4_differences + total_scale_differences += scale_differences + if fp4_differences or scale_differences: + tile_reports.append({ + "k_start": k_start, + "fp4_difference_count": fp4_differences, + "block_scale_difference_count": scale_differences, + }) + if fp4_differences and len(examples) < 20: + for row, packed_column in (actual_fp4 != fp4_reference).nonzero().tolist(): + global_column = k_start + packed_column * 2 + block_scale_byte = scale_reference[row, packed_column // 8].reshape(1) + decoded_scale = block_scale_byte.view(torch.float8_e4m3fn).float() + encode_scale = torch.minimum( + torch.ones_like(decoded_scale) / (decoded_scale * scale_torch), + torch.full_like(decoded_scale, torch.finfo(torch.float32).max), + ) + normalized = activation[row, global_column : global_column + 2].float() * encode_scale + examples.append({ + "row": row, + "global_column": global_column, + "source": [ + float(activation[row, global_column].float().item()), + float(activation[row, global_column + 1].float().item()), + ], + "actual_byte": int(actual_fp4[row, packed_column].item()), + "expected_byte": int(fp4_reference[row, packed_column].item()), + "block_scale_byte": int(block_scale_byte.item()), + "decoded_block_scale": float(decoded_scale.item()), + "encode_scale": float(encode_scale.item()), + "torch_normalized": normalized.tolist(), + }) + if len(examples) == 20: + break + + report = { + "device": torch.cuda.get_device_name(), + "cutlass_dsl": "4.6.2", + "metadata": metadata, + "linear": args.linear, + "activation_shape": list(activation.shape), + "tensor_scale": scale_torch.item(), + "tile_count": activation.shape[1] // TILE, + "fp4_difference_count": total_fp4_differences, + "block_scale_difference_count": total_scale_differences, + "equal": total_fp4_differences == 0 and total_scale_differences == 0, + "differing_tiles": tile_reports, + "difference_examples": examples, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_ring.py b/tools/validate_cute_nvfp4_ring.py new file mode 100644 index 0000000..51217ec --- /dev/null +++ b/tools/validate_cute_nvfp4_ring.py @@ -0,0 +1,271 @@ +"""Validate and time a bounded 128-row NVFP4 packed-tile ring prototype.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path + +import cutlass +import cutlass.cute as cute +import cutlass.torch as cutlass_torch +import torch +import torch.nn.functional as functional +from cutlass.cute.runtime import from_dlpack + +from h3_blackwell_runtime.nvfp4_quant import ( + nvfp4_activation_scale, + vortex_native_quantize_nvfp4, + vortex_native_quantize_nvfp4_into, + vortex_quantize_nvfp4, +) +from profile_nvfp4_linear import module_for_name, representative_inputs +from validate_cute_nvfp4_h3 import ( + fp4_tensor, + load_cutlass_example, + output_tensor, + scale_tensor, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cutlass-example", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--linear", + choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), + required=True, + ) + parser.add_argument("--warmup", type=int, default=5) + parser.add_argument("--iterations", type=int, default=20) + parser.add_argument("--rows", type=int, default=128) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def measure_cuda(fn, *, warmup: int, iterations: int) -> tuple[object, float]: + result = None + for _ in range(warmup): + result = fn() + torch.cuda.synchronize() + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + for _ in range(iterations): + result = fn() + finished.record() + finished.synchronize() + return result, started.elapsed_time(finished) / iterations + + +def main() -> None: + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + import comfy_kitchen as ck + + args = parse_args() + if args.warmup < 0 or args.iterations <= 0: + raise ValueError("--warmup must be non-negative and --iterations must be positive") + if args.rows <= 0 or args.rows % 128: + raise ValueError("--rows must be a positive multiple of 128") + + example = load_cutlass_example(args.cutlass_example, fuse_alpha=True) + block, inputs, metadata = representative_inputs(args) + linear = module_for_name(block, args.linear) + full_activation = inputs[args.linear].reshape(-1, linear.in_features).contiguous() + if args.rows > full_activation.shape[0]: + raise ValueError(f"--rows exceeds the available {full_activation.shape[0]} rows") + activation = full_activation[:args.rows].contiguous() + if activation.shape[1] % 128: + raise ValueError(f"Ring prototype requires K divisible by 128, got {activation.shape}") + global_scale = nvfp4_activation_scale(full_activation).float() + + with torch.inference_mode(): + expected_packed = vortex_quantize_nvfp4(activation, scale=global_scale) + ring_packed = vortex_native_quantize_nvfp4(activation, scale=global_scale) + packed_weight = linear._packed_weight() + expected_qdata, expected_tensor_scale, expected_block_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(expected_packed) + ) + ring_qdata, ring_tensor_scale, ring_block_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(ring_packed) + ) + b_qdata, tensor_scale_b, b_block_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) + ) + reference = functional.linear(expected_packed, packed_weight, None) + + a, _ = fp4_tensor(ring_qdata, swap_nibbles=False, reencode=True) + b, _ = fp4_tensor(b_qdata, swap_nibbles=False, reencode=True) + sfa = scale_tensor(ring_block_scales) + sfb = scale_tensor(b_block_scales) + output_bf16 = torch.zeros( + args.rows, b_qdata.shape[0], device="cuda", dtype=torch.bfloat16, + ) + c = output_tensor(output_bf16) + alpha = ring_tensor_scale.float() * tensor_scale_b.float() + alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) + gemm = example.Sm120BlockScaledGemmKernel( + cutlass.Float32, 16, (128, 128, 128), (128, 128), + ) + max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) + stream = cutlass_torch.default_stream() + compiled_gemm = cute.compile( + gemm, a, b, sfa, sfb, c, alpha_argument, max_active_clusters, stream, + ) + + def run_consumer(): + return compiled_gemm(a, b, sfa, sfb, c, alpha_argument, stream) + + def run_pack_into(): + return vortex_native_quantize_nvfp4_into( + activation, global_scale, ring_qdata, ring_block_scales, + ) + + def run_cute_ring(): + run_pack_into() + return run_consumer() + + def run_comfy_consumer(): + return ck.scaled_mm_nvfp4( + ring_qdata, + b_qdata, + tensor_scale_a=ring_tensor_scale, + tensor_scale_b=tensor_scale_b, + block_scale_a=ring_block_scales, + block_scale_b=b_block_scales, + out_dtype=torch.bfloat16, + alpha=ring_tensor_scale.float() * tensor_scale_b.float(), + ) + + def run_comfy_ring(): + run_pack_into() + return run_comfy_consumer() + + run_consumer() + torch.cuda.synchronize() + candidate = output_bf16[:, : linear.out_features] + delta = candidate.float() - reference.float() + + _, scale_ms = measure_cuda( + lambda: nvfp4_activation_scale(full_activation), + warmup=args.warmup, + iterations=args.iterations, + ) + _, producer_ms = measure_cuda( + run_pack_into, + warmup=args.warmup, + iterations=args.iterations, + ) + _, consumer_ms = measure_cuda( + run_consumer, + warmup=args.warmup, + iterations=args.iterations, + ) + _, actual_ring_ms = measure_cuda( + run_cute_ring, + warmup=args.warmup, + iterations=args.iterations, + ) + comfy_candidate, comfy_consumer_ms = measure_cuda( + run_comfy_consumer, + warmup=args.warmup, + iterations=args.iterations, + ) + _, comfy_ring_ms = measure_cuda( + run_comfy_ring, + warmup=args.warmup, + iterations=args.iterations, + ) + _, reference_ms = measure_cuda( + lambda: functional.linear( + vortex_quantize_nvfp4(activation, scale=global_scale), + packed_weight, + None, + ), + warmup=args.warmup, + iterations=args.iterations, + ) + + qdata_bytes = ring_qdata.numel() * ring_qdata.element_size() + sfa_bytes = ring_block_scales.numel() * ring_block_scales.element_size() + chunk_count = math.ceil(full_activation.shape[0] / args.rows) + modeled_cute_chunk_ms = producer_ms + consumer_ms + modeled_comfy_chunk_ms = producer_ms + comfy_consumer_ms + modeled_reference_canonical_ms = scale_ms + chunk_count * reference_ms + report = { + "device": torch.cuda.get_device_name(), + "cutlass_dsl": "4.6.2", + "metadata": metadata, + "linear": args.linear, + "mnk": [args.rows, linear.out_features, activation.shape[1]], + "full_activation_rows": full_activation.shape[0], + "modeled_chunk_count": chunk_count, + "ring": { + "row_capacity": args.rows, + "producer": "vortex_native_quantize_nvfp4", + "qdata_bytes": qdata_bytes, + "sfa_bytes": sfa_bytes, + "logical_bytes": qdata_bytes + sfa_bytes, + }, + "parity": { + "tensor_scale_equal": torch.equal( + ring_tensor_scale, expected_tensor_scale, + ), + "fp4_difference_count": int( + (ring_qdata != expected_qdata).sum().item() + ), + "block_scale_difference_count": int( + (ring_block_scales.view(torch.uint8) + != expected_block_scales.view(torch.uint8)).sum().item() + ), + "output_equal": torch.equal(candidate, reference), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "comfy_output_equal": torch.equal( + comfy_candidate[: args.rows, : linear.out_features], reference, + ), + }, + "timing": { + "warmup": args.warmup, + "iterations": args.iterations, + "producer_ms": producer_ms, + "global_scale_ms": scale_ms, + "cute_consumer_ms": consumer_ms, + "modeled_cute_chunk_ms": modeled_cute_chunk_ms, + "modeled_comfy_chunk_ms": modeled_comfy_chunk_ms, + "actual_into_ring_cute_gemm_ms": actual_ring_ms, + "comfy_consumer_ms": comfy_consumer_ms, + "actual_into_ring_comfy_gemm_ms": comfy_ring_ms, + "reference_vortex_scale_comfy_pack_gemm_ms": reference_ms, + "modeled_canonical_reference_ms": modeled_reference_canonical_ms, + "modeled_canonical_cute_ring_ms": scale_ms + chunk_count * actual_ring_ms, + "modeled_canonical_comfy_ring_ms": scale_ms + chunk_count * comfy_ring_ms, + "actual_ring_vs_reference": actual_ring_ms / reference_ms, + "comfy_ring_vs_reference": comfy_ring_ms / reference_ms, + "modeled_canonical_cute_ring_vs_reference": ( + scale_ms + chunk_count * actual_ring_ms + ) / modeled_reference_canonical_ms, + "modeled_canonical_comfy_ring_vs_reference": ( + scale_ms + chunk_count * comfy_ring_ms + ) / modeled_reference_canonical_ms, + }, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_ring_full.py b/tools/validate_cute_nvfp4_ring_full.py new file mode 100644 index 0000000..f8774fc --- /dev/null +++ b/tools/validate_cute_nvfp4_ring_full.py @@ -0,0 +1,278 @@ +"""Validate complete H3 projections through a reusable NVFP4 row ring.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path + +import cutlass +import cutlass.cute as cute +import cutlass.torch as cutlass_torch +import torch +import torch.nn.functional as functional +from cutlass.cute.runtime import from_dlpack + +from h3_blackwell_runtime.nvfp4_quant import ( + nvfp4_activation_scale, + vortex_native_quantize_nvfp4, + vortex_native_quantize_nvfp4_into, + vortex_quantize_nvfp4, +) +from profile_nvfp4_linear import module_for_name, representative_inputs +from validate_cute_nvfp4_h3 import ( + fp4_tensor, + load_cutlass_example, + output_tensor, + scale_tensor, +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--cutlass-example", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--linears", + nargs="+", + choices=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), + default=("attn_qkv_proj", "attn_out_proj", "mlp_fc1"), + ) + parser.add_argument("--capacity", type=int, default=2048) + parser.add_argument("--warmup", type=int, default=1) + parser.add_argument("--iterations", type=int, default=3) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def measure_cuda(fn, *, warmup: int, iterations: int) -> tuple[object, float]: + result = None + for _ in range(warmup): + result = fn() + torch.cuda.synchronize() + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + for _ in range(iterations): + result = fn() + finished.record() + finished.synchronize() + return result, started.elapsed_time(finished) / iterations + + +def validate_linear(args, example, block, inputs, name: str) -> dict: + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + linear = module_for_name(block, name) + activation = inputs[name].reshape(-1, linear.in_features).contiguous() + rows, features = activation.shape + global_scale = nvfp4_activation_scale(activation).float() + packed_weight = linear._packed_weight() + b_qdata, tensor_scale_b, b_block_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(packed_weight) + ) + + with torch.inference_mode(): + reference_packed = vortex_quantize_nvfp4( + activation, scale=global_scale, + ) + reference = functional.linear(reference_packed, packed_weight, None) + ring_seed = vortex_native_quantize_nvfp4( + activation[: args.capacity], scale=global_scale, + ) + _, ring_tensor_scale, ring_block_scales = ( + TensorCoreNVFP4Layout.get_plain_tensors(ring_seed) + ) + + a, a_backing = cutlass_torch.cute_tensor_like( + torch.zeros( + args.capacity, + features, + 1, + device="cuda", + dtype=torch.float32, + ), + cutlass.Float4E2M1FN, + is_dynamic_layout=True, + assumed_align=16, + ) + ring_qdata = a_backing.view(torch.uint8).flatten()[ + : args.capacity * features // 2 + ].reshape(args.capacity, features // 2) + b, _ = fp4_tensor(b_qdata, swap_nibbles=False, reencode=True) + sfa = scale_tensor(ring_block_scales) + sfb = scale_tensor(b_block_scales) + chunks = [ + (start, min(start + args.capacity, rows)) + for start in range(0, rows, args.capacity) + ] + padded_rows = len(chunks) * args.capacity + candidate_padded = torch.zeros( + padded_rows, b_qdata.shape[0], device="cuda", dtype=torch.bfloat16, + ) + c_chunks = [ + output_tensor( + candidate_padded[ + index * args.capacity : (index + 1) * args.capacity + ] + ) + for index in range(len(chunks)) + ] + alpha = ring_tensor_scale.float() * tensor_scale_b.float() + alpha_argument = from_dlpack(alpha.reshape(1).contiguous(), assumed_align=4) + gemm = example.Sm120BlockScaledGemmKernel( + cutlass.Float32, 16, (128, 128, 128), (128, 128), + ) + max_active_clusters = cutlass.utils.HardwareInfo().get_max_active_clusters(1) + stream = cutlass_torch.default_stream() + compiled_gemm = cute.compile( + gemm, a, b, sfa, sfb, c_chunks[0], alpha_argument, + max_active_clusters, stream, + ) + + def run_chunks(): + for index, (start, end) in enumerate(chunks): + vortex_native_quantize_nvfp4_into( + activation[start:end], + global_scale, + ring_qdata, + ring_block_scales, + hi_first=False, + ) + compiled_gemm( + a, b, sfa, sfb, c_chunks[index], alpha_argument, stream, + ) + return candidate_padded + + def run_complete_ring(): + nvfp4_activation_scale(activation) + return run_chunks() + + chunk_reports = [] + for index, (start, end) in enumerate(chunks): + vortex_native_quantize_nvfp4_into( + activation[start:end], + global_scale, + ring_qdata, + ring_block_scales, + hi_first=False, + ) + compiled_gemm( + a, b, sfa, sfb, c_chunks[index], alpha_argument, stream, + ) + torch.cuda.synchronize() + candidate = candidate_padded[start:end, : linear.out_features] + expected = reference[start:end] + delta = candidate.float() - expected.float() + chunk_reports.append({ + "start": start, + "rows": end - start, + "equal": torch.equal(candidate, expected), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + }) + + complete_candidate = candidate_padded[:rows, : linear.out_features] + reference_checksum = reference.float().sum().item() + candidate_checksum = complete_candidate.float().sum().item() + output_bytes = candidate_padded.numel() * candidate_padded.element_size() + del reference + del reference_packed + torch.cuda.empty_cache() + + _, ring_ms = measure_cuda( + run_complete_ring, warmup=args.warmup, iterations=args.iterations, + ) + del candidate + del complete_candidate + del c_chunks + del candidate_padded + torch.cuda.empty_cache() + _, reference_ms = measure_cuda( + lambda: functional.linear( + vortex_quantize_nvfp4(activation), packed_weight, None, + ), + warmup=args.warmup, + iterations=args.iterations, + ) + + qdata_bytes = ring_qdata.numel() * ring_qdata.element_size() + sfa_bytes = ring_block_scales.numel() * ring_block_scales.element_size() + return { + "name": name, + "mnk": [rows, linear.out_features, features], + "capacity": args.capacity, + "chunk_count": len(chunks), + "final_chunk_rows": chunks[-1][1] - chunks[-1][0], + "ring_bytes": qdata_bytes + sfa_bytes, + "parity": { + "all_chunks_equal": all(chunk["equal"] for chunk in chunk_reports), + "max_abs": max(chunk["max_abs"] for chunk in chunk_reports), + "mean_abs_max": max(chunk["mean_abs"] for chunk in chunk_reports), + "reference_checksum": reference_checksum, + "candidate_checksum": candidate_checksum, + }, + "timing": { + "warmup": args.warmup, + "iterations": args.iterations, + "ring_complete_ms": ring_ms, + "reference_complete_ms": reference_ms, + "ring_vs_reference": ring_ms / reference_ms, + "improvement_percent": (1.0 - ring_ms / reference_ms) * 100.0, + }, + "chunks": chunk_reports, + "output_bytes": output_bytes, + } + + +def main() -> None: + args = parse_args() + if args.capacity <= 0 or args.capacity % 128: + raise ValueError("--capacity must be a positive multiple of 128") + if args.warmup < 0 or args.iterations <= 0: + raise ValueError("--warmup must be non-negative and --iterations positive") + + example = load_cutlass_example(args.cutlass_example, fuse_alpha=True) + block, inputs, metadata = representative_inputs(args) + results = [ + validate_linear(args, example, block, inputs, name) + for name in args.linears + ] + report = { + "device": torch.cuda.get_device_name(), + "cutlass_dsl": "4.6.2", + "metadata": metadata, + "block_index": args.block_index, + "capacity": args.capacity, + "all_equal": all(result["parity"]["all_chunks_equal"] for result in results), + "projection_reference_total_ms": sum( + result["timing"]["reference_complete_ms"] for result in results + ), + "projection_ring_total_ms": sum( + result["timing"]["ring_complete_ms"] for result in results + ), + "results": results, + } + report["projection_total_improvement_percent"] = ( + 1.0 + - report["projection_ring_total_ms"] + / report["projection_reference_total_ms"] + ) * 100.0 + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_nvfp4_tile_producer.py b/tools/validate_cute_nvfp4_tile_producer.py new file mode 100644 index 0000000..d03c79e --- /dev/null +++ b/tools/validate_cute_nvfp4_tile_producer.py @@ -0,0 +1,224 @@ +"""Validate a CuTe BF16-to-NVFP4 producer on one 128x128 activation tile.""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Optional + +import cutlass +import cutlass.cute as cute +import cutlass.torch as cutlass_torch +import torch +from cutlass import Float32 +from cutlass.cutlass_dsl import dsl_user_op +from cutlass._mlir import ir +from cutlass._mlir.dialects import llvm +from cutlass.cute.runtime import from_dlpack + +from h3_blackwell_runtime.nvfp4_quant import nvfp4_activation_scale + + +TILE = 128 +BLOCK = 16 +BLOCKS_PER_ROW = TILE // BLOCK +JOBS = TILE * BLOCKS_PER_ROW + + +@dsl_user_op +def rcp_approx_ftz_f32( + x: Float32, + *, + loc: Optional[ir.Location] = None, + ip: Optional[ir.InsertionPoint] = None, +) -> Float32: + result = llvm.inline_asm( + Float32.mlir_type, + [x.ir_value(loc=loc, ip=ip)], + "rcp.approx.ftz.f32 $0, $1;", + "=f,f", + has_side_effects=False, + asm_dialect=0, + loc=loc, + ip=ip, + ) + return Float32(result) + + +@cute.kernel +def tile_producer_kernel( + source: cute.Tensor, + tensor_scale: cute.Tensor, + fp4: cute.Tensor, + block_scales: cute.Tensor, + scalar_scales: cute.Tensor, +): + row = cute.arch.thread_idx()[0] + fp4_linear = cute.make_tensor( + fp4.iterator, + cute.make_layout((TILE * TILE,), stride=(1,)), + ) + fp4_tiles = cute.zipped_divide(fp4_linear, (8,)) + scale_tiles = cute.zipped_divide(block_scales, (8,)) + fp4_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float4E2M1FN) + fp8_store = cute.make_copy_atom(cute.nvgpu.CopyUniversalOp(), cutlass.Float8E4M3FN) + source_fragment = cute.make_rmem_tensor((16,), cutlass.Float32) + normalized_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) + fp4_fragment = cute.make_rmem_tensor((8,), cutlass.Float4E2M1FN) + scale_source = cute.make_rmem_tensor((8,), cutlass.Float32) + scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float8E4M3FN) + decoded_scale_fragment = cute.make_rmem_tensor((8,), cutlass.Float32) + scale = tensor_scale[0] + + for block_column in cutlass.range_constexpr(BLOCKS_PER_ROW): + job = row * BLOCKS_PER_ROW + block_column + column = block_column * BLOCK + maximum = cutlass.Float32(0.0) + for element in cutlass.range_constexpr(BLOCK): + value = source[row, column + element] + source_fragment[element] = value + maximum = cutlass.max(cutlass.max(value, -value), maximum) + + raw_block_scale = (maximum / cutlass.Float32(6.0)) / scale + for element in cutlass.range_constexpr(8): + scale_source[element] = raw_block_scale + scale_values = scale_source.load() + scale_values = cute.where( + scale_values <= cutlass.Float32(448.0), + scale_values, + cutlass.Float32(448.0), + ) + scale_fragment.store(scale_values.to(cutlass.Float8E4M3FN)) + cute.copy(fp8_store, scale_fragment, scale_tiles[(None, job)]) + scalar_scales[job] = scale_fragment[0] + decoded_scale_fragment.store(scale_fragment.load().to(cutlass.Float32)) + decoded_scale = decoded_scale_fragment[0] + raw_encode_scale = rcp_approx_ftz_f32(decoded_scale * scale) + for element in cutlass.range_constexpr(8): + scale_source[element] = raw_encode_scale + encode_scale_values = scale_source.load() + encode_scale_values = cute.where( + encode_scale_values <= cutlass.Float32(3.402823466e38), + encode_scale_values, + cutlass.Float32(3.402823466e38), + ) + scale_source.store(encode_scale_values) + encode_scale = scale_source[0] + + for half in cutlass.range_constexpr(2): + for element in cutlass.range_constexpr(8): + normalized = source_fragment[half * 8 + element] * encode_scale + normalized_fragment[element] = normalized + fp4_fragment.store(normalized_fragment.load().to(cutlass.Float4E2M1FN)) + output_tile = job * 2 + half + cute.copy(fp4_store, fp4_fragment, fp4_tiles[(None, output_tile)]) + + +@cute.jit +def produce_tile( + source: cute.Tensor, + tensor_scale: cute.Tensor, + fp4: cute.Tensor, + block_scales: cute.Tensor, + scalar_scales: cute.Tensor, +): + tile_producer_kernel(source, tensor_scale, fp4, block_scales, scalar_scales).launch( + grid=(1, 1, 1), block=(TILE, 1, 1), + ) + + +def unswizzle_scales(physical: torch.Tensor) -> torch.Tensor: + rows, scale_columns = physical.shape + row = torch.arange(rows, device=physical.device).view(-1, 1) + block_column = torch.arange(scale_columns, device=physical.device).view(1, -1) + row_in_tile = row % 128 + tile = (row // 128) * (scale_columns // 4) + block_column // 4 + within = ( + ((row_in_tile % 32) // 2) * 32 + + block_column % 4 + + (row_in_tile // 32) * 4 + + (row_in_tile % 2) * 16 + ) + return physical.flatten()[(tile * 512 + within).long()] + + +def main() -> None: + import comfy_kitchen as ck + + torch.manual_seed(440420) + random_values = torch.randn(TILE, TILE, device="cuda", dtype=torch.bfloat16) + zero_values = torch.zeros_like(random_values) + sparse_values = torch.zeros_like(random_values) + sparse_values.flatten()[:16] = torch.tensor( + [-100, -6, -4, -3, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 3, 6, 100], + device="cuda", + dtype=torch.bfloat16, + ) + source_torch = torch.empty_like(random_values) + scale_torch = torch.empty(1, device="cuda", dtype=torch.float32) + + source = from_dlpack(source_torch, assumed_align=16).mark_layout_dynamic(leading_dim=1) + scale = from_dlpack(scale_torch, assumed_align=4).mark_layout_dynamic() + fp4, fp4_torch = cutlass_torch.cute_tensor_like( + torch.zeros_like(source_torch, dtype=torch.float32), + cutlass.Float4E2M1FN, + is_dynamic_layout=True, + assumed_align=16, + ) + block_scales_torch = torch.zeros(JOBS, 8, device="cuda", dtype=torch.uint8) + block_scales = from_dlpack(block_scales_torch.flatten(), assumed_align=16) + block_scales.element_type = cutlass.Float8E4M3FN + block_scales = block_scales.mark_layout_dynamic() + scalar_scales_torch = torch.zeros(JOBS, device="cuda", dtype=torch.uint8) + scalar_scales = from_dlpack(scalar_scales_torch, assumed_align=16) + scalar_scales.element_type = cutlass.Float8E4M3FN + scalar_scales = scalar_scales.mark_layout_dynamic() + + compiled = cute.compile(produce_tile, source, scale, fp4, block_scales, scalar_scales) + cases = [] + for name, values in (("random", random_values), ("zeros", zero_values), ("sparse_extremes", sparse_values)): + source_torch.copy_(values) + scale_torch.copy_(nvfp4_activation_scale(source_torch).float().reshape(1)) + expected_qdata, expected_physical_scales = ck.quantize_nvfp4( + source_torch, scale_torch, pad_16x=False, + ) + expected_low_first = ((expected_qdata & 0x0F) << 4) | ((expected_qdata & 0xF0) >> 4) + expected_scales = unswizzle_scales(expected_physical_scales.view(torch.uint8)) + compiled(source, scale, fp4, block_scales, scalar_scales) + torch.cuda.synchronize() + actual_fp4 = fp4_torch.view(torch.uint8).flatten()[:TILE * TILE // 2].reshape(TILE, TILE // 2) + actual_scales = block_scales_torch[:, 0].reshape(TILE, BLOCKS_PER_ROW) + cases.append({ + "name": name, + "tensor_scale": scale_torch.item(), + "fp4_difference_count": int((actual_fp4 != expected_low_first).sum().item()), + "fp4_equal": torch.equal(actual_fp4, expected_low_first), + "block_scale_difference_count": int((actual_scales != expected_scales).sum().item()), + "block_scales_equal": torch.equal(actual_scales, expected_scales), + "scalar_block_scales_equal": torch.equal( + scalar_scales_torch.reshape(TILE, BLOCKS_PER_ROW), expected_scales, + ), + "actual_block_scale_bytes": actual_scales.unique().tolist(), + "expected_block_scale_bytes": expected_scales.unique().tolist(), + }) + + report = { + "device": torch.cuda.get_device_name(), + "cutlass_dsl": "4.6.2", + "tile": [TILE, TILE], + "all_equal": all( + case["fp4_equal"] + and case["block_scales_equal"] + and case["scalar_block_scales_equal"] + for case in cases + ), + "cases": cases, + } + output = Path("/output/h3-blackwell-runtime/benchmarks/gb10-cute-nvfp4-tile-producer.json") + output.parent.mkdir(parents=True, exist_ok=True) + output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_qkv_block.py b/tools/validate_cute_qkv_block.py new file mode 100644 index 0000000..84811c1 --- /dev/null +++ b/tools/validate_cute_qkv_block.py @@ -0,0 +1,157 @@ +"""Alternate baseline and CuTe-QKV execution inside one loaded H3 block.""" + +from __future__ import annotations + +import argparse +import json +import os +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.block import H3DiTBlock +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--block-index", type=int, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument( + "--feature", choices=("qkv_ring", "modulate_fusion", "swiglu_fusion"), default="qkv_ring", + ) + parser.add_argument("--warmup", type=int, default=2) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def sync() -> None: + torch.cuda.synchronize() + + +def summarize(values: list[float]) -> dict[str, float]: + ordered = sorted(values) + middle = len(ordered) // 2 + median = ( + ordered[middle] + if len(ordered) % 2 + else (ordered[middle - 1] + ordered[middle]) / 2 + ) + return { + "mean_s": sum(values) / len(values), + "p50_s": median, + "min_s": ordered[0], + "max_s": ordered[-1], + } + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + block = H3DiTBlock.from_checkpoint( + checkpoint, args.block_index, attention_backend=args.attention, + ).eval() + adaln = H3CurveAdaLN.from_checkpoint( + checkpoint, f"blocks.{args.block_index}.adaln_proj", + ).eval() + packer = H3PromptPacker(checkpoint) + video, audio, _ = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + sigmas = beta_sigmas(args.steps, device=args.device) + sigma = sigmas[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rotation = h3_rope_rotation( + positions.to(args.device), + checkpoint.tensor("rope.inv_freq", dtype=torch.float32), + hidden.dtype, + ) + adaln_values = tuple(value.detach() for value in adaln(timesteps)) + + def run(enabled: bool): + if args.feature == "modulate_fusion": + block.fused_nvfp4_modulation = enabled + elif args.feature == "swiglu_fusion": + block.mlp.fused_nvfp4_swiglu = enabled + else: + if enabled: + os.environ["H3_CUTE_QKV_RING"] = "1" + else: + os.environ.pop("H3_CUTE_QKV_RING", None) + return block(hidden, rotation, *adaln_values, segments) + + with torch.inference_mode(): + reference = run(False) + candidate = run(True) + sync() + delta = candidate.float() - reference.float() + for _ in range(args.warmup): + run(False) + run(True) + sync() + baseline_times = [] + candidate_times = [] + last_reference = reference + last_candidate = candidate + for _ in range(args.iterations): + sync() + started = time.perf_counter() + last_reference = run(False) + sync() + baseline_times.append(time.perf_counter() - started) + + sync() + started = time.perf_counter() + last_candidate = run(True) + sync() + candidate_times.append(time.perf_counter() - started) + + baseline = summarize(baseline_times) + candidate_timing = summarize(candidate_times) + report = { + "device": torch.cuda.get_device_name(), + "block_index": args.block_index, + "feature": args.feature, + "hidden_shape": list(hidden.shape), + "iterations": args.iterations, + "equal": torch.equal(reference, candidate), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "reference_checksum": last_reference.float().sum().item(), + "candidate_checksum": last_candidate.float().sum().item(), + "baseline": baseline, + "candidate": candidate_timing, + "p50_improvement_percent": ( + 1.0 - candidate_timing["p50_s"] / baseline["p50_s"] + ) * 100.0, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_cute_qkv_runtime.py b/tools/validate_cute_qkv_runtime.py new file mode 100644 index 0000000..b6b2e74 --- /dev/null +++ b/tools/validate_cute_qkv_runtime.py @@ -0,0 +1,90 @@ +"""Validate the opt-in Nvfp4Linear CuTe QKV runtime dispatch.""" + +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path + +import torch + +from profile_nvfp4_linear import representative_inputs + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--rows", type=int, default=2048) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def measure(fn, warmup: int, iterations: int) -> float: + for _ in range(warmup): + fn() + torch.cuda.synchronize() + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + for _ in range(iterations): + fn() + end.record() + end.synchronize() + return start.elapsed_time(end) / iterations + + +def main() -> None: + args = parse_args() + block, inputs, metadata = representative_inputs(args) + linear = block.attention.qkv_proj + x = inputs["attn_qkv_proj"][: args.rows].contiguous() + if linear.role != "h3_attn_qkv": + raise RuntimeError(f"Expected h3_attn_qkv role, got {linear.role!r}") + + os.environ.pop("H3_CUTE_QKV_RING", None) + with torch.inference_mode(): + reference = linear(x) + os.environ["H3_CUTE_QKV_RING"] = "1" + with torch.inference_mode(): + candidate = linear(x) + torch.cuda.synchronize() + delta = candidate.float() - reference.float() + + with torch.inference_mode(): + ring_ms = measure(lambda: linear(x), args.warmup, args.iterations) + os.environ.pop("H3_CUTE_QKV_RING", None) + reference_ms = measure(lambda: linear(x), args.warmup, args.iterations) + + report = { + "device": torch.cuda.get_device_name(), + "metadata": metadata, + "block_index": args.block_index, + "rows": args.rows, + "role": linear.role, + "equal": torch.equal(candidate, reference), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "ring_ms": ring_ms, + "reference_ms": reference_ms, + "improvement_percent": (1.0 - ring_ms / reference_ms) * 100.0, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/validate_experiment_registry.py b/tools/validate_experiment_registry.py new file mode 100644 index 0000000..d33bbe1 --- /dev/null +++ b/tools/validate_experiment_registry.py @@ -0,0 +1,234 @@ +#!/usr/bin/env python3 +"""Validate research/experiment_registry.json without external dependencies.""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections import Counter +from pathlib import Path +from typing import Any + + +REQUIRED_STATUSES = { + "production_accepted", + "research_retained", + "architecture_rejected", + "performance_rejected", + "quality_rejected", + "temporarily_blocked", + "incomplete", +} + +REQUIRED_FIELDS = { + "id", + "name", + "family", + "status", + "hypothesis", + "implementation_strategy", + "source_locations", + "active_source_location", + "commit_hash", + "benchmark_artifacts", + "profiler_artifacts", + "environment", + "metrics", + "correctness_evidence", + "decision_rationale", + "reproducer_commands", + "timestamp", + "evidence_missing", + "production_behavior", + "source_recovery", +} + +NONEMPTY_STRINGS = { + "id", + "name", + "family", + "status", + "hypothesis", + "implementation_strategy", + "decision_rationale", +} + + +def type_name(value: Any) -> str: + if value is None: + return "null" + if isinstance(value, bool): + return "boolean" + if isinstance(value, dict): + return "object" + if isinstance(value, list): + return "array" + return type(value).__name__ + + +def validate_path_metadata(value: Any, location: str, errors: list[str]) -> None: + if not isinstance(value, dict): + errors.append(f"{location}: expected path metadata object") + return + path = value.get("path") + exists = value.get("exists") + if not isinstance(path, str) or not path.strip(): + errors.append(f"{location}.path: expected non-empty string") + if not isinstance(exists, bool): + errors.append(f"{location}.exists: expected boolean") + + +def validate_path_array(value: Any, location: str, errors: list[str]) -> None: + if not isinstance(value, list): + errors.append(f"{location}: expected array, got {type_name(value)}") + return + for index, item in enumerate(value): + validate_path_metadata(item, f"{location}[{index}]", errors) + + +def validate_source_array(value: Any, location: str, errors: list[str]) -> None: + validate_string_array(value, location, errors) + + +def validate_string_array(value: Any, location: str, errors: list[str]) -> None: + if not isinstance(value, list): + errors.append(f"{location}: expected array, got {type_name(value)}") + return + for index, item in enumerate(value): + if not isinstance(item, str) or not item.strip(): + errors.append(f"{location}[{index}]: expected non-empty string") + + +def validate_record(record: Any, index: int, seen_ids: set[str], errors: list[str]) -> None: + location = f"experiments[{index}]" + if not isinstance(record, dict): + errors.append(f"{location}: expected object, got {type_name(record)}") + return + + missing = sorted(REQUIRED_FIELDS - record.keys()) + if missing: + errors.append(f"{location}: missing required fields: {', '.join(missing)}") + + for field in NONEMPTY_STRINGS: + if field not in record: + continue + value = record[field] + if not isinstance(value, str) or not value.strip(): + errors.append(f"{location}.{field}: expected non-empty string") + + experiment_id = record.get("id") + if isinstance(experiment_id, str) and experiment_id.strip(): + if experiment_id in seen_ids: + errors.append(f"{location}.id: duplicate id {experiment_id!r}") + seen_ids.add(experiment_id) + + status = record.get("status") + if isinstance(status, str) and status not in REQUIRED_STATUSES: + errors.append(f"{location}.status: unknown status {status!r}") + + if "source_locations" in record: + validate_source_array(record["source_locations"], f"{location}.source_locations", errors) + if "active_source_location" in record and record["active_source_location"] is not None: + value = record["active_source_location"] + if not isinstance(value, str) or not value.strip(): + errors.append(f"{location}.active_source_location: expected non-empty string or null") + for field in ("benchmark_artifacts", "profiler_artifacts"): + if field in record: + validate_path_array(record[field], f"{location}.{field}", errors) + + if "commit_hash" in record and record["commit_hash"] is not None: + if not isinstance(record["commit_hash"], str) or not record["commit_hash"].strip(): + errors.append(f"{location}.commit_hash: expected non-empty string or null") + + for field in ("environment", "metrics"): + if field in record and not isinstance(record[field], dict): + errors.append(f"{location}.{field}: expected object, got {type_name(record[field])}") + + if "timestamp" in record and record["timestamp"] is not None: + if not isinstance(record["timestamp"], str) or not record["timestamp"].strip(): + errors.append(f"{location}.timestamp: expected non-empty string or null") + + if "source_recovery" in record: + value = record["source_recovery"] + if not isinstance(value, str) or not value.strip(): + errors.append(f"{location}.source_recovery: expected non-empty string") + + for field in ("correctness_evidence", "production_behavior"): + if field in record and record[field] is not None and not isinstance( + record[field], (dict, list, str) + ): + errors.append(f"{location}.{field}: expected object, array, string, or null") + + if "reproducer_commands" in record: + validate_string_array(record["reproducer_commands"], f"{location}.reproducer_commands", errors) + + if "evidence_missing" in record: + value = record["evidence_missing"] + if not isinstance(value, (bool, list)): + errors.append(f"{location}.evidence_missing: expected boolean or array") + elif isinstance(value, list): + validate_string_array(value, f"{location}.evidence_missing", errors) + + +def validate_registry(data: Any) -> tuple[list[str], Counter[str]]: + errors: list[str] = [] + counts: Counter[str] = Counter() + if not isinstance(data, dict): + return [f"top level: expected object, got {type_name(data)}"], counts + + allowed = data.get("allowed_statuses") + if not isinstance(allowed, list) or any(not isinstance(item, str) for item in allowed): + errors.append("allowed_statuses: expected array of strings") + elif len(allowed) != len(set(allowed)) or set(allowed) != REQUIRED_STATUSES: + expected = ", ".join(sorted(REQUIRED_STATUSES)) + errors.append(f"allowed_statuses: must contain exactly: {expected}") + + experiments = data.get("experiments") + if not isinstance(experiments, list): + errors.append(f"experiments: expected array, got {type_name(experiments)}") + return errors, counts + + seen_ids: set[str] = set() + for index, record in enumerate(experiments): + validate_record(record, index, seen_ids, errors) + if isinstance(record, dict) and record.get("status") in REQUIRED_STATUSES: + counts[record["status"]] += 1 + return errors, counts + + +def main() -> int: + default_registry = Path(__file__).resolve().parents[1] / "research" / "experiment_registry.json" + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("registry", nargs="?", type=Path, default=default_registry) + args = parser.parse_args() + + try: + with args.registry.open("r", encoding="utf-8") as handle: + data = json.load(handle) + except FileNotFoundError: + print(f"error: registry not found: {args.registry}", file=sys.stderr) + return 1 + except (OSError, UnicodeError) as exc: + print(f"error: cannot read registry: {exc}", file=sys.stderr) + return 1 + except json.JSONDecodeError as exc: + print(f"error: invalid JSON at line {exc.lineno}, column {exc.colno}: {exc.msg}", file=sys.stderr) + return 1 + + errors, counts = validate_registry(data) + if errors: + for error in errors: + print(f"error: {error}", file=sys.stderr) + return 1 + + experiments = data["experiments"] + status_counts = ", ".join( + f"{status}={counts[status]}" for status in sorted(REQUIRED_STATUSES) + ) + print(f"valid: experiments={len(experiments)}; statuses: {status_counts}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/validate_h3_fusion.py b/tools/validate_h3_fusion.py new file mode 100644 index 0000000..8f84a8b --- /dev/null +++ b/tools/validate_h3_fusion.py @@ -0,0 +1,153 @@ +"""Validate fused H3 elementwise kernels on real checkpoint blocks.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.block import H3DiTBlock +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents +from profile_h3_block import summarize + + +def sync() -> None: + torch.cuda.synchronize() + + +def timed(fn, warmup: int, iterations: int) -> dict[str, float]: + with torch.inference_mode(): + for _ in range(warmup): + fn() + values = [] + for _ in range(iterations): + sync() + started = time.perf_counter() + fn() + sync() + values.append(time.perf_counter() - started) + return summarize(values) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--blocks", nargs="+", type=int, default=(0, 24, 49)) + parser.add_argument("--attention", default="sage2") + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + packer = H3PromptPacker(checkpoint) + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rotation = h3_rope_rotation( + positions.to(args.device), checkpoint.tensor("rope.inv_freq", dtype=torch.float32), hidden.dtype, + ) + + results = [] + for block_index in args.blocks: + if not 0 <= block_index < 50: + raise ValueError("block indices must be in [0, 49]") + block = H3DiTBlock.from_checkpoint( + checkpoint, block_index, attention_backend=args.attention, + ).eval() + adaln = H3CurveAdaLN.from_checkpoint( + checkpoint, f"blocks.{block_index}.adaln_proj", + ).eval() + modulation = tuple(value.detach() for value in adaln(timesteps)) + + with torch.inference_mode(): + block.fused_elementwise = False + expected = block(hidden.clone(), rotation, *modulation, segments) + block.fused_elementwise = True + actual = block(hidden.clone(), rotation, *modulation, segments) + delta = actual.float() - expected.float() + parity = { + "equal": torch.equal(actual, expected), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "reference_checksum": expected.float().sum().item(), + "fused_checksum": actual.float().sum().item(), + } + if not parity["equal"]: + raise RuntimeError(f"block {block_index} fused output is not bit-exact: {parity}") + + block.fused_elementwise = False + eager_timing = timed( + lambda: block(hidden.clone(), rotation, *modulation, segments), + args.warmup, + args.iterations, + ) + block.fused_elementwise = True + fused_timing = timed( + lambda: block(hidden.clone(), rotation, *modulation, segments), + args.warmup, + args.iterations, + ) + results.append({ + "block": block_index, + "parity": parity, + "eager_timing": eager_timing, + "fused_timing": fused_timing, + "p50_speedup": eager_timing["p50_s"] / fused_timing["p50_s"], + "p50_latency_reduction": 1.0 - fused_timing["p50_s"] / eager_timing["p50_s"], + }) + print( + f"block {block_index}: exact, eager={eager_timing['p50_s'] * 1000:.3f}ms, " + f"fused={fused_timing['p50_s'] * 1000:.3f}ms, " + f"speedup={results[-1]['p50_speedup']:.3f}x", + flush=True, + ) + + report = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "resolution": [args.width, args.height], + "frames": aligned_frames, + "packed_tokens": hidden.shape[0], + "steps": args.steps, + "sampler_step": args.sampler_step, + "seed": args.seed, + "attention": args.attention, + "warmup": args.warmup, + "iterations": args.iterations, + "results": results, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_nvfp4_modulate_producer.py b/tools/validate_nvfp4_modulate_producer.py new file mode 100644 index 0000000..a17b7f2 --- /dev/null +++ b/tools/validate_nvfp4_modulate_producer.py @@ -0,0 +1,140 @@ +"""Validate fused H3 modulation and native NVFP4 production.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.attention import rms_norm +from h3_blackwell_runtime.block import H3DiTBlock, modulate_segments +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.h3_fusion import segment_index +from h3_blackwell_runtime.nvfp4_quant import ( + nvfp4_activation_scale, + vortex_native_quantize_modulated_nvfp4, +) +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--block-index", type=int, default=24) + parser.add_argument("--rows", type=int, default=2048) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + block = H3DiTBlock.from_checkpoint(checkpoint, args.block_index, attention_backend="sage2").eval() + adaln = H3CurveAdaLN.from_checkpoint( + checkpoint, f"blocks.{args.block_index}.adaln_proj", + ).eval() + packer = H3PromptPacker(checkpoint) + video, audio, _ = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + hidden, timesteps, segments, _, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rows = min(args.rows, hidden.shape[0]) + hidden = hidden[:rows].contiguous() + clipped_segments = [] + for start, stop, table_row in segments: + if start >= rows: + break + clipped_segments.append((start, min(stop, rows), table_row)) + shift_msa, scale_msa, *_ = (value.detach() for value in adaln(timesteps)) + + with torch.inference_mode(): + normalized = rms_norm(hidden, block.norm1_weight, block.norm_eps) + materialized = modulate_segments( + normalized, shift_msa, scale_msa, clipped_segments, + ) + reference_scale = nvfp4_activation_scale(materialized).float() + import comfy_kitchen as ck + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + reference_qdata, reference_sfa = ck.quantize_nvfp4( + materialized, + reference_scale, + pad_16x=TensorCoreNVFP4Layout.get_padded_shape(tuple(materialized.shape)) + != tuple(materialized.shape), + ) + fused_scale, fused_qdata, fused_sfa = vortex_native_quantize_modulated_nvfp4( + normalized, + shift_msa.contiguous(), + scale_msa.contiguous(), + segment_index(rows, clipped_segments, hidden.device), + ) + torch.cuda.synchronize() + + def run_fused(): + return vortex_native_quantize_modulated_nvfp4( + normalized, + shift_msa.contiguous(), + scale_msa.contiguous(), + segment_index(rows, clipped_segments, hidden.device), + ) + + for _ in range(args.warmup): + run_fused() + fused_times = [] + for _ in range(args.iterations): + torch.cuda.synchronize() + started = time.perf_counter() + run_fused() + torch.cuda.synchronize() + fused_times.append(time.perf_counter() - started) + + report = { + "device": torch.cuda.get_device_name(), + "block_index": args.block_index, + "rows": rows, + "width": hidden.shape[1], + "scale_equal": torch.equal(fused_scale, reference_scale), + "scale_reference": reference_scale.item(), + "scale_fused": fused_scale.item(), + "qdata_equal": torch.equal(fused_qdata, reference_qdata), + "qdata_differences": torch.count_nonzero(fused_qdata != reference_qdata).item(), + "sfa_equal": torch.equal(fused_sfa.view(torch.uint8), reference_sfa.view(torch.uint8)), + "sfa_differences": torch.count_nonzero( + fused_sfa.view(torch.uint8) != reference_sfa.view(torch.uint8) + ).item(), + "fused_producer_p50_ms": sorted(fused_times)[len(fused_times) // 2] * 1000.0, + } + report["equal"] = report["scale_equal"] and report["qdata_equal"] and report["sfa_equal"] + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if not report["equal"]: + raise RuntimeError("fused modulation producer is not byte-exact") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_nvfp4_modulate_trajectory.py b/tools/validate_nvfp4_modulate_trajectory.py new file mode 100644 index 0000000..a19ef6a --- /dev/null +++ b/tools/validate_nvfp4_modulate_trajectory.py @@ -0,0 +1,129 @@ +"""Compare baseline and fused-modulation H3 sampling in one resident model.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.denoiser import H3PackedDenoiser +from h3_blackwell_runtime.lora import load_lora_adapter, set_active_lora +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.sampler import sample_video_res_multistep +from h3_blackwell_runtime.t2v import random_av_latents +from h3_blackwell_runtime.token_refiner import H3TokenRefiner + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=2) + parser.add_argument("--warmup-steps", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--attention", default="sage2") + parser.add_argument( + "--feature", choices=("modulate_fusion", "swiglu_fusion", "lora_producer_fusion"), + default="modulate_fusion", + ) + parser.add_argument("--lora-path") + parser.add_argument("--lora-name", default="validation") + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + model = H3PackedDenoiser.from_checkpoint( + checkpoint, output_dtype=torch.bfloat16, attention_backend=args.attention, + ).eval() + refiner = None + if args.lora_path: + refiner = H3TokenRefiner(checkpoint, attention_backend=args.attention).eval() + load_lora_adapter(model, refiner, args.lora_name, args.lora_path, args.device) + set_active_lora(model, refiner, args.lora_name) + packer = H3PromptPacker(checkpoint) + if hasattr(checkpoint, "release_cache"): + checkpoint.release_cache() + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + + def run(enabled: bool, steps: int): + for block in model.backbone.blocks: + if args.feature == "lora_producer_fusion": + block.fused_nvfp4_modulation = enabled + block.mlp.fused_nvfp4_swiglu = enabled + elif args.feature == "swiglu_fusion": + block.mlp.fused_nvfp4_swiglu = enabled + else: + block.fused_nvfp4_modulation = enabled + torch.cuda.synchronize() + started = time.perf_counter() + result = sample_video_res_multistep( + model, + packer, + text, + video.clone(), + audio.clone(), + steps=steps, + seed=args.seed, + return_audio=True, + progress=True, + ) + torch.cuda.synchronize() + return result, time.perf_counter() - started + + with torch.inference_mode(): + if args.warmup_steps: + run(False, args.warmup_steps) + run(True, args.warmup_steps) + (reference_video, reference_audio), baseline_s = run(False, args.steps) + (candidate_video, candidate_audio), candidate_s = run(True, args.steps) + + video_delta = candidate_video.float() - reference_video.float() + audio_delta = candidate_audio.float() - reference_audio.float() + report = { + "device": torch.cuda.get_device_name(), + "resolution": [args.width, args.height], + "frames": aligned_frames, + "steps": args.steps, + "feature": args.feature, + "seed": args.seed, + "baseline_seconds": baseline_s, + "candidate_seconds": candidate_s, + "improvement_percent": (1.0 - candidate_s / baseline_s) * 100.0, + "video_equal": torch.equal(candidate_video, reference_video), + "audio_equal": torch.equal(candidate_audio, reference_audio), + "video_max_abs": video_delta.abs().max().item(), + "audio_max_abs": audio_delta.abs().max().item(), + "reference_checksums": [ + reference_video.float().sum().item(), reference_audio.float().sum().item(), + ], + "candidate_checksums": [ + candidate_video.float().sum().item(), candidate_audio.float().sum().item(), + ], + } + report["equal"] = report["video_equal"] and report["audio_equal"] + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if not report["equal"]: + raise RuntimeError("fused modulation trajectory is not bit-exact") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_nvfp4_swiglu_producer.py b/tools/validate_nvfp4_swiglu_producer.py new file mode 100644 index 0000000..7650115 --- /dev/null +++ b/tools/validate_nvfp4_swiglu_producer.py @@ -0,0 +1,135 @@ +"""Validate fused SwiGLU and native NVFP4 production on real H3 FC1 output.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.block import H3DiTBlock +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.nvfp4_quant import ( + nvfp4_activation_scale, + vortex_native_quantize_swiglu_nvfp4, +) +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--block-index", type=int, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + block = H3DiTBlock.from_checkpoint( + checkpoint, args.block_index, attention_backend="sage2", + ).eval() + adaln = H3CurveAdaLN.from_checkpoint( + checkpoint, f"blocks.{args.block_index}.adaln_proj", + ).eval() + packer = H3PromptPacker(checkpoint) + video, audio, _ = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn( + 1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16, + ) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rotation = h3_rope_rotation( + positions.to(args.device), + checkpoint.tensor("rope.inv_freq", dtype=torch.float32), + hidden.dtype, + ) + modulation = tuple(value.detach() for value in adaln(timesteps)) + captured = [] + block.fused_nvfp4_modulation = False + hook = block.mlp.fc1.register_forward_hook( + lambda _module, _inputs, output: captured.append(output.detach()), + ) + with torch.inference_mode(): + block(hidden, rotation, *modulation, segments) + hook.remove() + gate_up = captured[-1] + + with torch.inference_mode(): + gate, up = gate_up.chunk(2, dim=-1) + materialized = torch.nn.functional.silu(gate).mul_(up) + reference_scale = nvfp4_activation_scale(materialized).float() + import comfy_kitchen as ck + from comfy_kitchen.tensor import TensorCoreNVFP4Layout + + reference_qdata, reference_sfa = ck.quantize_nvfp4( + materialized, + reference_scale, + pad_16x=TensorCoreNVFP4Layout.get_padded_shape(tuple(materialized.shape)) + != tuple(materialized.shape), + ) + actual_scale, actual_qdata, actual_sfa = vortex_native_quantize_swiglu_nvfp4( + gate_up, + ) + torch.cuda.synchronize() + for _ in range(args.warmup): + vortex_native_quantize_swiglu_nvfp4(gate_up) + times = [] + for _ in range(args.iterations): + torch.cuda.synchronize() + started = time.perf_counter() + vortex_native_quantize_swiglu_nvfp4(gate_up) + torch.cuda.synchronize() + times.append(time.perf_counter() - started) + + report = { + "device": torch.cuda.get_device_name(), + "block_index": args.block_index, + "input_shape": list(gate_up.shape), + "output_shape": list(materialized.shape), + "scale_equal": torch.equal(actual_scale, reference_scale), + "scale_reference": reference_scale.item(), + "scale_actual": actual_scale.item(), + "qdata_differences": torch.count_nonzero(actual_qdata != reference_qdata).item(), + "sfa_differences": torch.count_nonzero( + actual_sfa.view(torch.uint8) != reference_sfa.view(torch.uint8) + ).item(), + "producer_p50_ms": sorted(times)[len(times) // 2] * 1000.0, + } + report["equal"] = ( + report["scale_equal"] + and report["qdata_differences"] == 0 + and report["sfa_differences"] == 0 + ) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if not report["equal"]: + raise RuntimeError("fused SwiGLU producer is not byte-exact") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_sage2_entry_fusion.py b/tools/validate_sage2_entry_fusion.py new file mode 100644 index 0000000..b19ae7a --- /dev/null +++ b/tools/validate_sage2_entry_fusion.py @@ -0,0 +1,335 @@ +"""Validate exact fused Sage2 entry preparation on randomized and real H3 tensors.""" + +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.attention import rms_norm, rms_rope_split_half_ +from h3_blackwell_runtime.block import H3DiTBlock, modulate_segments +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sage2_entry import prepare_qk +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents +from profile_attention_path import summarize +from profile_sage2_scheduler import run_mainloop + + +def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: + delta = actual.float() - expected.float() + result = { + "equal": torch.equal(actual, expected), + "different_elements": int(torch.count_nonzero(actual != expected).item()), + "max_abs": delta.abs().max().item() if delta.numel() else 0.0, + "mean_abs": delta.abs().mean().item() if delta.numel() else 0.0, + } + if not result["equal"] and result["different_elements"] <= 8: + indices = torch.nonzero(actual != expected, as_tuple=False) + result["mismatches"] = [ + { + "index": index.tolist(), + "actual": actual[tuple(index)].item(), + "expected": expected[tuple(index)].item(), + } + for index in indices + ] + return result + + +def measure(fn, warmup: int, iterations: int) -> dict: + for _ in range(warmup): + fn() + torch.cuda.synchronize() + before_allocated = torch.cuda.memory_allocated() + before_reserved = torch.cuda.memory_reserved() + torch.cuda.reset_peak_memory_stats() + samples = [] + for _ in range(iterations): + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + fn() + finished.record() + finished.synchronize() + samples.append(started.elapsed_time(finished) / 1000.0) + return { + "timing": summarize(samples), + "allocated_bytes_before": before_allocated, + "reserved_bytes_before": before_reserved, + "peak_allocated_bytes": torch.cuda.max_memory_allocated(), + "peak_reserved_bytes": torch.cuda.max_memory_reserved(), + "peak_allocated_delta_bytes": torch.cuda.max_memory_allocated() - before_allocated, + "peak_reserved_delta_bytes": torch.cuda.max_memory_reserved() - before_reserved, + } + + +def views(qkv: torch.Tensor, heads: int) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + sequence = qkv.shape[0] + inner = heads * 128 + return tuple(part.view(1, sequence, heads, 128) for part in qkv.split(inner, dim=-1)) + + +def q_quant(q: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + import sageattention.quant as sage_quant + + groups = math.ceil(q.shape[1] / 128) * 4 + output = torch.empty(q.shape, dtype=torch.int8, device=q.device) + scale = torch.empty((q.shape[0], q.shape[2], groups), dtype=torch.float32, device=q.device) + sage_quant._fused.quant_per_warp_int8_cuda(q, output, scale, 128, 32, 0) + return output, scale + + +def k_quant(k: torch.Tensor, mean: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: + import sageattention.quant as sage_quant + + output = torch.empty(k.shape, dtype=torch.int8, device=k.device) + scale = torch.empty( + (k.shape[0], k.shape[2], math.ceil(k.shape[1] / 64)), + dtype=torch.float32, + device=k.device, + ) + sage_quant._fused.quant_per_block_int8_fuse_sub_mean_cuda( + k, mean.squeeze(1), output, scale, 64, 0, + ) + return output, scale + + +def randomized_case(sequence: int, heads: int, seed: int) -> dict: + generator = torch.Generator(device="cuda").manual_seed(seed) + qkv = torch.randn( + (sequence, heads * 128 * 3), generator=generator, device="cuda", dtype=torch.bfloat16, + ) + rotation = torch.randn( + (1, sequence, 1, 48, 2, 2), generator=generator, device="cuda", dtype=torch.bfloat16, + ) + q_weight = torch.randn((128,), generator=generator, device="cuda", dtype=torch.bfloat16) + k_weight = torch.randn((128,), generator=generator, device="cuda", dtype=torch.bfloat16) + + reference_storage = qkv.clone() + q_reference, k_reference, _ = views(reference_storage, heads) + rms_rope_split_half_(q_reference, k_reference, rotation, q_weight, k_weight, 1e-5) + q_int8_reference, q_scale_reference = q_quant(q_reference) + + candidate_storage = qkv.clone() + q_candidate, k_candidate, _ = views(candidate_storage, heads) + q_int8_candidate, q_scale_candidate, q_prepared_candidate = prepare_qk( + q_candidate, + k_candidate, + rotation, + q_weight, + k_weight, + 1e-5, + materialize_q=True, + ) + return { + "sequence": sequence, + "heads": heads, + "q_prepared": difference(q_prepared_candidate, q_reference), + "k_prepared": difference(k_candidate, k_reference), + "q_int8": difference(q_int8_candidate, q_int8_reference), + "q_scale": difference(q_scale_candidate, q_scale_reference), + } + + +def real_case(block: H3DiTBlock, hidden: torch.Tensor, rotation: torch.Tensor, modulation, segments, warmup: int, iterations: int) -> dict: + shift_msa, scale_msa, *_ = modulation + h_msa = modulate_segments( + rms_norm(hidden, block.norm1_weight, block.norm_eps), shift_msa, scale_msa, segments, + ) + raw_qkv = block.attention.qkv_proj(h_msa) + heads = block.attention.heads + + reference_storage = raw_qkv.clone() + q_reference, k_reference, v_reference = views(reference_storage, heads) + rms_rope_split_half_( + q_reference, + k_reference, + rotation, + block.attention.q_norm_weight, + block.attention.k_norm_weight, + block.attention.eps, + ) + q_int8_reference, q_scale_reference = q_quant(q_reference) + k_mean_reference = k_reference.mean(dim=1, keepdim=True) + k_int8_reference, k_scale_reference = k_quant(k_reference, k_mean_reference) + + candidate_storage = raw_qkv.clone() + q_candidate, k_candidate, v_candidate = views(candidate_storage, heads) + q_int8_candidate, q_scale_candidate, q_prepared_candidate = prepare_qk( + q_candidate, + k_candidate, + rotation, + block.attention.q_norm_weight, + block.attention.k_norm_weight, + block.attention.eps, + materialize_q=True, + ) + k_mean_candidate = k_candidate.mean(dim=1, keepdim=True) + k_int8_candidate, k_scale_candidate = k_quant(k_candidate, k_mean_candidate) + + import sageattention.core as sage_core + + v_fp8, v_scale, _ = sage_core.per_channel_fp8( + v_reference, tensor_layout="NHD", scale_max=2.25, smooth_v=False, + ) + reference_output = torch.empty_like(q_reference) + candidate_output = torch.empty_like(q_reference) + run_mainloop( + q_int8_reference, + k_int8_reference, + v_fp8, + q_scale_reference, + k_scale_reference, + v_scale, + reference_output, + ) + run_mainloop( + q_int8_candidate, + k_int8_candidate, + v_fp8, + q_scale_candidate, + k_scale_candidate, + v_scale, + candidate_output, + ) + torch.cuda.synchronize() + + baseline_timing_storage = raw_qkv.clone() + baseline_q, baseline_k, _ = views(baseline_timing_storage, heads) + + def baseline_entry(): + rms_rope_split_half_( + baseline_q, + baseline_k, + rotation, + block.attention.q_norm_weight, + block.attention.k_norm_weight, + block.attention.eps, + ) + return q_quant(baseline_q) + + candidate_timing_storage = raw_qkv.clone() + candidate_q, candidate_k, _ = views(candidate_timing_storage, heads) + + def candidate_entry(): + return prepare_qk( + candidate_q, + candidate_k, + rotation, + block.attention.q_norm_weight, + block.attention.k_norm_weight, + block.attention.eps, + materialize_q=False, + ) + + return { + "block": block.block_index if hasattr(block, "block_index") else None, + "q_shape": list(q_reference.shape), + "q_stride": list(q_reference.stride()), + "q_prepared": difference(q_prepared_candidate, q_reference), + "k_prepared": difference(k_candidate, k_reference), + "q_int8": difference(q_int8_candidate, q_int8_reference), + "q_scale": difference(q_scale_candidate, q_scale_reference), + "k_mean": difference(k_mean_candidate, k_mean_reference), + "k_int8": difference(k_int8_candidate, k_int8_reference), + "k_scale": difference(k_scale_candidate, k_scale_reference), + "attention_output": difference(candidate_output, reference_output), + "baseline_entry": measure(baseline_entry, warmup, iterations), + "candidate_entry": measure(candidate_entry, warmup, iterations), + } + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--blocks", nargs="+", type=int, default=(0, 24, 49)) + parser.add_argument("--randomized-only", action="store_true") + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + randomized = [ + randomized_case(sequence, 2, args.seed + sequence) + for sequence in (1, 31, 32, 33, 127, 128, 129) + ] + real = [] + metadata = {} + if not args.randomized_only: + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device="cuda") + packer = H3PromptPacker(checkpoint) + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device="cuda", + ) + sigma = beta_sigmas(args.steps, device="cuda")[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn(1, args.text_tokens, 5376, device="cuda", dtype=torch.bfloat16) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rotation = h3_rope_rotation( + positions.to("cuda"), + checkpoint.tensor("rope.inv_freq", dtype=torch.float32), + hidden.dtype, + ) + for block_index in args.blocks: + block = H3DiTBlock.from_checkpoint(checkpoint, block_index, attention_backend="sage2").eval() + block.block_index = block_index + adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{block_index}.adaln_proj").eval() + with torch.inference_mode(): + real.append(real_case( + block, + hidden, + rotation, + tuple(value.detach() for value in adaln(timesteps)), + segments, + args.warmup, + args.iterations, + )) + metadata = { + "resolution": [args.width, args.height], + "frames": aligned_frames, + "tokens": hidden.shape[0], + "seed": args.seed, + "blocks": args.blocks, + } + report = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "metadata": metadata, + "randomized": randomized, + "real": real, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + parity_items = randomized + real + if any( + not value["equal"] + for item in parity_items + for key, value in item.items() + if isinstance(value, dict) and "equal" in value + ): + raise RuntimeError("Sage2 entry fusion parity failed") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_sage2_vprep.py b/tools/validate_sage2_vprep.py new file mode 100644 index 0000000..e644d63 --- /dev/null +++ b/tools/validate_sage2_vprep.py @@ -0,0 +1,118 @@ +"""Validate Vortex direct Sage2 V preparation against SageAttention 2.2.0.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import torch + +from h3_blackwell_runtime.sage2_entry import prepare_v +from profile_attention_path import summarize + + +def difference(actual: torch.Tensor, expected: torch.Tensor) -> dict: + delta = actual.float() - expected.float() + return { + "equal": torch.equal(actual, expected), + "different_elements": int(torch.count_nonzero(actual != expected).item()), + "max_abs": delta.abs().max().item() if delta.numel() else 0.0, + "mean_abs": delta.abs().mean().item() if delta.numel() else 0.0, + } + + +def measure(fn, *, warmup: int, iterations: int) -> dict: + for _ in range(warmup): + fn() + torch.cuda.synchronize() + samples = [] + for _ in range(iterations): + started = torch.cuda.Event(enable_timing=True) + finished = torch.cuda.Event(enable_timing=True) + started.record() + fn() + finished.record() + finished.synchronize() + samples.append(started.elapsed_time(finished) / 1000.0) + return summarize(samples) + + +def baseline(v: torch.Tensor): + import sageattention.core as sage_core + + return sage_core.per_channel_fp8( + v, tensor_layout="NHD", scale_max=2.25, smooth_v=False, + ) + + +def run_case(sequence: int, heads: int, seed: int, warmup: int, iterations: int) -> dict: + generator = torch.Generator(device="cuda").manual_seed(seed) + storage = torch.randn( + (sequence, heads * 128 * 3), + generator=generator, + device="cuda", + dtype=torch.bfloat16, + ) + v = storage[:, heads * 128 * 2 :].view(1, sequence, heads, 128) + reference_fp8, reference_scale, _ = baseline(v) + candidate_fp8, candidate_scale = prepare_v(v) + torch.cuda.synchronize() + result = { + "sequence": sequence, + "heads": heads, + "stride": list(v.stride()), + "fp8": difference(candidate_fp8, reference_fp8), + "scale": difference(candidate_scale, reference_scale), + } + if sequence >= 1024: + result["baseline_timing"] = measure( + lambda: baseline(v), warmup=warmup, iterations=iterations, + ) + result["candidate_timing"] = measure( + lambda: prepare_v(v), warmup=warmup, iterations=iterations, + ) + return result + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument( + "--lengths", + nargs="+", + type=int, + default=(1, 31, 32, 33, 63, 64, 65, 127, 128, 129, 37760, 37761, 37810), + ) + parser.add_argument("--heads", type=int, default=56) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + cases = [ + run_case(length, args.heads, args.seed + index, args.warmup, args.iterations) + for index, length in enumerate(args.lengths) + ] + report = { + "status": "pass" if all(case["fp8"]["equal"] and case["scale"]["equal"] for case in cases) else "fail", + "contract": { + "tensor_layout": "NHD", + "head_dim": 128, + "scale_max": 2.25, + "output": "Sage2 padded/permuted E4M3 V and FP32 per-channel scale", + }, + "cases": cases, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + print(json.dumps(report, indent=2), flush=True) + if report["status"] != "pass": + raise RuntimeError("Vortex Sage2 V preparation did not match SageAttention 2.2.0") + + +if __name__ == "__main__": + main() diff --git a/tools/validate_sage_layout.py b/tools/validate_sage_layout.py new file mode 100644 index 0000000..ff0d1b4 --- /dev/null +++ b/tools/validate_sage_layout.py @@ -0,0 +1,144 @@ +"""Validate projection-strided Sage2 NHD execution on real H3 blocks.""" + +from __future__ import annotations + +import argparse +import json +import os +import time +from pathlib import Path + +import torch + +from h3_blackwell_runtime.adaln import H3CurveAdaLN +from h3_blackwell_runtime.block import H3DiTBlock +from h3_blackwell_runtime.checkpoint import H3Checkpoint +from h3_blackwell_runtime.packing import H3PromptPacker +from h3_blackwell_runtime.rope import h3_rope_rotation +from h3_blackwell_runtime.sampler import _audio_sigma, _model_sigma, beta_sigmas +from h3_blackwell_runtime.t2v import random_av_latents +from profile_h3_block import summarize + + +def timed(fn, warmup: int, iterations: int) -> dict[str, float]: + with torch.inference_mode(): + for _ in range(warmup): + fn() + values = [] + for _ in range(iterations): + torch.cuda.synchronize() + started = time.perf_counter() + fn() + torch.cuda.synchronize() + values.append(time.perf_counter() - started) + return summarize(values) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--model-path", default="/models/minimax_h3_fl2va_pruned_nvfp4.safetensors") + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--width", type=int, default=1344) + parser.add_argument("--height", type=int, default=768) + parser.add_argument("--frames", type=int, default=124) + parser.add_argument("--steps", type=int, default=12) + parser.add_argument("--sampler-step", type=int, default=1) + parser.add_argument("--seed", type=int, default=440420) + parser.add_argument("--text-tokens", type=int, default=100) + parser.add_argument("--blocks", nargs="+", type=int, default=(0, 24, 49)) + parser.add_argument("--warmup", type=int, default=3) + parser.add_argument("--iterations", type=int, default=10) + parser.add_argument("--device", default="cuda") + return parser.parse_args() + + +def main() -> None: + args = parse_args() + torch.manual_seed(args.seed) + checkpoint = H3Checkpoint(args.model_path, device=args.device) + packer = H3PromptPacker(checkpoint) + video, audio, aligned_frames = random_av_latents( + args.width, args.height, args.frames, args.seed, device=args.device, + ) + sigma = beta_sigmas(args.steps, device=args.device)[args.sampler_step - 1] + native_audio = audio.to(torch.bfloat16) * (_audio_sigma(sigma) / sigma) + text = torch.randn(1, args.text_tokens, 5376, device=args.device, dtype=torch.bfloat16) + hidden, timesteps, segments, positions, _, _ = packer( + text, video, native_audio, _model_sigma(sigma), + ) + rotation = h3_rope_rotation( + positions.to(args.device), checkpoint.tensor("rope.inv_freq", dtype=torch.float32), hidden.dtype, + ) + + results = [] + original_layout = os.environ.get("H3_SAGE_QKV_LAYOUT") + try: + for block_index in args.blocks: + if not 0 <= block_index < 50: + raise ValueError("block indices must be in [0, 49]") + block = H3DiTBlock.from_checkpoint(checkpoint, block_index, attention_backend="sage2").eval() + block.fused_elementwise = True + adaln = H3CurveAdaLN.from_checkpoint(checkpoint, f"blocks.{block_index}.adaln_proj").eval() + modulation = tuple(value.detach() for value in adaln(timesteps)) + run = lambda: block(hidden.clone(), rotation, *modulation, segments) + + with torch.inference_mode(): + os.environ["H3_SAGE_QKV_LAYOUT"] = "hnd" + expected = run() + os.environ["H3_SAGE_QKV_LAYOUT"] = "strided_nhd" + actual = run() + delta = actual.float() - expected.float() + parity = { + "equal": torch.equal(actual, expected), + "max_abs": delta.abs().max().item(), + "mean_abs": delta.abs().mean().item(), + "reference_checksum": expected.float().sum().item(), + "candidate_checksum": actual.float().sum().item(), + } + if not parity["equal"]: + raise RuntimeError(f"block {block_index} Sage NHD output is not bit-exact: {parity}") + + os.environ["H3_SAGE_QKV_LAYOUT"] = "hnd" + hnd_timing = timed(run, args.warmup, args.iterations) + os.environ["H3_SAGE_QKV_LAYOUT"] = "strided_nhd" + nhd_timing = timed(run, args.warmup, args.iterations) + results.append({ + "block": block_index, + "parity": parity, + "hnd_timing": hnd_timing, + "strided_nhd_timing": nhd_timing, + "p50_speedup": hnd_timing["p50_s"] / nhd_timing["p50_s"], + "p50_latency_reduction": 1.0 - nhd_timing["p50_s"] / hnd_timing["p50_s"], + }) + print( + f"block {block_index}: exact, hnd={hnd_timing['p50_s'] * 1000:.3f}ms, " + f"strided_nhd={nhd_timing['p50_s'] * 1000:.3f}ms, speedup={results[-1]['p50_speedup']:.3f}x", + flush=True, + ) + finally: + if original_layout is None: + os.environ.pop("H3_SAGE_QKV_LAYOUT", None) + else: + os.environ["H3_SAGE_QKV_LAYOUT"] = original_layout + + report = { + "device": torch.cuda.get_device_name(), + "torch": torch.__version__, + "resolution": [args.width, args.height], + "frames": aligned_frames, + "packed_tokens": hidden.shape[0], + "steps": args.steps, + "sampler_step": args.sampler_step, + "seed": args.seed, + "attention": "sage2", + "fused_elementwise": True, + "warmup": args.warmup, + "iterations": args.iterations, + "results": results, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8") + + +if __name__ == "__main__": + main()