daniel/h3-blackwell-runtime
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Preserve Vortex performance research history

Browse source
This commit is contained in:
Daniel Maddern 2026-08-25 20:30:22 +07:00
parent bee998d840
commit ea14b98544
259 changed files with 57540 additions and 101 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

237
CURRENT_STATE.md
View file

@ -1,6 +1,6 @@
# H3 Runtime Current State
Status date: 2026-08-22
Status date: 2026-08-24
This document is the canonical snapshot of implemented scope and remaining work.
Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
@ -26,6 +26,10 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
- Ragged Ulysses sequence parallelism with 2/4/6/8-rank transport tests.
- Sequence-sharded 50-block execution and distributed final projection.
- True H3 NVFP4 tensor parallelism for attention QKV/output and MLP FC1/FC2.
- Bit-exact fused H3 modulation and residual gates on GB10: `3.0%` faster over
the canonical 12-step 1344x768/124-frame sampling run with identical video
and audio checksums, `3.6-3.9%` faster individual blocks, approximately 390
MiB lower peak allocated memory, and 32 passing deployed tests.
- Automatic visible-GPU launchers and 1/2/4/6/8 benchmark matrix tooling.
- Real-checkpoint one-rank Ulysses-versus-TP identity at 864x480, 141 frames,
and 12 steps, including exact video and audio latent equality.
@ -66,7 +70,216 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
- H3-specific attention backend optimized for real GB10 tensor shapes.
- Blackwell-native CUTLASS/CuTe or cuBLASLt NVFP4 GEMMs.
- CUDA graph capture and shape buckets.
- Fused Q/K RMSNorm, RoPE, and layout work on the Sage2 quality path.
- A four-GEMM NVFP4 roofline study for QKV, attention output, FC1, and FC2.
The active NVFP4 fusion profile is now measured on one canonical GB10 block:
32 of 53 launches belong to the four scale/pack/GEMM paths. Native packed data,
block scales, and linear outputs are exact at every H3 projection width after
generalizing the block-scale swizzle. The next implementation must remove
intermediate traffic across these exact boundaries rather than deploy the
standalone packer, which is not consistently faster than Comfy Kitchen.
The active subcomponent is direct QKV projection into Sage2's required layout,
followed by direct attention output into the NVFP4 output projection's
token-major layout. The existing `scaled_mm_nvfp4` wrapper exposes only a
contiguous BF16 output, so true copy elimination requires either a supported
strided epilogue from its underlying CUTLASS kernel or an owned projection
collective. A post-GEMM copy kernel is useful only as a diagnostic and does not
satisfy this boundary-removal target. For single-GPU Sage2, the supported
strided NHD interface made that post-GEMM kernel unnecessary: Q/K/V remain
strided views of the interleaved projection output and Sage emits contiguous
token-major-compatible NHD output. The bit-exact path reduces canonical sampling
from `301.05 s` to `290.23 s` and is enabled for Spark deployments with
`H3_SAGE_QKV_LAYOUT=strided_nhd`.
NVFP4 streaming feasibility is confirmed but not yet deployable. Comfy Kitchen's
cuBLAS interface requires complete activation and scale pointers, while CUTLASS
DSL 4.6.2 runs block-scaled FP4 on SM121 and accepts the same logical H3 data.
The experimental alpha-before-BF16 epilogue is bit-exact for QKV, attention
output, and FC1 on 128-row real tiles. FC2 still differs because its reference
uses a different reduction policy, so no streamed producer is enabled.
The explicit implementation policy is to retain FC2 on cuBLAS and develop the
streamed CuTe path only for QKV, attention output, and FC1. This is a numerical
fallback, not a silent compatibility path: FC2's reference reduction order is
part of the exactness contract.
The fixed 128-row P1 producer-consumer checkpoint is complete. The existing
32-thread DMA warp now produces four BF16 activation rows per lane directly into
the owned GEMM's staged E2M1 A and E4M3 SFA shared-memory layouts. B/SFB remain
on TMA, and their completion publishes the stage to the unchanged MMA consumer.
No complete global activation QDATA or SFA tensor is passed to the streamed
kernel. Every real 128-K tile and the complete BF16 GEMM output are bit-exact for
QKV, attention output, and FC1. Matching Comfy requires its `--use_fast_math`
`rcp.approx.ftz.f32` encode-scale operation. FC2 streaming is explicitly
rejected and retains the cuBLAS fallback. The prototype is validator-only and
still needs canonical M/padding support and runtime packaging.
See `benchmarks/gb10-cute-p1-stream-a-summary.json`.
The first timing gate rejects direct per-CTA streaming. For 128 rows, the exact
streamed kernel is `9.14-12.98x` slower than the complete Vortex-scale plus
Comfy-pack/GEMM reference because every output-N CTA rereads and repacks A.
Measured producer cost is approximately `0.56-0.60 us` per `(N,K)` tile, and
break-even would require reusing A across 28-102 N tiles. Duplicating that many
accumulators is not viable. The active design is now a bounded global packed-tile
ring or persistent work queue that produces each `(M,K)` tile once, shares it
across N consumers, and recycles the slot without materializing the complete
activation. See `benchmarks/gb10-cute-p1-stream-a-timing-summary.json`.
The bounded-ring implementation uses caller-owned native QDATA/SFA buffers and
an allocation-free `_into` producer. A capacity sweep with one full-activation
scale selected 2048 rows: 6.19 MB for QKV/FC1 and 8.26 MB for attention output.
At that capacity, complete 37,810-row projection parity is exact for QKV,
attention output, and FC1 in blocks 0, 24, and 49, including the final 946-row
chunk. The subsequent single-model alternating gate rejects runtime QKV dispatch:
all three block outputs are bit-exact, but median block time regresses by `0.52%`
to `0.81%`. QKV capacity checks at 3072, 4096, 8192, and 37888 rows also fail to
produce a block-level gain; 4096 is closest at `0.24%` slower. Attention output
and FC1 remain experimental, and FC2 remains on Comfy/cuBLAS. Do not run
trajectory validation or enable the backend until launch fusion or a different
persistent scheduler passes this block gate. See
`benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`.
The fresh post-optimization canonical run is `288.93 s` with unchanged video
and audio checksums. Block 24 is `468.22 ms` median, of which production NHD
Sage2 attention consumes `258.47 ms`. Internal attribution places `238.81 ms`
in the SM89 attention mainloop, versus `7.68 ms` Q/K quantization and `10.65 ms`
V quantization. The deployed SM121 path therefore still spends most of the
block in an Ada-style MMA kernel. Recompiling SageAttention's Hopper WGMMA
mainloop for SM121 is not possible: CUDA 13 ptxas rejects WGMMA instructions for
`sm_121a`. CUTLASS SM120/121 UMMA supports F8/F6/F4, not the INT8 QK operation
required for exact Sage2 parity, so a native exact attention rewrite is paused.
The next practical boundary was the two approximately 10.5 ms AdaLN modulation
passes. Their exact BF16 values now feed NVFP4 scale/pack for QKV and FC1 without
materializing the modulated values, while retaining the current Comfy GEMMs.
The producer is byte-exact for complete block 0, 24, and 49 inputs. Integrated
block medians improve by `0.28-0.79%`; warmed two-step and canonical 12-step
trajectories improve by `0.52%` and `0.56%`, respectively, with bit-identical
video and audio tensors. Spark enables the path with
`H3_NVFP4_MODULATE_FUSION=1`. See `NVFP4_MODULATE_FUSION_DESIGN.md`,
`SAGE2_BLACKWELL_DESIGN.md`, and
`benchmarks/gb10-post-optimization-profile-summary.json`.
The Spark hot runtime was rebuilt and recreated with image
`sha256:5f879c43374bcedb89745971d7d95d82afc8fcf9c41d30f11b257c2863b9fe28`.
Health and startup warmup pass with modulation fusion enabled. A resident real
generation smoke completed in `2.14 s` (`0.227 s` sampling) and produced a valid
22-frame 320x192 H.264 file. The first startup warmup includes one-time CUDA
extension compilation. See
`benchmarks/gb10-nvfp4-modulate-fusion-deployment-smoke.json`.
Exact SwiGLU-to-FC2 NVFP4 producer fusion is also complete. It preserves the
two reference BF16 boundaries, emits byte-identical tensor scale/QDATA/SFA, and
retains the exact Comfy FC2 GEMM. Blocks 0, 24, and 49 improve by `2.14-2.26%`.
The warmed canonical 12-step run improves from `289.14 s` to `277.36 s`
(`4.07%`) with bit-identical video and audio tensors. Spark enables it with
`H3_NVFP4_SWIGLU_FUSION=1`. See `NVFP4_SWIGLU_FUSION_DESIGN.md` and
`benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`.
Optional BF16 materialization inside both fused producers was tested for active
Turbo LoRA requests. The isolated canonical Turbo-4 trajectory was bit-exact,
but regressed from `131.11 s` to `135.14 s` (`3.08%`), so the prototype was
rejected. Active LoRA retains the exact materialized fallback instead of using
either producer fusion. See
`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`.
The active Spark image is now
`sha256:1d340e14cb6fc45ccfdbe63dde8db2a2b3aea94b493702c8a08e1f8d5b4f7b83`
with both accepted NVFP4 producer fusions enabled for eligible base requests.
Health, startup warmup, three deployed CUDA parity tests, and a resident real
generation smoke pass. The final smoke completed in `2.321 s` with `0.217 s`
sampling and produced a valid 22-frame 320x192 H.264 file. See
`benchmarks/gb10-post-lora-revert-deployment-smoke.json`.
A fresh fully fused Nsight recapture now supersedes the old approximately
515 ms block profile. Block 24 is `458.78 ms` median uninstrumented and
`465.78 ms` across the Nsight GPU span, with 41 kernels and only `0.084 ms` of
inter-kernel idle time. Sage2 is `57.60%` of kernel time, the four NVFP4 GEMMs
are `26.87%`, packing is `8.92%`, norm/RoPE is `4.27%`, and the two remaining
gate/add kernels are `2.35%`. A complete warmed step takes `23.708 s` and shows
the same distribution. Hardware counters attribute `56.25%` of the warm-cache
off-chip request proxy to the NVFP4 GEMMs even though Sage2 remains the time
bottleneck. See `benchmarks/gb10-fully-fused-fresh-nsight-summary.json`.
The real block-24 Sage2 scheduler study and exact SM89 P0 retune are complete.
Manual preparation plus the
unchanged prequantized SM89 mainloop is byte-exact against public SageAttention
2.2.0. Uninstrumented medians are `2.37 ms` for K mean/smoothing, `3.80 ms` for
Q quantization, `3.83 ms` for K subtract-mean quantization, `5.08 ms` for V
transpose/pad/permute, `5.53 ms` for V scale/FP8 quantization, and `237.09 ms`
for the fused mainloop. Nsight Compute reports 255 registers/thread, 32 KiB
dynamic shared memory/CTA, `16.83%` achieved occupancy, and no eligible warp in
`63.53%` of scheduler cycles. INT8 QK and FP8 PV each use `37.77%` of elapsed
tensor-pipe capacity; combined tensor activity is `75.54%`. The kernel is
scheduler/compute limited rather than off-chip-bandwidth limited: L2 hit rate is
`98.84%`, while fixed-latency dependency and math-pipe stalls dominate. Tail
CTAs add less than 1 ms.
The P0 mapped the exact register cliff: caps from 255 through 170 registers
remain at two CTAs and `16.67%` theoretical occupancy; only 168 registers reaches
three CTAs and `25%`, while generating `4.95 billion` local spill requests and
worsening no-eligible cycles to `78.79%`. Narrowed scopes reduced static spills
from 44/44 to 12/12 bytes and dynamic spill requests from `1.46 million` to
`0.40 million`, but changed interleaved latency by only `+0.06%` and worsened
no-eligible cycles. In-place score reuse, early K prefetch, and independent
softmax-chain interleaving were also byte-exact and neutral or slower.
The `7.68%` shared excess maps entirely to repeated V-staging `LDGSTS.128`
instructions. Padding V to a 128-byte shared stride increased shared memory to
40 KiB but left all `626,970,624` excessive wavefronts unchanged and changed
latency by `-0.04%`. No variant crossed the 3% complete-block gate, so none was
integrated or deployed. See
`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json` and the associated
P0 latency JSON and NCU reports.
The exact Sage2 entry-fusion P1 is also complete and rejected. A single CUDA
kernel fused strided-NHD Q/K RMSNorm, split-half RoPE, and Sage2 Q INT8
quantization while leaving K mean/quantization, V preparation, and the SM89
mainloop unchanged. Randomized edge lengths and canonical block 0/24/49 tensors
were bit-exact through prepared Q/K, Q/K quantization, scales, K mean, Sage2
output, and complete block output. Entry-only median latency improved by
`20.9-23.4%`, but canonical complete-block median improvement was only `0.73%`,
`0.86%`, and `0.53%` for blocks 0, 24, and 49. The candidate removed one launch
(`87` to `86`), did not change peak memory, and reduced complete-block L2 traffic
by only `0.136-0.155%`. It therefore failed the required 1% gate; the opt-in
runtime branch was removed and two-step/12-step validation was skipped. See
`benchmarks/gb10-sage2-p1-entry-fusion-analysis.json` and its referenced parity,
timing, and Nsight reports.
The exact Sage2 V-preparation P2 is complete and rejected at its isolated gate.
An owned three-stage CUDA path consumes projection-strided NHD BF16 V and emits
Sage2's padded/permuted E4M3 V plus FP32 per-channel scales without materializing
the approximately 517 MiB BF16 transpose tensor. FP8 bytes and scales are exact
for 13 boundary lengths from 1 through 37,810 tokens with 56 heads. On the
canonical shape, median V preparation improves from `10.56 ms` to `6.39 ms`
(`39.48%`), but the `4.17 ms` absolute saving projects to only `0.91%` of the
`458.78 ms` complete block and misses the required `6.0 ms` isolated go gate.
Complete-block and trajectory validation were therefore skipped, and production
dispatch remains unchanged. See
`benchmarks/gb10-sage2-p2-vprep-analysis.json`.
The exact Sage2 mainloop P3 temporal-pair experiment is also complete and
rejected. Two warp pairs alternated QK/online-softmax and prior-tile PV while
retaining private per-warp scores, softmax state, and output accumulators. The
isolated extension is sanitizer-clean and byte-exact over 13 adversarial short
shapes plus the real 37,810-token block-24 SHA. In a 50-sample alternating run,
mainloop median changed from `245.44 ms` to `245.20 ms`, only `0.10%`, and
missed the absolute `<220 ms` gate. Ptxas reports 254 registers/thread and
32/24-byte static store/load spills versus baseline 255 registers and 24/24-byte
spills. NCU, block integration, and trajectory validation were skipped.
Production remains unchanged. See
`benchmarks/gb10-sage2-p3-temporal-pair-analysis.json` and
`research/sage2_temporal_pair/`.
The follow-on four-GEMM NVFP4 roofline study selects MLP FC2 as the next exact
kernel target. QKV, attention output, and FC1 sustain `315-321` dense-equivalent
TFLOP/s and sit above the practical GB10 compute/memory ridge point. FC2 instead
uses the stream-K kernel, sustains only `106.02` TFLOP/s, and generates a
`12.16 GB` warm-cache off-chip request proxy, `16.11x` its ideal unique bytes.
Its measured arithmetic intensity is `479` FLOP/byte versus a practical ridge
near `1,177` FLOP/byte. The next experiment should reduce FC2 K-split rereads and
reuse fused SwiGLU producer tiles; the other three GEMMs are not first targets.
See `benchmarks/gb10-four-gemm-nvfp4-roofline.json`.
## Quality Work Remaining
@ -108,13 +321,19 @@ Historical handoffs in `PLAN.md` and `PARITY.md` may describe older states.
## Recommended Execution Order
1. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
2. Correct the inactive fused Sol path.
3. Add explicit task schemas and automated single-GPU mode tests.
4. Implement full Ref2VA, including reference-audio encoding.
5. Build the H3-specific attention backend and CUDA graph buckets.
6. Harden the service API and operational deployment.
7. Complete RunPod NCCL validation and distributed scaling benchmarks.
1. Run the four-GEMM NVFP4 roofline study on QKV, attention output, FC1, and FC2.
2. Capture matched SM120 and SM100 component profiles and package Sage2 on SM120.
3. Resume NVFP4 GEMM/epilogue work only with a design that preserves the
accepted producer fusions and exact BF16 boundaries.
4. Validate full quoted-dialogue video lip-sync and close the audio prompt change.
5. Add explicit task schemas and automated single-GPU mode tests.
6. Implement full Ref2VA, including reference-audio encoding.
7. Add CUDA graph buckets after the kernel and shape policies stabilize.
8. Harden the service API and operational deployment.
9. Complete RunPod NCCL validation and distributed scaling benchmarks.
See `PERFORMANCE_ROADMAP.md` for measured component costs, architecture-specific
targets, quality gates, and the rationale for this ordering.
The current single-GPU T2VA/FL2VA runtime is mature. Distributed execution is
implemented and CPU/one-GPU validated, with real multi-GPU NCCL results still

47
DISTRIBUTED.md
View file

@ -93,6 +93,46 @@ replace multi-GPU NCCL parity testing.
## RunPod
### Regional volume bootstrap
Prepare the same benchmark assets in any RunPod data center with one guarded
command. The tool creates or reuses a network volume, attaches it to a temporary
two-vCPU pod, downloads the two pinned Hugging Face artifacts, verifies their
SHA-256 hashes, uploads a `git archive` of the current commit, writes a manifest,
and terminates the CPU pod in a `finally` block.
Set credentials in the process environment; do not pass secrets on the command
line or commit them:
```powershell
$env:RUNPOD_API_KEY = "..."
$env:HF_TOKEN = "..."
python .\tools\prepare_runpod_region.py --region US-MO-2 --dry-run
python .\tools\prepare_runpod_region.py --region US-MO-2 `
--volume-name h3-vortex-us-mo2 --yes
```
Resume an interrupted preparation without creating another volume:
```powershell
python .\tools\prepare_runpod_region.py --region US-MO-2 `
--volume-id VOLUME_ID --yes
```
Matching files are skipped by checksum. Partial `aria2` downloads resume in
place. The resulting volume contains:
- `ComfyUI/models/diffusion_models/minimax_h3_fl2va_pruned_nvfp4.safetensors`
- `ComfyUI/models/text_encoders/qwen3vl_32b_minimax_h3_nvfp4_awq.safetensors`
- `h3-runtime/h3-blackwell-runtime-<commit>.tar.gz`
- `h3-runtime/regional-bootstrap-manifest.json`
- `h3-runtime/artifacts-ready`
The archive intentionally contains committed files only. The manifest records a
dirty-worktree flag so an operator can see when local uncommitted changes were
excluded. CPU compute is automatically terminated on success, failure, or
Ctrl+C; the network volume is retained for GPU pods.
The target is one eight-GPU RTX PRO 6000 Blackwell machine. Query current stock
using the guarded API v2 client:
@ -149,6 +189,13 @@ and NCCL 2.27.7 selected `P2P/CUMEM` with GDR enabled and zero NVLS channels.
RunPod allocations varied between cross-NUMA `SYS` and same-NUMA `NODE`
topologies, so topology must be recorded with every scaling result.
One B200 in US-NC-2 completed the same 1344x768 SDPA tensor benchmarks in
`79.72s` at 124 frames and `257.86s` at 243 frames. Relative to one RTX PRO
6000, these are `1.23x` and `1.21x` speedups, but sampling cost is `2.64x` and
`2.69x` higher at the measured `$6.79/hour` B200 rate. The full timings,
checksums, memory peaks, and comparison are in
`benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`.
## Remaining GPU Gates
- Real NCCL identity at 4, 6, and 8 GPUs; two-GPU transport is complete.

4
Dockerfile.spark
View file

@ -31,6 +31,10 @@ RUN git clone https://github.com/Saganaki22/ComfyUI-sol-attn.git /opt/ComfyUI-so
RUN python -m pip install --no-cache-dir --no-deps -e . \
&& python -c "import comfy_kitchen, torch; from sageattn3 import sageattn3_blackwell; assert hasattr(torch.ops.comfy_kitchen, 'rms_rope_split_half_'); assert hasattr(comfy_kitchen, 'int8_attention'); assert hasattr(comfy_kitchen, 'int8_attention_is_available'); print(torch.__version__, torch.version.cuda)"
ENV TORCH_EXTENSIONS_DIR=/opt/h3-blackwell-runtime/.torch_extensions
RUN TORCH_CUDA_ARCH_LIST=12.1 python -c \
"from h3_blackwell_runtime.nvfp4_quant import _vortex_scale_extension; print(_vortex_scale_extension().__file__)"
ENV H3_MODEL_PATH=/models/minimax_h3_ref2va_pruned_nvfp4.safetensors
ENV PYTHONPATH=/opt/ComfyUI-sol-attn
ENV TORCH_COMPILE_DISABLE=0 TORCHDYNAMO_DISABLE=0

54
NVFP4_MODULATE_FUSION_DESIGN.md Normal file
View file

@ -0,0 +1,54 @@
# NVFP4 Modulation Producer Fusion
## Target
Fuse the two per-block AdaLN modulation operations into the exact NVFP4
activation scale and packing producer for attention QKV and MLP FC1. Retain the
current Comfy Kitchen GEMMs and all BF16 numerical boundaries.
At the canonical 37,810-token shape, block 24 spends a median 10.47 ms on MSA
modulation and 10.60 ms on MLP modulation. Production already mutates the two
RMSNorm outputs in place, so this fusion targets memory traffic rather than a
separate allocation.
## Dataflow
Current:
1. Read normalized BF16 input and write modulated BF16 output.
2. Read modulated output to compute the global NVFP4 activation scale.
3. Read modulated output again to produce packed E2M1 data and E4M3 scales.
4. Execute the unchanged Comfy NVFP4 GEMM.
Prototype:
1. Read normalized input, reproduce the reference modulation BF16 boundary, and
reduce its absolute maximum without materializing the modulated tensor.
2. Read normalized input again, reproduce the same BF16 values, and write exact
QDATA/SFA directly into caller-owned buffers.
3. Execute the unchanged Comfy NVFP4 GEMM.
## Exactness Contract
- Preserve segment-specific shift and scale selection.
- Preserve `(x * (scale + 1)) + shift` operation order and BF16 rounding.
- Preserve Vortex's approximate-reciprocal NVFP4 encoding behavior.
- Produce byte-identical QDATA and E4M3 block scales.
- Produce bit-identical QKV/FC1 outputs before changing block dispatch.
## Gate
1. Adversarial modulation and quantization tests.
2. Exact packed bytes and linear outputs for blocks 0, 24, and 49.
3. Alternating module-forward benchmark in one loaded block.
4. Full 50-block step and two-step trajectory only after all three blocks gain.
5. Canonical 12-step trajectory before enabling the path by default.
## Result
The producer is byte-exact for complete 37,810-row block 0, 24, and 49 inputs.
It takes 6.02 ms for block 24 QKV input. Integrated block medians improve by
0.28-0.79%, a warmed two-step trajectory improves by 0.52%, and the warmed
canonical 12-step run improves from 285.996 s to 284.408 s (0.56%). Video and
audio tensors are bit-identical. Spark deployments enable the path with
`H3_NVFP4_MODULATE_FUSION=1`.

224
NVFP4_STREAMING_DESIGN.md Normal file
View file

@ -0,0 +1,224 @@
# NVFP4 Streaming Design
## Decision
Comfy Kitchen 0.2.31 cannot consume incrementally produced NVFP4 tiles. Its
Python path allocates complete activation QDATA, complete swizzled block scales,
and a complete BF16 output before calling `cublas_gemm_blockwise_fp4` with fixed
full-tensor pointers. The registered operator has no tile callback, producer
interface, output stride, user output, or custom epilogue argument.
True activation-materialization removal therefore requires an owned CUTLASS or
CuTe DSL mainloop. Wrapping `scaled_mm_nvfp4` cannot provide it.
## Available Foundation
The Spark image contains:
- CUDA 13.0.88.
- NVIDIA CUTLASS DSL 4.6.2.
- CUDA Python bindings.
- Working SM121 block-scaled E2M1/E4M3 MMA support.
The official CUTLASS 4.6 SM120/SM121 persistent cooperative block-scaled GEMM
was compiled and executed on GB10 with E2M1 A/B, E4M3 scales, FP32 accumulation,
and BF16 output. A 1024x1024x1024 case passed its reference check at `16.9664 us`
and approximately `126.57 TFLOP/s`.
The public kernel still takes complete global A, B, SFA, and SFB tensors and
uses separate TMA descriptors for all four. CUTLASS does not expose a ready
BF16-to-NVFP4 input producer callback.
## Numerical Constraint
H3's activation quantization uses:
1. One exact global BF16 absolute maximum.
2. One FP32 tensor scale derived from that maximum.
3. One E4M3 block scale per 16 activation values.
4. E2M1 values quantized using both scales.
No activation tile can be packed reference-exactly until the full activation's
global reduction is complete. The minimum exact design therefore has two
phases:
1. Reduce the complete BF16 activation to one FP32 scale.
2. Produce packed tiles and consume them in block-scaled MMA.
A future cooperative mega-kernel could combine the phases around a grid-wide
barrier, but that is not the first prototype. Programmatic dependent launch can
overlap independent weight preparation but cannot transfer shared-memory tiles
between kernels.
## Reuse Constraint
The SM121 block-scaled kernel uses cluster shape `1x1x1`; there is no TMA
multicast path for sharing one packed activation tile among output-N CTAs.
Replacing global packed A with an ordinary per-CTA software producer would make
every output-N CTA reread BF16 A and repeat scale/pack work. BF16 A is materially
larger than packed E2M1 plus E4M3 scales, so this can lose despite removing the
initial packed-tensor write.
The owned kernel must therefore test an N-group schedule where one CTA retains
one or more packed A K-tiles while accumulating multiple output-N tiles. The
number of simultaneous N tiles is constrained by accumulator registers and
shared memory. This reuse factor is a first-class tuning parameter.
## Prototype Sequence
### P0: Owned Prepacked Baseline
Port the official persistent cooperative SM120/SM121 CuTe DSL kernel behind an
experimental Vortex entry point. Match Comfy Kitchen for the four H3 projection
shapes using existing packed activation and weight tensors.
Acceptance:
- Exact BF16 output versus `scaled_mm_nvfp4`.
- H3 widths 5376, 7168, and 14336 plus QKV output width 21504.
- No regression beyond measurement noise before changing the A producer.
Current P0 result: real 128-row H3 tiles execute through the CuTe kernel. Raw
block-scaled BF16 output is bit-exact for QKV, attention output, and FC1. P0 is
not complete:
- The experimental epilogue now applies H3's FP32 global-scale product before
BF16 conversion. QKV, attention output, and FC1 are bit-exact.
- FC2 at K=14336 differs even before global scaling (`max_abs=8192`, mean
`0.015625`). The correct alpha epilogue reduces final error to mean
`2.42e-5`, but bit equality still fails with `max_abs=16`. Its cuBLAS path
uses a different reduction/Stream-K policy; CUTLASS K tiles 128 and 256
produce the same non-reference result.
- The stock CuTe Float32 output mode fails the official example's own GB10
reference check and cannot be used as an accumulation oracle.
Do not begin P1 until the FC2 reduction contract is exact or FC2 is deliberately
kept on the existing cuBLAS fallback. See
`benchmarks/gb10-cute-p0-h3-summary.json`.
Decision: FC2 remains explicitly on the existing cuBLAS path. P1 and later
streaming work target QKV, attention output, and FC1 only. This preserves the
reference FC2 Stream-K reduction order and therefore full-model bit parity while
the owned producer-consumer path is developed independently.
### P1: Single-Tile Software A Producer
Replace the A and SFA TMA loads for one fixed output tile with a software warp
that loads BF16 A, generates E4M3 scales and E2M1 values into the existing staged
shared-memory layouts, and commits a producer-consumer pipeline stage. Keep
prepacked B/SFB and the existing MMA consumer and BF16 epilogue.
Acceptance:
- Packed bits and scale bytes agree with Comfy for every consumed tile.
- GEMM output is bit-exact for adversarial and randomized tile inputs.
- No global QDATA or SFA allocation in the captured kernel.
Current P1 checkpoint: the fixed 128-row producer-consumer path is exact. The
existing 32-thread DMA warp loads four BF16 rows per lane, writes E2M1 and E4M3
directly into the staged `sA`/`sSFA` layouts, and lets retained B/SFB TMA
completion publish each stage to the unchanged MMA consumer. The kernel takes
BF16 A and one tensor scale; it does not take or allocate global activation
QDATA or SFA.
Every 128-K tile has zero differing packed or scale bytes for real QKV,
attention-output, and FC1 activations. Their complete 128-row GEMM outputs are
also bit-exact after the alpha-before-BF16 epilogue. Exact Comfy compatibility
requires PTX `rcp.approx.ftz.f32`, because Comfy Kitchen builds the quantizer
with `--use_fast_math`; ordinary FP32 division changes E2M1 decisions at
midpoints. CUTLASS DSL 4.6.2 also requires a static contiguous view for vector
FP4 stores, as lowering the same store through a rank-2 dynamic layout aborts
MLIR construction. Evidence is in
`benchmarks/gb10-cute-p1-stream-a-summary.json`.
P1 is not deployment-complete. The next work is canonical row-count and padding
support and an owned runtime module rather than validator-time patching of the
CUTLASS example. FC2 continues to reject streaming and remains on Comfy/cuBLAS.
### P2: N-Group Reuse
Accumulate multiple output-N tiles per produced A tile. Sweep N-group size and
mainloop stages while recording registers, occupancy, achieved tensor
throughput, BF16 bytes read, and total projection latency.
Reject the no-materialization design if repeated BF16 reads or reduced occupancy
make it slower than the prepacked baseline. In that case, test a bounded global
ring buffer of packed tiles as the producer-consumer compromise.
Current P2 result: reject the one-producer-per-output-N-CTA schedule. At 128
rows, streamed execution is `9.14x`, `12.98x`, and `12.44x` slower than the
complete scale+pack+GEMM reference for QKV, attention output, and FC1. Producer
overhead scales at approximately `0.56-0.60 us` per `(N tile, K tile)`. Merely
matching the removed activation quantization cost would require theoretical N
reuse factors of 78, 28, and 102 respectively. Those factors are not practical
through duplicate accumulator state, especially with the existing 232-register
consumer requirement.
The next prototype is therefore a bounded global packed-tile ring or persistent
work queue: produce each `(M,K)` A tile once, let all required N consumers reuse
it, then recycle the slot. This deliberately restores bounded packed global
traffic while avoiding complete activation materialization. See
`benchmarks/gb10-cute-p1-stream-a-timing-summary.json`.
The bounded-ring capacity sweep is complete. A caller-owned `_into` CUDA
producer writes exact QDATA and SFA into reusable buffers, and one full-activation
scale is reused by every chunk. Capacities 128, 512, 1024, 2048, and 4096 were
measured; 2048 rows is the selected point. It uses 6.19 MB for QKV/FC1 and 8.26
MB for attention output. Smaller chunks leave launch overhead, while 4096 rows
regresses.
At 2048 rows, measured chunk latency improves by `10.7%` for QKV, `14.4%` for
attention output, and `11.3%` for FC1. Modeled canonical totals improve by
`10.1-12.5%` after charging one full-activation scale reduction. All packed
bytes, scale bytes, and BF16 outputs are exact. QKV, attention output, and FC1
therefore advance to complete real-projection validation; FC2 remains on
Comfy/cuBLAS. See `benchmarks/gb10-cute-p2-ring-capacity-summary.json`.
Complete 37,810-row projection parity now passes for QKV, attention output, and
FC1 in blocks 0, 24, and 49. All 19 chunks, including the final 946-row chunk,
are bit-exact with equal full checksums. QKV has a stable approximately 32 ms
ring time and improves in every tested block, so it advances to opt-in resident
runtime validation. Attention output timing is mixed. FC1 timing is unstable
under duplicate-model unified-memory pressure, including one container stop,
despite exact outputs. Do not approve those two roles from this harness. See
`benchmarks/gb10-cute-p2-ring-full-projection-summary.json`.
The opt-in QKV-only integration passes numerical validation but fails the
single-model performance gate. Alternating baseline/ring calls in one loaded
block are bit-exact for blocks 0, 24, and 49, while the 2048-row schedule is
`0.52-0.81%` slower at block level. Capacity checks from 3072 through a full
37888-row workspace do not recover a gain; 4096 rows is closest at `0.24%`
slower. The reusable output also requires serialized model requests because the
next projection reuses its storage after GPU work is enqueued. Keep this backend
disabled. The next valid prototype must fuse chunk launches through a persistent
work queue or replace the scheduler before repeating this block gate; trajectory
validation is intentionally skipped. See
`benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`.
### P3: Full H3 Projection
Support canonical H3 M/N/K shapes, padding, current weight/block-scale layout,
and token-row slicing. Integrate through `Nvfp4Linear` behind an opt-in feature
flag.
### P4: Exact Epilogue
After the owned GEMM is exact, add the explicit BF16 conversion, gate multiply,
residual add, and final BF16 rounding sequence. Validate this independently from
input streaming.
## Validation Ladder
Every phase must pass, in order:
1. Adversarial scale, E4M3, E2M1, signed-zero, midpoint, NaN, and infinity cases.
2. Randomized packed-tile and GEMM comparisons.
3. Real QKV, output, FC1, and FC2 projections at blocks 0, 24, and 49.
4. Complete block equality.
5. All 50 blocks in a denoiser step.
6. Two-step video/audio trajectory equality.
7. Canonical 12-step video/audio checksum equality.
8. Full block Nsight recapture and peak-memory comparison.
No streamed path becomes a deployment default before all applicable exactness
gates pass.

30
NVFP4_SWIGLU_FUSION_DESIGN.md Normal file
View file

@ -0,0 +1,30 @@
# NVFP4 SwiGLU Producer Fusion
## Implementation
The H3 MLP now recomputes its exact BF16 SwiGLU values inside the Vortex global
scale and NVFP4 pack passes instead of materializing the 37,810x14,336 BF16
activation. The FC2 Comfy Kitchen NVFP4 GEMM and its reduction order are
unchanged.
The fused producer preserves both BF16 boundaries:
1. `silu(gate)` rounded to BF16.
2. The BF16 SiLU result multiplied by BF16 `up` and rounded to BF16.
It then preserves the established Vortex tensor-scale compatibility cast,
E4M3 block scales, E2M1 encoding, approximate reciprocal, and Comfy physical
scale layout.
## Validation
- Randomized BF16 tensor scale, QDATA, and SFA are byte-exact.
- Real block-24 FC1 output is byte-exact; producer median is 21.93 ms.
- Blocks 0, 24, and 49 are bit-exact and improve by 2.14-2.26%.
- Warmed two-step video/audio tensors are bit-exact.
- Warmed canonical 12-step tensors are bit-exact and improve from 289.14 s to
277.36 s, or 4.07%.
Spark single-GPU deployments enable the path with
`H3_NVFP4_SWIGLU_FUSION=1`. Active FC2 LoRA, full-precision, pre-scale,
autograd, and unsupported dtype/layout cases use the materialized fallback.

565
PERFORMANCE_ROADMAP.md Normal file
View file

@ -0,0 +1,565 @@
# H3 Performance Roadmap
This document is the canonical performance plan for the direct MiniMax H3
runtime on GB10/SM121, RTX PRO 6000/SM120, and B200/SM100. It separates measured
results from projections and requires numerical and visual gates before an
approximate optimization becomes a default.
## Executive Decision
The first component, bit-exact H3 modulation and residual-gate fusion on GB10,
is complete.
This is the best first engineering target because:
1. The current path spends about `39 ms` per representative block in modulation
and residual gates, before counting the separate `25.8 ms` SwiGLU operation.
2. A Triton reference with explicit BF16 rounding already exists and has been
validated bit-exact against eager H3 block behavior.
3. The optimization does not alter attention semantics, weights, precision, or
scheduler behavior.
4. The same elementwise design applies to SM100, SM120, and SM121.
5. It provides a low-risk test of the profiling, fallback, parity, and deployment
machinery required by later custom kernels.
The next component is NVFP4 scale, activation packing, GEMM, and epilogue
fusion. Component 1 established the numerical contracts and validation
infrastructure required to attempt it safely. A custom attention target follows
this component and should preserve SageAttention 2.2.0's validated numerical
policy while adopting zero-copy strided QKV input and SM-specific fused
execution. Sol sparsity must remain experimental until full latent, audio, and
visual quality gates pass.
Completed result: blocks 0, 24, and 49 are bit-exact and `3.6-3.9%` faster;
the canonical 12-step sampling run is checksum-identical and improves from
`310.11 s` to `301.05 s` (`3.0%`). Peak allocated memory falls by
`408,825,856` bytes. Evidence is in
`benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json`.
Internal closeout summary:
> Vortex Component 1 is complete. We implemented bit-exact H3 modulation and
> residual-gate fusion on GB10, reducing canonical 12-step sampling from 310.11
> seconds to 301.05 seconds while retaining identical video and audio checksums.
> The fused runtime improves individual transformer blocks by 3.6-3.9%, reduces
> peak allocated memory by approximately 390 MiB, and passes 32 deployed
> numerical and integration tests. This establishes that Vortex can remove
> meaningful H3 execution overhead without changing the model's numerical
> program or generated output.
This is measured engineering evidence, not a projected optimization.
## Measured GB10 Baseline
The representative workload is 1344x768, 124 frames, 100 synthetic refined-text
tokens, and 37,810 packed tokens. Measurements use block 24 unless noted.
Warmed top-level sampling step:
| Backend | Step time | Relative to Sage2 |
| --- | ---: | ---: |
| SageAttention 2.2.0 | `25.414 s` | baseline |
| Sol BF16, tau 1.3 | `21.573 s` | `1.18x` |
| Sol residual-INT8 Q/K, tau 1.3 | `19.774 s` | `1.29x` |
The denoiser consumes effectively all warmed-step time. Packing takes about
`15 ms`; output transformation and the RES update together take about `1.4 ms`.
Representative Sage2 block:
| Component | P50 | Approximate block share |
| --- | ---: | ---: |
| Attention | `259.9 ms` | `50.5%` |
| NVFP4 QKV/out/FC1/FC2 projections | `151.1 ms` | `29.3%` |
| Modulation, residual gates, and SwiGLU | `64.9 ms` | `12.6%` |
| QKV and output layout materialization | `20.3 ms` | `4.0%` |
| Norm and RoPE | `19.7 ms` | `3.8%` |
| Full module forward | `514.9 ms` | baseline |
SageAttention 2.2.0 internals:
| Component | P50 |
| --- | ---: |
| Q/K INT8 quantization | `6.45 ms` |
| V FP8 quantization | `10.19 ms` |
| Attention forward | `239.00 ms` |
| K smoothing and framework estimate | `2.98 ms` |
Sol residual-INT8 Q/K internals at tau 1.3 with exact conditioning KV:
| Component | P50 |
| --- | ---: |
| K summary and residual quantization | `4.45 ms` |
| V summary | `2.45 ms` |
| Q quantization and routing threshold | `4.27 ms` |
| Total preparation | `11.53 ms` |
| Routed forward estimate | `114.88 ms` |
| Total | `126.42 ms` |
INT8 P-by-V is not useful on GB10: it increases total Sol attention time to
`138.65 ms`.
## Quality Boundary
Current Sol routing is a performance bound, not an accepted Sage2 replacement.
| Sol policy | Relative L2 versus Sage2 | Performance |
| --- | ---: | ---: |
| Fully dense, tau -100 | `0.0171` | `3.2x` slower than Sage2 |
| Exact conditioning KV, tau 0.8 | about `0.471` | about `1.12x` faster |
| Exact conditioning KV, tau 1.3 | about `0.626` | about `1.87x` faster in isolated attention |
No measured Sol setting is both faster than Sage2 and numerically Sage-like.
Exact conditioning sinks protect prompt/audio rows but do not remove sparse
video-attention error. A timestep or block hybrid therefore needs complete
latent, audio, lip-sync, and visual evaluation; attention-output timing alone
cannot approve it.
## Opportunity Ranking
The recoverable ranges below overlap and must not be added directly.
| Priority | Target | Measured cost | Plausible saving | Risk | Portability |
| ---: | --- | ---: | ---: | --- | --- |
| Complete | Fuse modulation and residual gates | `39.1 ms/block` projected scope | `3.6-3.9%` measured block gain | Low | Validated on SM121 |
| 1 | Fuse NVFP4 scale, pack, GEMM, and epilogue | `151.1 ms/block` projections | `30-75 ms/block` | Medium | Architecture-specific |
| 2 | Retain Vortex scale discovery | up to `21 ms/block` versus Comfy scale path | `3-9%` block gain already measured | Low | SM100/120/121, retune launch |
| 4 | Sage-compatible zero-copy dense attention | `259.9 ms/block` attention plus `20.3 ms` layout | `70-140 ms/block` | High | Separate SM100/120/121 policy |
| 5 | Fuse SwiGLU with FC1/FC2 boundaries | `25.8 ms/block` plus projection traffic | `5-15 ms/block` | Medium | SM100/120/121 |
| 6 | CUDA graphs or persistent denoiser execution | Not isolated yet | likely `1-5%` sampling | Medium | Shape-bucket-specific |
| 7 | Quality-gated sparse scheduling | up to `140 ms/block` attention bound | Unknown accepted saving | Very high | Backend-specific |
Native Sage NHD was tested and rejected as a standalone optimization. It was
numerically exact but did not materially improve median path time and had worse
tail latency. Eliminating layout cost requires fusion with QKV production,
RMSNorm/RoPE, or the attention kernel rather than changing Sage's layout flag.
## Implementation Phases
### Phase 1: Exact H3 Elementwise Fusion
Status: complete on GB10/SM121. Spark deployments enable
`H3_FUSED_ELEMENTWISE=1`; other deployments retain the eager default until
validated on their architecture.
Implement an opt-in direct-runtime path for:
1. RMSNorm output plus segmented AdaLN scale/shift modulation.
2. Attention residual gate/add.
3. MLP segmented scale/shift modulation.
4. MLP residual gate/add.
5. A request-layout segment-index cache reused by all 50 blocks and sampling
steps.
Preserve explicit BF16 rounding boundaries. Keep the eager implementation as a
fallback until all gates pass.
The achieved `3.0%` sampling gain is materially below the original theoretical
`20-35 ms/block` opportunity. Before extending this fusion, profile:
1. Which modulation, gate, or add launches remain outside the fused path.
2. Whether tensor conversions or intermediate allocations remain.
3. Whether the fused kernels are launch-bound or bandwidth-bound.
4. Whether generated kernels perform unnecessary loads or stores.
5. Which remaining elementwise operations can move into exact NVFP4 GEMM
epilogues.
The GB10 Nsight follow-up is complete. One warmed block has 53 kernel launches.
The fused path contains exactly two modulation and two residual gate/add
launches; no modulation, gate, or add launch remains outside it. Those four
kernels still consume about `20.48 ms`, showing that the residual is tensor
traffic rather than launch overhead. The four NVFP4 projections account for 32
launches across scale reduction/finalization, scalar conversions, block-scale
initialization, activation packing, and GEMM. Their current intermediate tensor
sizes sum to approximately `7.73 GB` per block when counted at each projection
boundary. See
`benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json`.
Success criteria:
- Bit-exact outputs at blocks 0, 24, and 49.
- Bit-exact one-step full-denoiser video and audio outputs.
- No change to a complete 12-step latent checksum.
- At least `5%` median block improvement or `3%` warmed sampling improvement.
- No peak-memory regression.
### Phase 2: Architecture Baselines
Capture the same component report on each architecture before writing a shared
kernel abstraction:
| Architecture | Required baseline |
| --- | --- |
| SM121 GB10 | Sage2 2.2, current Vortex scale path, 38K and 74K tokens |
| SM120 RTX PRO 6000 | SDPA, packaged Sage2 2.2, 38K and 74K tokens |
| SM100 B200 | SDPA, forced cuDNN SDPA, Sage2 if supported, 38K and 74K tokens |
Record actual kernel names, clocks, power, SM utilization, achieved bandwidth,
and achieved tensor throughput. Do not infer one architecture's policy from
another's result.
### Phase 3: NVFP4 Fused Projection Prototype
Status: active next component on GB10/SM121.
The first profile also exposed and fixed a native activation-packer layout bug:
the previous width-specific block-scale swizzle failed at the attention output's
7168-feature width. The replacement general 128-row by 4-scale-column mapping is
packed-bit exact at H3 widths 5376, 7168, and 14336, and all four linear outputs
now match Comfy exactly. Standalone native packing is not consistently faster,
so it remains prototype infrastructure rather than a deployed backend.
Component 2 execution order, revised from the measured Nsight profile:
1. Make QKV projection land directly in the attention backend's required
layout, with Q/K normalization and RoPE operating there.
2. Make attention output land directly in the token-major layout consumed by
the output projection.
3. Eliminate complete NVFP4 packed-activation and block-scale materialization
through a CUTLASS/CuTe producer-consumer or persistent design.
4. Integrate reference-exact residual gates into output-projection and FC2
epilogues.
5. Optimize the standalone packer only after the boundary-removing paths exist.
6. Recapture the complete block profile after every accepted boundary removal.
The first two layout items are complete for single-GPU Sage2 on GB10. Sage2
accepts the projection-strided NHD Q/K/V views directly, including in-place Q/K
normalization and RoPE, and emits contiguous NHD output that reshapes to
token-major rows without a copy. This removes three QKV copies and the attention
output copy without replacing the projection GEMM. Blocks 0, 24, and 49 and the
two-step and canonical 12-step trajectories are bit-exact. Canonical sampling
improves from the Component 1 baseline of `301.05 s` to `290.23 s` (`3.6%`),
with cumulative improvement of `6.4%` from the original `310.11 s` baseline.
The accepted Nsight recapture falls from 53 to 49 launches and contains none of
the four large BF16 layout-copy kernels.
Evidence is in
`benchmarks/gb10-sage-strided-nhd-summary-1344x768-124f-seed440420.json`.
The target NVFP4 dataflow is not another wrapper around
`scaled_mm_nvfp4`. Tensor-core GEMMs require exact packed-data and block-scale
layouts, so avoiding durable HBM intermediates likely requires an owned
CUTLASS/CuTe collective with tile-level producer-consumer execution.
Interface investigation confirms this boundary. Comfy Kitchen 0.2.31 allocates
complete QDATA, block-scale, and BF16 output tensors before invoking a cuBLAS
blockwise FP4 GEMM with fixed pointers. It cannot consume incremental tiles.
CUTLASS DSL 4.6.2's stock SM121 E2M1/E4M3 persistent kernel passes on GB10, but
its public mainloop also TMA-loads prebuilt A and SFA. The owned implementation
will follow `NVFP4_STREAMING_DESIGN.md`: preserve the mandatory global-scale
reduction, replace the A/SFA producer in one fixed tile, then measure multi-N
reuse before attempting full H3 shapes.
P0 is complete for the selected streamed roles. The owned CuTe bridge and
alpha-before-BF16 epilogue are bit-exact
for QKV, attention output, and FC1, proving E2M1, E4M3, and global-scale
interoperability. FC2 remains non-exact because its reference uses a different
reduction policy (`max_abs=16` after the correct epilogue). Resolve that policy
or retain an explicit FC2 cuBLAS fallback before the streaming A producer begins.
FC2 fallback is selected. The streamed backend will reject FC2 dispatch and use
the existing Comfy/cuBLAS implementation there; QKV, attention output, and FC1
advance to the P1 software A/SFA producer.
The fixed 128-row P1 producer-consumer checkpoint is complete. The stock
kernel's 32-thread DMA warp produces four rows per lane directly into staged
E2M1 A and E4M3 SFA shared memory; only B and SFB retain TMA loads. Every 128-K
tile has zero packed-data and scale-byte differences for real QKV,
attention-output, and FC1 activations, and all three complete GEMM outputs are
bit-exact after the fused alpha epilogue. The streamed kernel receives BF16 A
plus its tensor scale and has no global activation QDATA or SFA input.
The final arithmetic contract includes Comfy's `--use_fast_math` behavior:
encode scale must use PTX `rcp.approx.ftz.f32`. Correctly rounded division changes
FP4 midpoint decisions for real H3 values. CUTLASS DSL 4.6.2 also requires a
static contiguous destination view for vector FP4 stores. Evidence is in
`benchmarks/gb10-cute-p1-stream-a-summary.json`.
The P1 timing gate rejects direct per-output-N-CTA streaming. At 128 rows, the
exact streamed kernel is `9.14x` slower than the complete reference projection
for QKV, `12.98x` slower for attention output, and `12.44x` slower for FC1.
Producer overhead is linear at approximately `0.56-0.60 us` per `(N,K)` tile.
Break-even against the removed quantizer would require theoretical N reuse of
78, 28, and 102 tiles respectively, which cannot be implemented by retaining
duplicate accumulator state under the current register budget.
The next P2 prototype is a bounded global packed-tile ring or persistent work
queue. It must produce each `(M,K)` A tile once, expose it to all N consumers,
and recycle the slot, trading bounded global packed traffic for elimination of
the measured repeated BF16 conversion. Do not add canonical-shape runtime
dispatch for the rejected schedule. FC2 remains an explicit Comfy/cuBLAS
fallback. Evidence is in
`benchmarks/gb10-cute-p1-stream-a-timing-summary.json`.
The bounded-ring P2 checkpoint now uses caller-owned native QDATA/SFA buffers and
an allocation-free `_into` producer. A capacity sweep with one full-activation
scale selected 2048 rows. The ring occupies 6.19 MB for QKV/FC1 and 8.26 MB for
attention output. Measured chunk latency improves QKV by `10.7%`, attention
output by `14.4%`, and FC1 by `11.3%`; modeled canonical totals improve by
`10.1-12.5%`. Every packed byte, scale byte, and BF16 output matches the
reference. The 4096-row point regresses, confirming that larger buffers are not
monotonically better. Evidence is in
`benchmarks/gb10-cute-p2-ring-capacity-summary.json`.
Complete 37,810-row projection parity now passes for QKV, attention output, and
FC1 in blocks 0, 24, and 49. Every one of the 19 chunks, including the final
946-row tail, is bit-exact. QKV has a stable approximately 32 ms ring time and a
positive result across all three blocks. Attention-output timing is mixed. FC1
timing is not usable from the duplicate-model harness because multi-gigabyte
output pressure caused large variance and one container stop.
The opt-in QKV runtime gate is complete and rejects the current bounded-ring
schedule. An alternating baseline/ring benchmark inside one loaded block removes
the clock, allocator, input, and model-copy bias present in separate-process
profiles. Blocks 0, 24, and 49 remain bit-exact, but the 2048-row ring regresses
median block time by `0.81%`, `0.52%`, and `0.52%`, respectively. On block 24,
3072 rows regress by `1.00%`, 4096 by `0.24%`, 8192 by `4.80%`, and a 37888-row
full workspace by `14.49%`. The isolated projection win does not survive the
chunk launch and scheduler overhead. Keep the implementation opt-in and disabled;
do not spend trajectory-validation compute until a launch-fused work queue or a
different persistent scheduler passes this same block gate. Attention output and
FC1 remain experimental, and FC2 remains the explicit Comfy/cuBLAS fallback.
Evidence is in `benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json`.
The post-optimization profile identifies Sage2's mainloop as the largest cost.
Canonical sampling is `288.93 s`; block 24 is
`468.22 ms`, and its production NHD Sage2 call is `258.47 ms`. The existing
SM89 mainloop alone takes `238.81 ms`, while Q/K and V quantization together take
`18.34 ms`. A direct Hopper WGMMA retarget was rejected by SM121 ptxas, and
CUTLASS SM120/121 UMMA does not support the INT8 QK operation needed for exact
Sage2 parity. Do not substitute Sage3's different FP4 algorithm under the exact
optimization contract.
The next implementable exact target was AdaLN modulation into NVFP4 production.
The MSA and MLP modulation passes cost `10.47 ms` and `10.60 ms` and materialize
their values in the RMSNorm buffers solely for QKV and FC1. The deployed fusion
recomputes the exact BF16 values inside Vortex scale/pack and retains Comfy
GEMMs. Complete producer bytes match in blocks 0, 24, and 49. Alternating block
medians improve by `0.28-0.79%`; warmed two-step and 12-step runs improve by
`0.52%` and `0.56%`, with bit-identical outputs. Spark deployments enable
`H3_NVFP4_MODULATE_FUSION=1`. Evidence is in
`benchmarks/gb10-nvfp4-modulate-fusion-summary.json`.
SwiGLU-to-FC2 producer fusion is complete. Recomputing the exact BF16 SiLU and
multiply boundaries inside Vortex scale/pack removes the complete intermediate
activation while retaining the reference Comfy FC2 GEMM. The real producer is
byte-exact and takes `21.93 ms`. Alternating blocks 0, 24, and 49 improve by
`2.14-2.26%`; the warmed canonical 12-step trajectory improves by `4.07%` with
identical video and audio tensors. Enable `H3_NVFP4_SWIGLU_FUSION=1` for Spark
single-GPU deployments. Evidence is in
`benchmarks/gb10-nvfp4-swiglu-fusion-summary.json`.
The mandatory post-fusion profile supersedes the earlier approximately 515 ms
block distribution. With all production flags enabled, fresh block-24 timing is
`458.78 ms` median without Nsight and `465.78 ms` across the Nsight GPU span.
The 41 kernels have only `0.084 ms` total inter-kernel idle time, so CPU launch
gaps are not a block-level bottleneck. A complete warmed sampling step takes
`23.708 s`, contains 2,694 kernels, and has `12.995 ms` total inter-kernel idle
time.
The fresh block kernel-time distribution is:
| Component | Time | Share |
| --- | ---: | ---: |
| Sage2 preparation and mainloop | `268.22 ms` | `57.60%` |
| Four NVFP4 GEMMs | `125.11 ms` | `26.87%` |
| NVFP4 scale and packing | `41.53 ms` | `8.92%` |
| Norm and RoPE | `19.86 ms` | `4.27%` |
| Two residual gate/add kernels | `10.94 ms` | `2.35%` |
The one-step distribution independently matches these shares within 0.5
percentage points. The standalone modulation and SwiGLU kernels are gone.
Their remaining producer work is `13.04 ms` for modulated QKV/FC1 packing and
`23.04 ms` for SwiGLU-to-FC2 packing. The block still records 8,009,578,496
positive self-allocated bytes, but there are no large activation-layout copy
kernels or explicit CUDA memcpy operations. Remaining conversions are small
AdaLN table and scalar operations.
GB10 exposes no direct physical DRAM-byte counter. Nsight Compute measures
`320.999 GB` of L2-request traffic for the block. With cache control disabled,
L2 read/write miss sectors imply a `42.117 GB` off-chip request proxy: NVFP4
GEMMs account for `56.25%`, packing `18.25%`, Sage2 `15.09%`, norm/RoPE
`6.54%`, and gates `3.87%`. This separates the time bottleneck, Sage2, from the
off-chip traffic bottleneck, the NVFP4 GEMMs. Do not select a new kernel from
the old profile. See
`benchmarks/gb10-fully-fused-fresh-nsight-summary.json`.
The follow-up real block-24 Sage2 decomposition now selects the next exact
kernel experiment. Manual preparation plus the existing prequantized mainloop
is byte-exact against public SageAttention 2.2.0. Uninstrumented median phase
times are `2.37 ms` K smoothing, `7.63 ms` combined Q/K quantization, `10.61 ms`
combined V transpose/quantization, and `237.09 ms` for the fused mainloop.
The full-counter mainloop capture is profiler-perturbed to `258.92 ms`, so its
duration is not used as the baseline. Its ratios establish the bottleneck:
| Mainloop metric | Result |
| --- | ---: |
| Registers per thread | `255` |
| Dynamic shared memory per CTA | `32 KiB` |
| Achieved occupancy | `16.83%` / `8.08` warps per SM |
| Scheduler cycles with no eligible warp | `63.53%` |
| INT8 QK tensor-pipe utilization | `37.77%` of elapsed cycles |
| FP8 PV tensor-pipe utilization | `37.77%` of elapsed cycles |
| Combined tensor-pipe utilization | `75.54%` |
| Memory throughput | `31.61%` |
| L2 hit rate | `98.84%` |
| L2 request traffic | `161.50 GB` |
| Off-chip request proxy | `1.85 GB` |
| Excess shared-memory wavefronts | `7.68%` |
This explains why high SM activity did not imply a saturated tensor pipeline.
QK and PV alternate on separate INT and FP tensor sub-pipelines, while online
softmax, scaling, conversion, and synchronization occupy scalar pipelines.
Only `0.46` warps per scheduler are eligible on average. Fixed-latency
dependencies consume `2.01` of the `5.48` cycles between issued instructions,
and math-pipe throttle consumes another `1.24`; memory scoreboards are much
smaller. The 255-register footprint limits the kernel to two CTAs per SM, so it
cannot hide these dependencies. Q and KV tail sweeps each add less than 1 ms and
exclude tail scheduling as the primary target.
The source-identical classic SM89 P0 is complete and rejected. The exact
three-CTA cliff is 168 registers/thread, not approximately 170: 170 still has
two CTAs and `16.67%` theoretical occupancy, while 168 reaches `25%` at the cost
of `4.95 billion` local spill requests and `78.79%` no-eligible cycles. Caps from
240 through 170 never change residency and progressively worsen scheduler
eligibility.
Byte-exact source variants also missed the 3% gate. Narrowed scopes reduced
static spills from 44/44 to 12/12 bytes and dynamic spill requests by `73%`, but
improved interleaved latency by only `0.06%`. In-place score storage was
`+0.01%`; early K prefetch was `-0.30%`; independent softmax-chain interleaving
was `-0.02%`. The persistent 128-register FP32 output fragment plus exact QK or
instantaneous PV fragments prevents a source-only lifetime cleanup from
reaching the next residency tier.
The shared-memory follow-up is also closed. Source attribution maps all
`626,970,624` excessive wavefronts to four repeated V-staging `LDGSTS.128`
instructions. A 128-byte padded V stride raised dynamic shared memory from 32
to 40 KiB but left the excess count exactly unchanged and changed latency by
`-0.04%`. No complete-block benchmark or deployment was run because no mainloop
variant crossed 3%. Evidence is in
`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`, the per-variant
latency JSON files, and the P0 NCU reports.
Materializing the optional BF16 LoRA input while these fused producers pack
NVFP4 was also tested against the official Turbo-4 adapter. The isolated
1344x768, 124-frame trajectory was bit-exact but regressed from `131.11 s` to
`135.14 s` (`3.08%`). Reject this implementation: its extra BF16 writes cost
more than the removed standalone producer. Active LoRA must retain the existing
materialized fallback unless a future design consumes the producer values
without global-memory materialization. Evidence is in
`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`.
An exact residual-gate epilogue must preserve this numerical program:
1. Complete NVFP4 GEMM accumulation.
2. Convert to the reference BF16 GEMM output.
3. Apply the reference-equivalent gate multiplication and its BF16 boundary.
4. Add the BF16 residual using the reference operation order.
5. Round the final output to BF16.
Bit-exact GEMM output alone is insufficient. Validate randomized epilogues,
representative real blocks, all 50 blocks in one denoiser step, a two-step
trajectory, and the canonical 12-step trajectory before enabling an epilogue.
The direct-layout target extends naturally to distributed execution: QKV tiles
must be able to land in rank-destined Ulysses buffers, and attention output must
land in the output projection's local token-major layout without an intervening
transpose or contiguous copy.
Prototype behind `Nvfp4Linear` so model code remains unchanged. Compare:
1. Comfy Kitchen baseline.
2. Vortex scale plus Comfy pack/GEMM.
3. Fused scale and activation pack.
4. Fused scale, pack, GEMM, bias, and output epilogue.
5. Transformer Engine or CUTLASS reference where supported.
Gate every variant on packed activation identity, linear output error, block
output error, full denoiser-step error, and final media quality.
### Phase 4: Dense Sage-Compatible Attention
The first owned attention kernel should remain dense and target Sage2 behavior:
1. Consume strided BSHD views directly from fused QKV output.
2. Preserve Sage2 per-warp INT8 Q/K quantization and FP8 V policy.
3. Preserve its accumulation and K-smoothing behavior.
4. Fuse Q/K/V preparation with the attention launch where profitable.
5. Emit token-major output suitable for the NVFP4 output projection without an
intermediate transpose/contiguous allocation.
6. Tune independently for SM100, SM120, and SM121.
The initial goal is `1.5x` attention speedup with bit-exact Sage2 output where
the operation order permits it; any nonzero tolerance must be declared and
quality-gated before implementation. A `2x` attention target is stretch.
### Phase 5: Sparse Hybrid Research
Only after the dense backend passes:
1. Keep early quality-sensitive sampling steps dense.
2. Keep first/last sensitive transformer blocks dense.
3. Force text, reference, and audio conditioning KV blocks exact.
4. Sweep tau by timestep using complete generated media, not random attention
tensors alone.
5. Reject any policy that degrades prompt adherence, identity, lip sync, audio
onset, or temporal stability.
## Performance Targets
### GB10 / SM121
The current two-step profile implies roughly `305 s` for a warmed 12-step
1344x768/124-frame sampling run.
| Target | Estimated sampling | Speedup |
| --- | ---: | ---: |
| Conservative | `215-240 s` | `1.3-1.4x` |
| Engineering | `170-195 s` | `1.6-1.8x` |
| Stretch | `130-145 s` | `2.1-2.3x` |
| Aggressive ceiling | about `120 s` | about `2.5x` |
Use `1.5x` as the commitment target, `1.8x` as the engineering target, and
`2.3x` as stretch. Do not plan around a `5x` end-to-end gain.
### RTX PRO 6000 / SM120
These projections have lower confidence because the measured RTX baseline uses
PyTorch SDPA rather than packaged Sage2.
| Workload | Current SDPA | Engineering target | Stretch target |
| --- | ---: | ---: | ---: |
| 1344x768, 124 frames | `97.94 s` | `45-60 s` | `35-45 s` |
| 1344x768, 243 frames | `311.02 s` | `120-170 s` | `90-120 s` |
Packaging and profiling Sage2 2.2 on SM120 is the first RTX action. Expected
overall improvement is `1.6-2.2x` at 124 frames and `1.8-2.6x` at 243 frames;
stretch ranges are `2.2-2.8x` and `2.6-3.5x` respectively. These are planning
ranges, not commitments.
### B200 / SM100
Measured B200 SDPA sampling is `79.72 s` at 124 frames and `257.86 s` at 243
frames, only `1.23x` and `1.21x` faster than RTX PRO 6000 despite much larger
theoretical FP4 and memory-bandwidth capability. No optimized B200 target is
assigned until component profiling confirms actual SDPA dispatch, NVFP4 kernel
selection, clocks, power, utilization, and achieved throughput.
The current B200 result measures this runtime path, not B200's hardware ceiling.
## Evidence
- `benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json`
- `benchmarks/gb10-37810-token-optimization-profile-seed440420.json`
- `benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`
- `benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`
- `tools/profile_h3_block.py`
- `tools/profile_attention_components.py`
- `tools/profile_hybrid_attention.py`
- `tools/profile_sampling_stages.py`
Component timings use synchronized probes for attribution. End-to-end claims
must use separate uninstrumented runs. Resident services remained loaded during
GB10 profiling, so medians are preferred over means and final claims require an
isolated rerun.

27
PLAN.md
View file

@ -93,6 +93,33 @@ Current NVFP4 activation quantization status:
- `vortex_native` BF16->NVFP4 pack is parity-safe for real H3 FC1/FC2 activation shapes, including qdata, cuBLAS tiled block-scale layout, signed zero, tie-to-even FP4 rounding, and downstream `scaled_mm_nvfp4` linear output.
- Keep Comfy Kitchen as the default production packer. Pack-only sweep `8d90cec` shows native 512-thread pack is essentially tied on FC1 (`1.090 ms` vs CK `1.098 ms`) but still slower on FC2 (`2.977 ms` vs CK `2.886 ms`). Use `vortex_native` only as an experimental benchmark path until the wide-FC2 pack kernel is redesigned.
Major cross-architecture performance opportunities:
1. Better attention kernels, especially for 38K-74K-token sequences.
2. Eliminating Q/K/V transpose and contiguous copies.
3. Fusing NVFP4 activation scaling, packing, GEMM, bias, and epilogues.
4. Replacing or tuning Comfy Kitchen GEMMs independently for SM100, SM120, and SM121.
5. CUDA graphs or persistent execution to reduce thousands of kernel launches per sample.
GB10 profiling at 1344x768/124 frames produced 37,810 packed tokens. Sage2
attention remains the parity baseline; native NHD layout was exact but did not
improve median path time, and Sol-Attn was faster but materially changed the
attention output. The first retained optimization routes H3 activation-scale
discovery through the parity-safe Vortex kernel when
`H3_NVFP4_SCALE_BACKEND=vortex`, while retaining Comfy Kitchen's packer and
GEMM. Representative blocks 0, 24, and 49 improved by `3.1%`, `8.8%`, and
`3.3%` respectively with identical output checksums. Details are recorded in
`benchmarks/gb10-37810-token-optimization-profile-seed440420.json`.
The expanded sampling profile confirms that the denoiser consumes effectively
all warmed-step latency. At block 24, Sage2 spends about `50.5%` in attention,
`29.3%` in NVFP4 projections, `12.6%` in modulation/gates/SwiGLU, `4.0%` in
layout materialization, and `3.8%` in norm/RoPE. Sol's sparse timing is a useful
performance bound, but not an accepted backend: its first speed-positive
settings differ substantially from Sage2, while fully dense Sol is `3.2x`
slower. See
`benchmarks/gb10-sampling-component-profile-1344x768-124f-seed440420.json`.
Validation and profiling sequence:
1. Use `tools/profile_h3_block.py` to measure one representative H3 block before writing kernels. Record QKV, RoPE/RMS, attention kernel, output projection, MLP fc1, activation, MLP fc2, modulation/gating, and total block time.

14
README.md
View file

@ -23,6 +23,8 @@ Implemented and validated:
- Opt-in EasyCache and H3-Cache experiments.
- Ragged Ulysses sequence parallelism and true NVFP4 tensor parallelism with
automatic 1/2/4/6/8-GPU launch tooling.
- Opt-in bit-exact Triton fusion for H3 segmented modulation and residual gates;
enabled by default in the validated Spark deployment.
The main remaining feature gap is full arbitrary Ref2VA, especially reference
video/audio, identity, and voice conditioning. Multi-GPU code is CPU- and
@ -148,6 +150,12 @@ read/write is available and NCCL selected `P2P/CUMEM`; topology varied between
`SYS` and `NODE` across RunPod allocations. See
`benchmarks/rtxpro6000-server-1v2-sdpa-scaling-seed440420.json`.
A matched one-GPU B200 SDPA run took `79.72s` at 1344x768/124 frames and
`257.86s` at 1344x768/243 frames. That is only `1.23x` and `1.21x` faster than
one RTX PRO 6000, while the measured sampling cost at RunPod rates was `2.64x`
and `2.69x` higher. See
`benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json`.
Matched GB10 backend results and parity evidence are recorded in [`PLAN.md`](PLAN.md)
and [`PARITY.md`](PARITY.md). Standalone `tools/compare_*`, `tools/trace_*`,
`tools/inspect_*`, and `tools/patch_comfy_*` commands are diagnostic utilities,
@ -165,6 +173,12 @@ not runtime startup requirements.
- [`FLASH4.md`](FLASH4.md): FlashAttention-4 integration and benchmark.
- [`PARITY.md`](PARITY.md): direct-versus-Comfy evidence ledger.
- [`PLAN.md`](PLAN.md): historical investigation and future kernel plan.
- [`PERFORMANCE_ROADMAP.md`](PERFORMANCE_ROADMAP.md): measured bottlenecks,
architecture targets, quality gates, and the ordered optimization plan.
- [`VORTEX_RESEARCH_AGENDA.md`](VORTEX_RESEARCH_AGENDA.md): flagship research
thesis, novel systems contributions, evaluation targets, and project ranking.
- [`NVFP4_STREAMING_DESIGN.md`](NVFP4_STREAMING_DESIGN.md): measured GEMM
interface boundary and the exact producer-consumer kernel prototype sequence.
## Known Gaps

152
SAGE2_BLACKWELL_DESIGN.md Normal file
View file

@ -0,0 +1,152 @@
# Sage2 Blackwell Mainloop
## Decision
The adjacent NVFP4 producer fusions are complete. The next exact kernel
experiment is now a retune of SageAttention 2.2.0's existing classic SM89
mainloop. Do not replace the algorithm with UMMA, FP8 Q/K, or FP4 attention.
The canonical GB10 profile is dominated by the attention kernel:
- Complete block 24: 468.22 ms median.
- Sage2 NHD path: 260.02 ms median.
- SM89 attention mainloop: 238.81 ms median.
- Q/K quantization: 7.68 ms median.
- V quantization: 10.65 ms median.
- Remaining framework overhead: 2.87 ms estimated.
The fresh real block-24 decomposition supersedes those internal timings:
- Fused mainloop: 237.09 ms uninstrumented median.
- K mean/smoothing: 2.37 ms median.
- Q quantization: 3.80 ms median.
- K subtract-mean quantization: 3.83 ms median.
- V transpose/pad/permute: 5.08 ms median.
- V scale/FP8 quantization: 5.53 ms median.
- Manual decomposition versus public Sage2: byte-exact.
Q/K RMSNorm plus RoPE is 12.20 ms and is not the first target. NVFP4 ring
dispatch is also excluded because it failed the controlled block gate.
## Existing Contract
For the 37,810-token H3 shape, preserve these Sage2 inputs and outputs:
- Batch 1, 56 heads, head dimension 128, non-causal attention.
- Projection-strided BF16 NHD Q/K/V input.
- Per-warp INT8 Q with 128-row blocks and 32-row warp groups.
- Per-block INT8 K with 64-row blocks.
- FP8 V and the existing per-channel scale layout.
- Sage2's `fp32+fp16` instantaneous-buffer accumulation order.
- Contiguous BF16 NHD output with exact equality to Sage2 2.2.0.
The current mainloop is
`sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf`.
It receives Q/K INT8 tensors, FP8 V, Q/K/V scales, and emits the final BF16
attention output.
## Rejected Shortcut
SageAttention's Hopper SM90 WGMMA source was compiled experimentally for
`sm_121a`. CUDA 13 ptxas rejects `wgmma.fence`, `wgmma.mma_async`,
`wgmma.commit_group`, and `wgmma.wait_group` for SM121, including both INT8 and
FP8 forms. A Hopper binary or source retarget is therefore not viable.
CUTLASS 4.6 SM120/121 exposes `tcgen05`/UMMA for F8/F6/F4 but not INT8. It
therefore cannot preserve Sage2's INT8 QK operation. Changing Q/K to FP8 or FP4
would define a different attention algorithm and require a quality gate rather
than exact parity.
## Scheduler Diagnosis
The full-counter capture reports 255 registers/thread and 32 KiB dynamic shared
memory for each 128-thread CTA. Registers limit residency to two CTAs, producing
`16.83%` achieved occupancy and only `0.46` eligible warps per scheduler. No
warp is eligible in `63.53%` of scheduler cycles. Fixed-latency dependency
stalls are `2.01` cycles and math-pipe throttle is `1.24` cycles out of the
`5.48` cycles between issued instructions.
The fused phase attribution is counter-based rather than separate wall time:
- INT8 QK uses `37.77%` of elapsed tensor-pipe capacity.
- FP8 PV uses `37.77%` of elapsed tensor-pipe capacity.
- Scalar FMA and ALU pipelines use `15.01%` and `8.81%`, covering scale,
online-softmax, conversion, normalization, and output work.
- Combined tensor activity is `75.54%`; the INT and FP sub-pipelines alternate,
so neither individually exceeds `37.77%`.
- Memory throughput is `31.61%`, L2 hit rate is `98.84%`, and the warm
off-chip request proxy is only `1.85 GB`. This is not a DRAM bottleneck.
- Shared accesses create `626,970,624` excessive wavefronts, `7.68%` of all
shared wavefronts.
- Real Q and KV tail CTAs each cost less than 1 ms.
## Completed P0
The source-identical classic `mma.sync` retune was completed without an accepted
variant. The exact three-CTA threshold is 168 registers/thread; 170 remains at
two CTAs. The 168-register cap causes `4.95 billion` local spill requests and
worsens no-eligible cycles from `63.13%` to `78.79%`. Narrowed scopes reduce
static spills from 44/44 to 12/12 bytes but improve real-input latency by only
`0.06%`. In-place score reuse, early K prefetch, and independent softmax-chain
interleaving are also neutral or slower.
The shared excess is produced by repeated V-staging `LDGSTS.128` instructions.
A 128-byte padded V layout leaves all `626,970,624` excessive wavefronts
unchanged and provides no latency gain. This counter is therefore not an
actionable layout target for the exact kernel.
The P0 gate was:
1. Adversarial short-shape comparison against the Sage2 mainloop.
2. Exact real block-24 attention output at 37,810 tokens.
3. Mainloop median below 220 ms before block integration.
4. Blocks 0, 24, and 49 exact after integration.
5. Alternating block benchmark must improve all three blocks before trajectory
validation.
Use the Blackwell SageAttention3/CUTLASS code only as a scheduler reference. Its
FP4 algorithm and output are not a correctness replacement for Sage2.
No variant exceeded the 3% mainloop gate, so block integration and deployment
were intentionally skipped. Detailed evidence is in
`benchmarks/gb10-sage2-p0-register-scheduler-analysis.json`.
## Completed P1 And P2
P1 fused strided-NHD Q/K RMSNorm, split-half RoPE, and Q INT8 quantization. It
was byte-exact, but complete-block gains were only `0.53-0.86%`, below the 1%
gate, so its runtime branch was removed.
P2 replaced Sage2's separate BF16 V transpose and FP8 quantization with an owned
three-stage direct-NHD producer. It emits byte-identical FP8 V and FP32 scales
for 13 boundary lengths. Canonical V preparation improves from `10.56 ms` to
`6.39 ms` (`39.48%`), but the `4.17 ms` saving is only `0.91%` of the complete
block and misses the `6.0 ms` isolated go threshold. No block integration or
trajectory validation was run. The validator-only implementation remains useful
for a future owned attention backend; production Sage2 remains unchanged.
These results close the remaining preparation-only Sage2 opportunities. The
next Sage-class optimization with plausible multi-percent block impact is a new
warp-specialized QK/PV mainloop or a quality-gated Blackwell-native attention
algorithm, not another launch-boundary fusion.
## Completed P3
P3 tested the smallest exact temporal QK/PV specialization. Warp pairs A and B
alternated INT8 QK plus online softmax against the other pair's prior-tile FP8
PV. Every warp retained its original query rows and private `RS`, `RS_f8`, `RO`,
`m`, and `d`; no numerical state crossed warp boundaries. The candidate is
compute-sanitizer clean and byte-exact across 13 short boundary cases and the
real block-24 output SHA.
The schedule does not produce a material latency gain. In a rotating 50-sample
comparison, baseline median is `245.44 ms` and the candidate is `245.20 ms`, a
`0.10%` improvement. It remains far above the `<220 ms` gate. Ptxas reports 254
registers/thread and 32/24-byte static store/load spills for the candidate,
versus 255 registers and 24/24-byte spills for the clean baseline build. NCU and
block integration were skipped because the latency gate failed.
This closes temporal warp separation as an exact Sage2 optimization on the
current SM89 instruction stream. A multi-percent attention gain now requires a
substantially new mainloop or a quality-gated Blackwell-native attention
algorithm rather than another exact schedule rearrangement.

6
TURBO.md
View file

@ -30,6 +30,12 @@ output = NVFP4_base(x) + strength * (alpha / rank) * B(A(x))
The BF16 branch receives the original activation. It does not receive the base checkpoint's `pre_quant_scale`-modified activation, and the LoRA delta is never merged into packed NVFP4 weights. Fused QKV uses rank/alpha 384; the other released projections use rank/alpha 128.
Active LoRA intentionally uses the materialized projection fallback. Emitting
the exact BF16 LoRA input while the NVFP4 modulation and SwiGLU producers pack
was bit-exact, but slowed an isolated canonical Turbo-4 trajectory by `3.08%`.
The rejected result is recorded in
`benchmarks/gb10-nvfp4-lora-producer-fusion-turbo4-isolated.json`.
Turbo sampling maintains independent native video and audio states. The 4-step and 8-step variants use their own shifted sigma grids and positive data-ward training-Euler updates. The original beta/RES sampler remains unchanged when Turbo is disabled.
## Hot API

185
VORTEX_RESEARCH_AGENDA.md Normal file
View file

@ -0,0 +1,185 @@
# Vortex Research Agenda
## Flagship Research Project
**Vortex: Reference-Exact, Communication-Avoiding Video Diffusion on Commodity PCIe GPUs**
The core research question:
> Can a large multimodal video diffusion model achieve near-NVLink scaling on
> ordinary PCIe hardware while preserving reference-exact output?
The existing `86.1%` two-GPU efficiency is the starting evidence, not the final
contribution.
## Novel Architecture
Build an end-to-end H3 block where:
- NVFP4 QKV projections write directly into rank-destined communication buffers.
- Q, K, and V are never separately materialized or repacked.
- Ulysses communication begins at tile granularity while later projection tiles
are still computing.
- Video, audio, text, and conditioning tokens use different sharding policies.
- Small conditioning segments are replicated when replication is cheaper than
communication.
- Received attention output writes directly into the output-projection layout.
- Reference BF16 rounding boundaries are preserved exactly.
- The scheduler automatically accounts for PIX, PXB, PHB, NODE, and SYS topology.
The target pipeline is:
```text
NVFP4 projection
|
v
rank-destined QKV tiles
|
v overlapping
PCIe all-to-all
|
v overlapping
attention
|
v
output projection layout
```
There is no intermediate QKV pack, full synchronization point, or redundant
memory round trip.
Communication and computation overlap has already been explored by systems such
as Ulysses Unbound and topology-aware systems such as SwiftFusion. The
contribution therefore cannot simply be "we overlapped an all-to-all."
The novel combination is:
- Direct NVFP4 GEMM-to-collective dataflow.
- Ragged multimodal sequence parallelism.
- Segment-aware replication and sharding.
- Reference-exact numerical behavior.
- Automatic PCIe topology adaptation.
- Scaling across inexpensive non-NVLink hardware.
That combination appears meaningfully ahead of public H3 implementations.
## Rounding-Boundary-Aware Exact Fusion
Create a compiler or kernel-generation framework that understands where the
original model performs BF16 rounding.
Ordinary fusion can silently change output because fused multiply-add performs
one rounding where separate operations perform two. NVIDIA documents this
numerical distinction in its CUDA floating-point guidance.
The system would:
- Trace the reference H3 graph.
- Mark mandatory numerical materialization boundaries.
- Fuse everything between safe boundaries.
- Generate Triton or CUDA kernels.
- Prove equivalence with adversarial and randomized testing.
- Preserve bitwise behavior across SM100, SM120, and SM121 where possible.
Apply it to:
- AdaLN modulation.
- Residual gates.
- SwiGLU.
- QK normalization and RoPE.
- NVFP4 scaling and packing.
- GEMM epilogues.
This could be a broader contribution than Vortex itself. It would answer:
> How much of a mixed-precision transformer can be fused without altering its
> numerical program?
That is a serious systems and numerical-computing paper.
## Trajectory-Bounded Sparse Attention
This is the highest-risk project.
The Sol results show why simple threshold sparsity is not enough:
- It becomes fast at higher thresholds.
- Attention-level relative error rises to roughly `0.5-0.6`.
- Local attention error does not reveal what happens to the final video.
Instead of using one global threshold, allocate an error budget across:
- Denoising timestep.
- Transformer block.
- Attention head.
- Video, audio, text, and conditioning segments.
- Spatial and temporal regions.
The runtime would choose dense or sparse attention dynamically, with exact
fallback when its predicted error exceeds the remaining trajectory budget.
Existing research already covers distributed sparse attention, including DSA,
and feature reuse such as FasterCache. The potential Vortex contribution is
different:
> Optimize sparsity against final multimodal trajectory error, not isolated
> attention-tensor error.
That requires proper evaluation of image quality, temporal consistency, audio
quality, lip synchronization, and prompt adherence.
## Practical Ranking
| Research direction | Novelty | Technical risk | Commercial value | Recommendation |
| --- | --- | --- | --- | --- |
| PCIe-native ragged Ulysses | Very high | Medium | Exceptional | First |
| Rounding-aware exact fusion | Very high | Medium | Exceptional | Build alongside |
| Trajectory-bounded sparsity | Very high | Very high | Potentially exceptional | Later |
| NVFP4 block superkernel | Medium-high | Medium | Very high | Core component |
| Topology-autonomous planner | Medium | Low-medium | High | Supporting work |
| Conventional caching | Medium-low | Medium | High | Only with a novel error guarantee |
## Landmark Result
A compelling target is:
| Scale | Target efficiency | Interconnect |
| --- | ---: | --- |
| 2 GPUs | Above `90%` | PCIe, no NVLink |
| 4 GPUs | Above `80%` | PCIe |
| 8 GPUs | Above `70%` | PCIe or cross-NUMA |
| Output | Reference-exact or formally bounded | All scales |
| Portability | SM100, SM120, SM121 | Same runtime |
Also measure:
- End-to-end latency, not sampling alone.
- Exposed communication time.
- Bytes transferred per block.
- HBM traffic and achieved bandwidth.
- Tensor-core utilization.
- Cost per generated second.
- Energy per generated second.
- PIX/NODE/PHB/SYS topology sensitivity.
- Comparison against SGLang, xDiT, and LightX2V.
## Recommendation
Do not lead with sparse attention yet. Lead with:
> Reference-exact, communication-avoiding ragged Ulysses with direct NVFP4
> projection-to-collective dataflow.
The first fused modulation kernel establishes the exact-fusion infrastructure.
Use that infrastructure next to build the direct QKV-to-Ulysses path.
The measured single-GPU implementation order is direct QKV-to-attention layout,
direct attention-to-output-projection layout, streamed NVFP4 scale/pack/GEMM,
and exact residual-gate epilogues. This order attacks the measured `105.83 ms`
layout boundary first while preserving the larger projection-to-collective
architecture as the distributed destination.
If strong four-GPU and eight-GPU scaling over PCIe can be demonstrated while
retaining exact output, this stops being merely an excellent H3 runtime. It
becomes credible new research into how large generative models should be
executed without proprietary high-bandwidth interconnects.

107
benchmarks/b200-vs-rtxpro6000-server-sdpa-1344x768-seed440420.json Normal file
View file

@ -0,0 +1,107 @@
{
"device": "NVIDIA B200",
"comparison_device": "NVIDIA RTX PRO 6000 Blackwell Server Edition",
"attention": "sdpa",
"mode": "tensor",
"world_size": 1,
"steps": 12,
"seed": 440420,
"torch": "2.9.1+cu130",
"hourly_usd": {
"b200": 6.79,
"rtx_pro_6000": 2.09
},
"b200_host": {
"data_center": "US-NC-2",
"driver": "580.105.08",
"memory_mib": 183359,
"power_limit_w": 1000
},
"results": [
{
"resolution": [
1344,
768
],
"frames": 124,
"video_seconds_at_24fps": 5.166666666666667,
"b200": {
"model_load_seconds": 11.831760542932898,
"conditioning_seconds": 45.36778333503753,
"sampling_seconds": 79.72125827614218,
"sampling_cost_usd": 0.15036315102639039,
"sampling_peak_allocated_bytes": 17246854656,
"checksums": [
-83553.09375,
-397.89813232421875
],
"console_step_seconds_rounded": [
6.4,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1,
6.1
]
},
"rtx_pro_6000": {
"sampling_seconds": 97.94385590963066,
"sampling_cost_usd": 0.05686184968086891
},
"b200_speedup": 1.2285789013812125,
"b200_sampling_cost_ratio": 2.6443591242685138
},
{
"resolution": [
1344,
768
],
"frames": 243,
"video_seconds_at_24fps": 10.125,
"b200": {
"model_load_seconds": 10.712746233213693,
"conditioning_seconds": 38.57332478091121,
"sampling_seconds": 257.8619639207609,
"sampling_cost_usd": 0.48635631528387957,
"sampling_peak_allocated_bytes": 22955644416,
"checksums": [
41606.85546875,
629.4112548828125
],
"console_step_seconds_rounded": [
21.4,
21.1,
21.1,
21.1,
21.0,
21.0,
21.0,
21.0,
21.0,
21.0,
21.0,
21.0
]
},
"rtx_pro_6000": {
"sampling_seconds": 311.01632468774915,
"sampling_cost_usd": 0.18056225516594326
},
"b200_speedup": 1.2061349411862938,
"b200_sampling_cost_ratio": 2.6935657999890426
}
],
"cost_scope": "Sampling wall time only. Model/Qwen loading, text conditioning, VAE decode, media encoding, muxing, and pod startup are excluded.",
"notes": [
"The 124-frame run preceded the 243-frame run on the same host.",
"Both successful runs followed one pre-sampling dependency failure, so host and network-volume file caches were warm.",
"Console step durations are rounded to one decimal place; sampling_seconds is the authoritative synchronized benchmark timer.",
"The RTX PRO 6000 measurements are copied from rtxpro6000-server-1v2-sdpa-scaling-seed440420.json."
]
}

75
benchmarks/gb10-37810-token-optimization-profile-seed440420.json Normal file
View file

@ -0,0 +1,75 @@
{
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"torch": "2.9.1+cu130",
"resolution": [
1344,
768
],
"frames": 124,
"packed_tokens": 37810,
"seed": 440420,
"attention": "sage2",
"representative_block_results": [
{
"block": 0,
"comfy_scale_p50_seconds": 0.5251280819647945,
"vortex_scale_p50_seconds": 0.5088961640140042,
"speedup": 1.031896326006387,
"latency_reduction": 0.0309103978786619,
"checksum": -8279139.0
},
{
"block": 24,
"comfy_scale_p50_seconds": 0.5646577654988505,
"vortex_scale_p50_seconds": 0.5149167295021471,
"speedup": 1.0966001552227602,
"latency_reduction": 0.08809059050619694,
"checksum": 89071408.0
},
{
"block": 49,
"comfy_scale_p50_seconds": 0.524194426019676,
"vortex_scale_p50_seconds": 0.5067900969879702,
"speedup": 1.034342283196033,
"latency_reduction": 0.03320204902570367,
"checksum": -40617144320.0
}
],
"block_24_quantization_p50_milliseconds": {
"attn_qkv_proj": {
"comfy": 9.078,
"vortex": 3.908
},
"attn_out_proj": {
"comfy": 10.035,
"vortex": 5.151
},
"mlp_fc1": {
"comfy": 7.27,
"vortex": 5.182
},
"mlp_fc2": {
"comfy": 19.187,
"vortex": 10.163
}
},
"attention_findings": [
"Sage2 was substantially faster than PyTorch SDPA, forced cuDNN SDPA, and FlashAttention-4 at 37,810 tokens.",
"Sage2 NHD produced an exact output but did not materially improve median total path time and had worse tail latency.",
"Strict Sol-Attn reached about 157 ms kernel time versus about 260 ms for Sage2, but its output differed materially from Sage2 and is not a correctness-preserving replacement.",
"Explicit Sage2 kernel variants did not provide an exact speed improvement over the Sage2 dispatcher."
],
"validation": {
"linear_reference_diff": "zero for all four Vortex-scale plus Comfy-pack H3 linears",
"block_checksums": "identical for Comfy and Vortex paths at blocks 0, 24, and 49",
"remote_contracts": "28 passed",
"full_model_smoke": "not run: the deployed image predates the distributed runner; direct preview did not produce an artifact while resident services remained active"
},
"notes": [
"The optimization changes activation-scale discovery only; Comfy Kitchen still packs activations and executes scaled_mm_nvfp4.",
"The default H3_NVFP4_SCALE_BACKEND=torch path is unchanged.",
"The native Vortex packer is not enabled because the attention output projection failed parity in this profile.",
"Resident H3 and Qwen services remained running during profiling, so medians are preferred over means and tail latency requires an isolated rerun."
]
}

22
benchmarks/gb10-component2-layout-hnd-2step.json Normal file
View file

@ -0,0 +1,22 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"attention": "sage2",
"resolution": [
1344,
768
],
"frames": 124,
"steps": 2,
"seed": 440420,
"text_tokens": 100,
"elapsed_seconds": 52.49690314201871,
"stage_trace": [],
"checksums": [
-276716.375,
727.1075439453125
],
"peak_allocated_bytes": 18867826176,
"peak_reserved_bytes": 21846032384,
"measurement_policy": "uninstrumented sampling wall time"
}

22
benchmarks/gb10-component2-layout-strided-nhd-12step.json Normal file
View file

@ -0,0 +1,22 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"attention": "sage2",
"resolution": [
1344,
768
],
"frames": 124,
"steps": 12,
"seed": 440420,
"text_tokens": 100,
"elapsed_seconds": 290.2269145210157,
"stage_trace": [],
"checksums": [
-91481.7890625,
678.401611328125
],
"peak_allocated_bytes": 17680283648,
"peak_reserved_bytes": 19809697792,
"measurement_policy": "uninstrumented sampling wall time"
}

22
benchmarks/gb10-component2-layout-strided-nhd-2step.json Normal file
View file

@ -0,0 +1,22 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"attention": "sage2",
"resolution": [
1344,
768
],
"frames": 124,
"steps": 2,
"seed": 440420,
"text_tokens": 100,
"elapsed_seconds": 50.95522483601235,
"stage_trace": [],
"checksums": [
-276716.375,
727.1075439453125
],
"peak_allocated_bytes": 17681987584,
"peak_reserved_bytes": 19809697792,
"measurement_policy": "uninstrumented sampling wall time"
}

69
benchmarks/gb10-component2-nsys-summary-1344x768-124f-seed440420.json Normal file
View file

@ -0,0 +1,69 @@
{
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"workload": {
"resolution": [1344, 768],
"frames": 124,
"packed_tokens": 37810,
"block": 24,
"attention": "sage2",
"seed": 440420
},
"capture": {
"tool": "Nsight Systems 2025.3.2",
"scope": "one warmed H3 block between cudaProfilerStart and cudaProfilerStop",
"fused_elementwise": true,
"runtime_kernel_launches": 53,
"cudaLaunchKernel_calls": 45,
"cuLaunchKernelEx_calls": 8
},
"component1_residual": {
"modulation_launches": 2,
"modulation_total_ms": 9.5432,
"gate_add_launches": 2,
"gate_add_total_ms": 10.936032,
"unfused_modulation_gate_or_add_launches": 0,
"conclusion": "The four fused kernels perform substantial tensor traffic and are not launch-bound. Eliminate that traffic only at adjacent NVFP4 boundaries."
},
"nvfp4": {
"launches": 32,
"gemm_launches": 4,
"gemm_total_ms": 162.339136,
"activation_pack_launches": 4,
"activation_pack_total_ms": 13.121248,
"absmax_partial_launches": 4,
"absmax_partial_total_ms": 10.767616,
"scale_finalize_launches": 4,
"scale_finalize_total_ms": 0.013536,
"block_scale_zero_launches": 4,
"block_scale_zero_total_ms": 0.292992,
"small_scale_conversion_launches": 12,
"intermediate_materialization_bytes_across_four_projections": 7732981760,
"conclusion": "Scale discovery, activation packing, buffer initialization, GEMM, and output allocation remain separate. The global scale dependency requires an explicit synchronization design in any exact fused implementation."
},
"other_boundaries": {
"qkv_and_attention_layout_copy_launches": 4,
"qkv_and_attention_layout_copy_total_ms": 105.83168,
"swiglu_silu_total_ms": 9.390784,
"swiglu_multiply_total_ms": 14.38736
},
"native_pack_validation": {
"initial_finding": "The previous width-specific block-scale swizzle failed for the 7168-feature attention output projection while QDATA remained exact.",
"fix": "Replaced width-specific indexing with the general 128-row by 4-scale-column tiled offset.",
"validated_feature_widths": [5376, 7168, 14336],
"qdata_difference_count": 0,
"block_scale_difference_count": 0,
"linear_output_max_abs_difference": 0.0,
"deployment_decision": "Do not enable the standalone native packer for performance; it is not consistently faster than Comfy packing. Use it as exact infrastructure for the fused prototype."
},
"validation": {
"deployed_tests": "33 passed",
"new_regression": "Comfy packed QDATA and block-scale identity at feature widths 5376, 7168, and 14336"
},
"source_artifacts": [
"benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json",
"benchmarks/gb10-nvfp4-pack-parity-component2-fixed-1344x768-124f-seed440420.json",
"benchmarks/gb10-nvfp4-native-fixed-component2-1344x768-124f-seed440420.json",
"benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json"
]
}

88
benchmarks/gb10-cute-nvfp4-conversion-contract.json Normal file
View file

@ -0,0 +1,88 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"value_count": 41,
"padded_value_count": 48,
"fp4_torch_shape": [
48
],
"fp4_bytes": [
255,
239,
222,
205,
188,
171,
154,
137,
8,
16,
17,
34,
50,
51,
68,
84,
85,
102,
118,
119,
7,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"expected_low_first_bytes": [
255,
239,
222,
205,
188,
171,
154,
137,
8,
16,
17,
34,
50,
51,
68,
84,
85,
102,
118,
119,
7,
0,
0,
0
],
"fp4_prefix_equal": true,
"fp8_equal": true,
"fp8_difference_count": 0
}

92
benchmarks/gb10-cute-nvfp4-tile-producer.json Normal file
View file

@ -0,0 +1,92 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"tile": [
128,
128
],
"all_equal": true,
"cases": [
{
"name": "random",
"tensor_scale": 0.0013885498046875,
"fp4_difference_count": 0,
"fp4_equal": true,
"block_scale_difference_count": 0,
"block_scales_equal": true,
"scalar_block_scales_equal": true,
"actual_block_scale_bytes": [
107,
110,
111,
112,
113,
114,
115,
116,
117,
118,
119,
120,
121,
122,
123,
124,
125,
126
],
"expected_block_scale_bytes": [
107,
110,
111,
112,
113,
114,
115,
116,
117,
118,
119,
120,
121,
122,
123,
124,
125,
126
]
},
{
"name": "zeros",
"tensor_scale": 0.0,
"fp4_difference_count": 0,
"fp4_equal": true,
"block_scale_difference_count": 0,
"block_scales_equal": true,
"scalar_block_scales_equal": true,
"actual_block_scale_bytes": [
126
],
"expected_block_scale_bytes": [
126
]
},
{
"name": "sparse_extremes",
"tensor_scale": 0.037109375,
"fp4_difference_count": 0,
"fp4_equal": true,
"block_scale_difference_count": 0,
"block_scales_equal": true,
"scalar_block_scales_equal": true,
"actual_block_scale_bytes": [
0,
126
],
"expected_block_scale_bytes": [
0,
126
]
}
]
}

85
benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json Normal file
View file

@ -0,0 +1,85 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"rows": 128,
"mnk": [
128,
5376,
7168
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"cute_shapes": {
"a": "(128, 7168, 1)",
"b": "(5376, 7168, 1)",
"sfa": "(128, 448, 1)",
"sfb": "(5376, 448, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 0.0263671875,
"b": 0.00072479248046875,
"alpha": 1.911073923110962e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": 257945.5,
"max_abs": 1368.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": 13503609856.0,
"candidate_checksum": 257945.5,
"equal": false,
"max_abs": 71826088.0,
"mean_abs": 2754388.25
},
"reference_checksum": 257945.5,
"candidate_checksum": 257945.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

83
benchmarks/gb10-cute-p0-attn-out-128rows.json Normal file
View file

@ -0,0 +1,83 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"rows": 128,
"mnk": [
128,
5376,
7168
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"cute_shapes": {
"a": "(128, 7168, 1)",
"b": "(5376, 7168, 1)",
"sfa": "(128, 448, 1)",
"sfb": "(5376, 448, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 0.0263671875,
"b": 0.00072479248046875,
"alpha": 1.911073923110962e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": 13503609856.0,
"max_abs": 71827456.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"reference_checksum": 13503609856.0,
"candidate_checksum": 13503609856.0,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"reference_checksum": 257945.5,
"candidate_checksum": 258224.125,
"equal": false,
"max_abs": 8.0,
"mean_abs": 0.07462421804666519,
"relative_l2": 0.0028051051776856184,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

85
benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json Normal file
View file

@ -0,0 +1,85 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"rows": 128,
"mnk": [
128,
21504,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(21504, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(21504, 336, 1)",
"c": "(128, 21504, 1)"
},
"tensor_scales": {
"a": 0.0205078125,
"b": 0.00141143798828125,
"alpha": 2.8945505619049072e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -18469.02734375,
"max_abs": 187.0,
"nonzero": 2752509,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -640210368.0,
"candidate_checksum": -18469.02734375,
"equal": false,
"max_abs": 6455109.0,
"mean_abs": 455697.25
},
"reference_checksum": -18469.02734375,
"candidate_checksum": -18469.02734375,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

83
benchmarks/gb10-cute-p0-attn-qkv-128rows.json Normal file
View file

@ -0,0 +1,83 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"rows": 128,
"mnk": [
128,
21504,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(21504, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(21504, 336, 1)",
"c": "(128, 21504, 1)"
},
"tensor_scales": {
"a": 0.0205078125,
"b": 0.00141143798828125,
"alpha": 2.8945505619049072e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -640210368.0,
"max_abs": 6455296.0,
"nonzero": 2752509,
"finite": true
},
"raw_blockscaled_parity": {
"reference_checksum": -640210368.0,
"candidate_checksum": -640210368.0,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"reference_checksum": -18469.02734375,
"candidate_checksum": -18533.14453125,
"equal": false,
"max_abs": 1.0,
"mean_abs": 0.018117837607860565,
"relative_l2": 0.002810996025800705,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

62
benchmarks/gb10-cute-p0-h3-summary.json Normal file
View file

@ -0,0 +1,62 @@
{
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"cutlass_dsl": "4.6.2",
"rows": 128,
"kernel": "official persistent cooperative SM120/SM121 block-scaled GEMM",
"results": [
{
"linear": "attn_qkv_proj",
"mnk": [128, 21504, 5376],
"raw_blockscaled_equal": true,
"raw_max_abs": 0.0,
"fused_alpha_equal": true,
"fused_alpha_max_abs": 0.0
},
{
"linear": "attn_out_proj",
"mnk": [128, 5376, 7168],
"raw_blockscaled_equal": true,
"raw_max_abs": 0.0,
"fused_alpha_equal": true,
"fused_alpha_max_abs": 0.0
},
{
"linear": "mlp_fc1",
"mnk": [128, 28672, 5376],
"raw_blockscaled_equal": true,
"raw_max_abs": 0.0,
"fused_alpha_equal": true,
"fused_alpha_max_abs": 0.0
},
{
"linear": "mlp_fc2",
"mnk": [128, 5376, 14336],
"raw_blockscaled_equal": false,
"raw_max_abs": 8192.0,
"raw_mean_abs": 0.01562502235174179,
"fused_alpha_equal": false,
"fused_alpha_max_abs": 16.0,
"fused_alpha_mean_abs": 0.000024163342459360138,
"tile_k_128_and_256_identical": true
}
],
"findings": [
"Comfy E2M1 values and E4M3 scale layouts are interoperable with CUTLASS after logical FP4 re-encoding.",
"The raw block-scaled operation is bit-exact for QKV, attention output, and FC1.",
"The custom alpha-before-BF16 epilogue is bit-exact for QKV, attention output, and FC1.",
"FC2 still requires matching the reference GEMM reduction policy; changing the stock K tile from 128 to 256 does not restore equality.",
"The official stock Float32 epilogue failed its own GB10 reference check and is not a valid accumulation oracle."
],
"status": "P0 remains active; do not implement the streaming A producer until the exact alpha epilogue and FC2 reduction contract pass.",
"source_artifacts": [
"benchmarks/gb10-cute-p0-attn-qkv-128rows.json",
"benchmarks/gb10-cute-p0-attn-out-128rows.json",
"benchmarks/gb10-cute-p0-mlp-fc1-128rows.json",
"benchmarks/gb10-cute-p0-mlp-fc2-128rows.json",
"benchmarks/gb10-cute-p0-attn-qkv-128rows-alpha.json",
"benchmarks/gb10-cute-p0-attn-out-128rows-alpha.json",
"benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json",
"benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json"
]
}

85
benchmarks/gb10-cute-p0-mlp-fc1-128rows-alpha.json Normal file
View file

@ -0,0 +1,85 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"rows": 128,
"mnk": [
128,
28672,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(28672, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(28672, 336, 1)",
"c": "(128, 28672, 1)"
},
"tensor_scales": {
"a": 0.00494384765625,
"b": 0.0023651123046875,
"alpha": 1.1692754924297333e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -3814965.5,
"max_abs": 95.0,
"nonzero": 3670016,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -326266716160.0,
"candidate_checksum": -3814965.5,
"equal": false,
"max_abs": 8126369.0,
"mean_abs": 646601.3125
},
"reference_checksum": -3814965.5,
"candidate_checksum": -3814965.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

83
benchmarks/gb10-cute-p0-mlp-fc1-128rows.json Normal file
View file

@ -0,0 +1,83 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"rows": 128,
"mnk": [
128,
28672,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(28672, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(28672, 336, 1)",
"c": "(128, 28672, 1)"
},
"tensor_scales": {
"a": 0.00494384765625,
"b": 0.0023651123046875,
"alpha": 1.1692754924297333e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -326266716160.0,
"max_abs": 8126464.0,
"nonzero": 3670016,
"finite": true
},
"raw_blockscaled_parity": {
"reference_checksum": -326266716160.0,
"candidate_checksum": -326266716160.0,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"reference_checksum": -3814965.5,
"candidate_checksum": -3814898.0,
"equal": false,
"max_abs": 0.5,
"mean_abs": 0.010720730759203434,
"relative_l2": 0.0028073240537196398,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

85
benchmarks/gb10-cute-p0-mlp-fc2-128rows-alpha.json Normal file
View file

@ -0,0 +1,85 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc2",
"rows": 128,
"mnk": [
128,
5376,
14336
],
"tile_shape_mnk": [
128,
128,
256
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"cute_shapes": {
"a": "(128, 14336, 1)",
"b": "(5376, 14336, 1)",
"sfa": "(128, 896, 1)",
"sfb": "(5376, 896, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 1.1953125,
"b": 0.001129150390625,
"alpha": 0.0013496875762939453
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -1426078.5,
"max_abs": 14208.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -1057548288.0,
"candidate_checksum": -1426078.5,
"equal": false,
"max_abs": 10537088.0,
"mean_abs": 838238.5
},
"reference_checksum": -1426062.125,
"candidate_checksum": -1426078.5,
"equal": false,
"max_abs": 16.0,
"mean_abs": 2.4163342459360138e-05,
"relative_l2": 1.2930971934110858e-05,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

83
benchmarks/gb10-cute-p0-mlp-fc2-128rows.json Normal file
View file

@ -0,0 +1,83 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc2",
"rows": 128,
"mnk": [
128,
5376,
14336
],
"tile_shape_mnk": [
128,
128,
256
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"cute_shapes": {
"a": "(128, 14336, 1)",
"b": "(5376, 14336, 1)",
"sfa": "(128, 896, 1)",
"sfb": "(5376, 896, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 1.1953125,
"b": 0.001129150390625,
"alpha": 0.0013496875762939453
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -1057541632.0,
"max_abs": 10551296.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"reference_checksum": -1057548288.0,
"candidate_checksum": -1057541632.0,
"equal": false,
"max_abs": 8192.0,
"mean_abs": 0.01562502235174179
},
"reference_checksum": -1426062.125,
"candidate_checksum": -1428981.125,
"equal": false,
"max_abs": 64.0,
"mean_abs": 1.5995242595672607,
"relative_l2": 0.002791827078908682,
"numerical_note": "The stock SM121 kernel's validated BF16 epilogue rounds before the external global-scale product. Exact H3 integration requires applying alpha in a custom epilogue before BF16 conversion."
}

46
benchmarks/gb10-cute-p1-real-tiles-attn-out.json Normal file
View file

@ -0,0 +1,46 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"activation_shape": [
128,
7168
],
"tensor_scale": 0.0263671875,
"tile_count": 56,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"equal": true,
"differing_tiles": [],
"difference_examples": []
}

46
benchmarks/gb10-cute-p1-real-tiles-attn-qkv.json Normal file
View file

@ -0,0 +1,46 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"activation_shape": [
128,
5376
],
"tensor_scale": 0.0205078125,
"tile_count": 42,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"equal": true,
"differing_tiles": [],
"difference_examples": []
}

46
benchmarks/gb10-cute-p1-real-tiles-mlp-fc1.json Normal file
View file

@ -0,0 +1,46 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"activation_shape": [
128,
5376
],
"tensor_scale": 0.00494384765625,
"tile_count": 42,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"equal": true,
"differing_tiles": [],
"difference_examples": []
}

86
benchmarks/gb10-cute-p1-stream-a-attn-out-128rows-alpha.json Normal file
View file

@ -0,0 +1,86 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"rows": 128,
"mnk": [
128,
5376,
7168
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"cute_shapes": {
"a": "(128, 7168, 1)",
"b": "(5376, 7168, 1)",
"sfa": "(1,)",
"sfb": "(5376, 448, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 0.0263671875,
"b": 0.00072479248046875,
"alpha": 1.911073923110962e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": 257945.5,
"max_abs": 1368.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": 13503609856.0,
"candidate_checksum": 257945.5,
"equal": false,
"max_abs": 71826088.0,
"mean_abs": 2754388.25
},
"reference_checksum": 257945.5,
"candidate_checksum": 257945.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

86
benchmarks/gb10-cute-p1-stream-a-attn-qkv-128rows-alpha.json Normal file
View file

@ -0,0 +1,86 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"rows": 128,
"mnk": [
128,
21504,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(21504, 5376, 1)",
"sfa": "(1,)",
"sfb": "(21504, 336, 1)",
"c": "(128, 21504, 1)"
},
"tensor_scales": {
"a": 0.0205078125,
"b": 0.00141143798828125,
"alpha": 2.8945505619049072e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -18469.02734375,
"max_abs": 187.0,
"nonzero": 2752509,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -640210368.0,
"candidate_checksum": -18469.02734375,
"equal": false,
"max_abs": 6455109.0,
"mean_abs": 455697.25
},
"reference_checksum": -18469.02734375,
"candidate_checksum": -18469.02734375,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

86
benchmarks/gb10-cute-p1-stream-a-mlp-fc1-128rows-alpha.json Normal file
View file

@ -0,0 +1,86 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"rows": 128,
"mnk": [
128,
28672,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(28672, 5376, 1)",
"sfa": "(1,)",
"sfb": "(28672, 336, 1)",
"c": "(128, 28672, 1)"
},
"tensor_scales": {
"a": 0.00494384765625,
"b": 0.0023651123046875,
"alpha": 1.1692754924297333e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -3814965.5,
"max_abs": 95.0,
"nonzero": 3670016,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -326266716160.0,
"candidate_checksum": -3814965.5,
"equal": false,
"max_abs": 8126369.0,
"mean_abs": 646601.3125
},
"reference_checksum": -3814965.5,
"candidate_checksum": -3814965.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

54
benchmarks/gb10-cute-p1-stream-a-summary.json Normal file
View file

@ -0,0 +1,54 @@
{
"date": "2026-08-23",
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"tile_shape_mnk": [128, 128, 128],
"producer_threads": 32,
"rows_per_producer_thread": 4,
"activation_input": "BF16",
"global_activation_qdata": false,
"global_activation_sfa": false,
"retained_tma_inputs": ["B", "SFB"],
"streamed_roles": {
"attn_qkv_proj": {
"mnk": [128, 21504, 5376],
"packed_fp4_differences": 0,
"block_scale_differences": 0,
"bf16_output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0
},
"attn_out_proj": {
"mnk": [128, 5376, 7168],
"packed_fp4_differences": 0,
"block_scale_differences": 0,
"bf16_output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0
},
"mlp_fc1": {
"mnk": [128, 28672, 5376],
"packed_fp4_differences": 0,
"block_scale_differences": 0,
"bf16_output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0
}
},
"mlp_fc2": {
"streaming_enabled": false,
"fallback": "Comfy/cuBLAS",
"validator_rejects_streaming": true
},
"numerical_contract": {
"block_scale_order": "(amax / 6.0f) / global_scale",
"block_scale_clamp": "fminf(value, 448.0f)",
"encode_scale": "fminf(rcp.approx.ftz.f32(rounded_scale * global_scale), FLT_MAX)",
"fp4_conversion": "CuTe E2M1 round-to-nearest ties-to-even",
"epilogue": "FP32 global alpha before BF16 conversion"
},
"status": "Fixed 128-row P1 producer-consumer checkpoint is exact; not runtime-integrated or performance-approved."
}

50
benchmarks/gb10-cute-p1-stream-a-timing-summary.json Normal file
View file

@ -0,0 +1,50 @@
{
"date": "2026-08-23",
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"rows": 128,
"tile_shape_mnk": [128, 128, 128],
"warmup": 5,
"iterations": 20,
"results": {
"attn_qkv_proj": {
"n_tiles": 168,
"k_tiles": 42,
"prepacked_gemm_ms": 0.4769120216369629,
"activation_quantize_ms": 0.05241439938545227,
"reference_complete_projection_ms": 0.49579682350158694,
"streamed_gemm_ms": 4.529531097412109,
"streamed_vs_prepacked": 9.497624073020535,
"streamed_vs_reference_complete": 9.135861471281917,
"repeated_producer_overhead_ms": 4.052619075775146,
"minimum_theoretical_n_reuse_to_cover_quantize_cost": 78
},
"attn_out_proj": {
"n_tiles": 42,
"k_tiles": 56,
"prepacked_gemm_ms": 0.054211199283599854,
"activation_quantize_ms": 0.05055999755859375,
"reference_complete_projection_ms": 0.11322239637374878,
"streamed_gemm_ms": 1.4691216468811035,
"streamed_vs_prepacked": 27.09996580587633,
"streamed_vs_reference_complete": 12.975539238999248,
"repeated_producer_overhead_ms": 1.4149104475975036,
"minimum_theoretical_n_reuse_to_cover_quantize_cost": 28
},
"mlp_fc1": {
"n_tiles": 224,
"k_tiles": 42,
"prepacked_gemm_ms": 0.46705121994018556,
"activation_quantize_ms": 0.051630401611328126,
"reference_complete_projection_ms": 0.45939040184020996,
"streamed_gemm_ms": 5.716193771362304,
"streamed_vs_prepacked": 12.238901275312733,
"streamed_vs_reference_complete": 12.442997825954953,
"repeated_producer_overhead_ms": 5.249142551422119,
"minimum_theoretical_n_reuse_to_cover_quantize_cost": 102
}
},
"producer_overhead_per_n_k_tile_ms_range": [0.0005579445739181673, 0.000601577571257442],
"decision": "Reject one-A-producer-per-output-N-CTA scheduling. Required N reuse factors of 28-102 are not practical with duplicate accumulator state. Prototype a bounded global packed-tile ring or persistent work queue before canonical-shape runtime integration.",
"numerical_status": "All timed streamed outputs remain bit-exact."
}

98
benchmarks/gb10-cute-p1-timing-p0-attn-out.json Normal file
View file

@ -0,0 +1,98 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"rows": 128,
"mnk": [
128,
5376,
7168
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": false,
"streamed_activation_materialization": null,
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 1.084223985671997,
"mean_ms": 0.054211199283599854
},
"reference_timing": {
"backend": "vortex_scale_plus_comfy_pack_gemm",
"activation_quantize_mean_ms": 0.05055999755859375,
"complete_projection_mean_ms": 0.11322239637374878
},
"cute_shapes": {
"a": "(128, 7168, 1)",
"b": "(5376, 7168, 1)",
"sfa": "(128, 448, 1)",
"sfb": "(5376, 448, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 0.0263671875,
"b": 0.00072479248046875,
"alpha": 1.911073923110962e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": 257945.5,
"max_abs": 1368.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": 13503609856.0,
"candidate_checksum": 257945.5,
"equal": false,
"max_abs": 71826088.0,
"mean_abs": 2754388.25
},
"reference_checksum": 257945.5,
"candidate_checksum": 257945.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

98
benchmarks/gb10-cute-p1-timing-p0-attn-qkv.json Normal file
View file

@ -0,0 +1,98 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"rows": 128,
"mnk": [
128,
21504,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": false,
"streamed_activation_materialization": null,
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 9.538240432739258,
"mean_ms": 0.4769120216369629
},
"reference_timing": {
"backend": "vortex_scale_plus_comfy_pack_gemm",
"activation_quantize_mean_ms": 0.05241439938545227,
"complete_projection_mean_ms": 0.49579682350158694
},
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(21504, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(21504, 336, 1)",
"c": "(128, 21504, 1)"
},
"tensor_scales": {
"a": 0.0205078125,
"b": 0.00141143798828125,
"alpha": 2.8945505619049072e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -18469.02734375,
"max_abs": 187.0,
"nonzero": 2752509,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -640210368.0,
"candidate_checksum": -18469.02734375,
"equal": false,
"max_abs": 6455109.0,
"mean_abs": 455697.25
},
"reference_checksum": -18469.02734375,
"candidate_checksum": -18469.02734375,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

98
benchmarks/gb10-cute-p1-timing-p0-mlp-fc1.json Normal file
View file

@ -0,0 +1,98 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"rows": 128,
"mnk": [
128,
28672,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": false,
"streamed_activation_materialization": null,
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 9.341024398803711,
"mean_ms": 0.46705121994018556
},
"reference_timing": {
"backend": "vortex_scale_plus_comfy_pack_gemm",
"activation_quantize_mean_ms": 0.051630401611328126,
"complete_projection_mean_ms": 0.45939040184020996
},
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(28672, 5376, 1)",
"sfa": "(128, 336, 1)",
"sfb": "(28672, 336, 1)",
"c": "(128, 28672, 1)"
},
"tensor_scales": {
"a": 0.00494384765625,
"b": 0.0023651123046875,
"alpha": 1.1692754924297333e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -3814965.5,
"max_abs": 95.0,
"nonzero": 3670016,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -326266716160.0,
"candidate_checksum": -3814965.5,
"equal": false,
"max_abs": 8126369.0,
"mean_abs": 646601.3125
},
"reference_checksum": -3814965.5,
"candidate_checksum": -3814965.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

96
benchmarks/gb10-cute-p1-timing-stream-attn-out.json Normal file
View file

@ -0,0 +1,96 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"rows": 128,
"mnk": [
128,
5376,
7168
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"streamed_activation_materialization": {
"global_qdata": false,
"global_sfa": false
},
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 29.38243293762207,
"mean_ms": 1.4691216468811035
},
"cute_shapes": {
"a": "(128, 7168, 1)",
"b": "(5376, 7168, 1)",
"sfa": "(1,)",
"sfb": "(5376, 448, 1)",
"c": "(128, 5376, 1)"
},
"tensor_scales": {
"a": 0.0263671875,
"b": 0.00072479248046875,
"alpha": 1.911073923110962e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": 257945.5,
"max_abs": 1368.0,
"nonzero": 688128,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": 13503609856.0,
"candidate_checksum": 257945.5,
"equal": false,
"max_abs": 71826088.0,
"mean_abs": 2754388.25
},
"reference_checksum": 257945.5,
"candidate_checksum": 257945.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

96
benchmarks/gb10-cute-p1-timing-stream-attn-qkv.json Normal file
View file

@ -0,0 +1,96 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"rows": 128,
"mnk": [
128,
21504,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"streamed_activation_materialization": {
"global_qdata": false,
"global_sfa": false
},
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 90.59062194824219,
"mean_ms": 4.529531097412109
},
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(21504, 5376, 1)",
"sfa": "(1,)",
"sfb": "(21504, 336, 1)",
"c": "(128, 21504, 1)"
},
"tensor_scales": {
"a": 0.0205078125,
"b": 0.00141143798828125,
"alpha": 2.8945505619049072e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -18469.02734375,
"max_abs": 187.0,
"nonzero": 2752509,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -640210368.0,
"candidate_checksum": -18469.02734375,
"equal": false,
"max_abs": 6455109.0,
"mean_abs": 455697.25
},
"reference_checksum": -18469.02734375,
"candidate_checksum": -18469.02734375,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

96
benchmarks/gb10-cute-p1-timing-stream-mlp-fc1.json Normal file
View file

@ -0,0 +1,96 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"rows": 128,
"mnk": [
128,
28672,
5376
],
"tile_shape_mnk": [
128,
128,
128
],
"swap_nibbles": false,
"reencode_fp4": true,
"zero_a": false,
"fuse_alpha": true,
"stream_a": true,
"streamed_activation_materialization": {
"global_qdata": false,
"global_sfa": false
},
"timing": {
"warmup": 5,
"iterations": 20,
"total_ms": 114.3238754272461,
"mean_ms": 5.716193771362304
},
"cute_shapes": {
"a": "(128, 5376, 1)",
"b": "(28672, 5376, 1)",
"sfa": "(1,)",
"sfb": "(28672, 336, 1)",
"c": "(128, 28672, 1)"
},
"tensor_scales": {
"a": 0.00494384765625,
"b": 0.0023651123046875,
"alpha": 1.1692754924297333e-05
},
"raw_output": {
"dtype": "torch.bfloat16",
"checksum": -3814965.5,
"max_abs": 95.0,
"nonzero": 3670016,
"finite": true
},
"raw_blockscaled_parity": {
"applicable": false,
"reference_checksum": -326266716160.0,
"candidate_checksum": -3814965.5,
"equal": false,
"max_abs": 8126369.0,
"mean_abs": 646601.3125
},
"reference_checksum": -3814965.5,
"candidate_checksum": -3814965.5,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"relative_l2": 0.0,
"numerical_note": "The experimental epilogue applies the FP32 global-scale product before BF16 conversion."
}

67
benchmarks/gb10-cute-p2-ring-attn-out.json Normal file
View file

@ -0,0 +1,67 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
128,
5376,
7168
],
"ring": {
"row_capacity": 128,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 458752,
"sfa_bytes": 57344,
"logical_bytes": 516096
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.010279680490493775,
"global_scale_ms": 0.01211359977722168,
"cute_consumer_ms": 0.03870527982711792,
"modeled_ring_ms": 0.06109856009483337,
"actual_into_ring_cute_gemm_ms": 0.09652607917785644,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.10898783683776855,
"modeled_ring_vs_reference": 0.5605998051487184,
"actual_ring_vs_reference": 0.8856591889381034
}
}

67
benchmarks/gb10-cute-p2-ring-attn-qkv.json Normal file
View file

@ -0,0 +1,67 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"mnk": [
128,
21504,
5376
],
"ring": {
"row_capacity": 128,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 344064,
"sfa_bytes": 43008,
"logical_bytes": 387072
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.015116159915924072,
"global_scale_ms": 0.017698240280151368,
"cute_consumer_ms": 0.4326623916625977,
"modeled_ring_ms": 0.46547679185867313,
"actual_into_ring_cute_gemm_ms": 0.4637001419067383,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.47588897705078126,
"modeled_ring_vs_reference": 0.978120558167505,
"actual_ring_vs_reference": 0.9743872295181523
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-attn-out-1024.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
1024,
5376,
7168
],
"full_activation_rows": 37810,
"modeled_chunk_count": 37,
"ring": {
"row_capacity": 1024,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 3670016,
"sfa_bytes": 458752,
"logical_bytes": 4128768
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 50,
"producer_ms": 0.08987903594970703,
"global_scale_ms": 4.179701232910157,
"cute_consumer_ms": 0.44698047637939453,
"modeled_cute_chunk_ms": 0.5368595123291016,
"modeled_comfy_chunk_ms": 0.5336019134521484,
"actual_into_ring_cute_gemm_ms": 0.5717164611816407,
"comfy_consumer_ms": 0.4437228775024414,
"actual_into_ring_comfy_gemm_ms": 0.6249113464355469,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.5978803253173828,
"modeled_canonical_reference_ms": 26.301273269653322,
"modeled_canonical_cute_ring_ms": 25.33321029663086,
"modeled_canonical_comfy_ring_ms": 27.301421051025393,
"actual_ring_vs_reference": 0.9562389611635855,
"comfy_ring_vs_reference": 1.0452114243829898,
"modeled_canonical_cute_ring_vs_reference": 0.9631933038717398,
"modeled_canonical_comfy_ring_vs_reference": 1.0380265917591927
}
}

72
benchmarks/gb10-cute-p2-ring-capacity-attn-out-128.json Normal file
View file

@ -0,0 +1,72 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
128,
5376,
7168
],
"full_activation_rows": 37810,
"ring": {
"row_capacity": 128,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 458752,
"sfa_bytes": 57344,
"logical_bytes": 516096
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.019654719829559325,
"global_scale_ms": 3.735771484375,
"cute_consumer_ms": 0.07179071903228759,
"modeled_ring_ms": 3.8272169232368465,
"actual_into_ring_cute_gemm_ms": 0.09406271934509278,
"comfy_consumer_ms": 0.05615776062011719,
"actual_into_ring_comfy_gemm_ms": 0.07875840187072754,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.08742112159729004,
"modeled_ring_vs_reference": 43.77908740255154,
"actual_ring_vs_reference": 1.0759724609619812,
"comfy_ring_vs_reference": 0.9009081607707142
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-attn-out-2048.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
2048,
5376,
7168
],
"full_activation_rows": 37810,
"modeled_chunk_count": 19,
"ring": {
"row_capacity": 2048,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 7340032,
"sfa_bytes": 917504,
"logical_bytes": 8257536
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.22272863388061523,
"global_scale_ms": 3.323794250488281,
"cute_consumer_ms": 0.775920639038086,
"modeled_cute_chunk_ms": 0.9986492729187012,
"modeled_comfy_chunk_ms": 1.0811997032165528,
"actual_into_ring_cute_gemm_ms": 1.008663330078125,
"comfy_consumer_ms": 0.8584710693359375,
"actual_into_ring_comfy_gemm_ms": 1.1376866912841797,
"reference_vortex_scale_comfy_pack_gemm_ms": 1.178064956665039,
"modeled_canonical_reference_ms": 25.70702842712402,
"modeled_canonical_cute_ring_ms": 22.488397521972654,
"modeled_canonical_comfy_ring_ms": 24.939841384887693,
"actual_ring_vs_reference": 0.8562034923214508,
"comfy_ring_vs_reference": 0.9657249244598827,
"modeled_canonical_cute_ring_vs_reference": 0.8747956842123643,
"modeled_canonical_comfy_ring_vs_reference": 0.9701565256983631
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-attn-out-4096.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
4096,
5376,
7168
],
"full_activation_rows": 37810,
"modeled_chunk_count": 10,
"ring": {
"row_capacity": 4096,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 14680064,
"sfa_bytes": 1835008,
"logical_bytes": 16515072
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 50,
"producer_ms": 0.3313638305664062,
"global_scale_ms": 2.197855987548828,
"cute_consumer_ms": 1.3521932983398437,
"modeled_cute_chunk_ms": 1.6835571289062499,
"modeled_comfy_chunk_ms": 1.7651193237304688,
"actual_into_ring_cute_gemm_ms": 1.8329510498046875,
"comfy_consumer_ms": 1.4337554931640626,
"actual_into_ring_comfy_gemm_ms": 1.6257913208007813,
"reference_vortex_scale_comfy_pack_gemm_ms": 1.596439666748047,
"modeled_canonical_reference_ms": 18.1622526550293,
"modeled_canonical_cute_ring_ms": 20.527366485595703,
"modeled_canonical_comfy_ring_ms": 18.45576919555664,
"actual_ring_vs_reference": 1.148149277409534,
"comfy_ring_vs_reference": 1.0183856957854998,
"modeled_canonical_cute_ring_vs_reference": 1.130221392438976,
"modeled_canonical_comfy_ring_vs_reference": 1.0161608004305602
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-attn-out-512.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_out_proj",
"mnk": [
512,
5376,
7168
],
"full_activation_rows": 37810,
"modeled_chunk_count": 74,
"ring": {
"row_capacity": 512,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 1835008,
"sfa_bytes": 229376,
"logical_bytes": 2064384
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 50,
"producer_ms": 0.029301760196685792,
"global_scale_ms": 2.2483973693847656,
"cute_consumer_ms": 0.14257984161376952,
"modeled_cute_chunk_ms": 0.1718816018104553,
"modeled_comfy_chunk_ms": 0.2101740860939026,
"actual_into_ring_cute_gemm_ms": 0.20412223815917968,
"comfy_consumer_ms": 0.1808723258972168,
"actual_into_ring_comfy_gemm_ms": 0.21875711441040038,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.21859840393066407,
"modeled_canonical_reference_ms": 18.424679260253907,
"modeled_canonical_cute_ring_ms": 17.353442993164062,
"modeled_canonical_comfy_ring_ms": 18.436423835754393,
"actual_ring_vs_reference": 0.93377734918835,
"comfy_ring_vs_reference": 1.0007260367728332,
"modeled_canonical_cute_ring_vs_reference": 0.9418586206056386,
"modeled_canonical_comfy_ring_vs_reference": 1.0006374371751383
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-attn-qkv-2048.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "attn_qkv_proj",
"mnk": [
2048,
21504,
5376
],
"full_activation_rows": 37810,
"modeled_chunk_count": 19,
"ring": {
"row_capacity": 2048,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 5505024,
"sfa_bytes": 688128,
"logical_bytes": 6193152
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.12289536476135254,
"global_scale_ms": 1.8337306213378906,
"cute_consumer_ms": 2.898988037109375,
"modeled_cute_chunk_ms": 3.0218834018707272,
"modeled_comfy_chunk_ms": 1.69343674659729,
"actual_into_ring_cute_gemm_ms": 1.5230323791503906,
"comfy_consumer_ms": 1.5705413818359375,
"actual_into_ring_comfy_gemm_ms": 1.7044099426269532,
"reference_vortex_scale_comfy_pack_gemm_ms": 1.7054486083984375,
"modeled_canonical_reference_ms": 34.2372541809082,
"modeled_canonical_cute_ring_ms": 30.77134582519531,
"modeled_canonical_comfy_ring_ms": 34.217519531250005,
"actual_ring_vs_reference": 0.8930391520742736,
"comfy_ring_vs_reference": 0.9993909721076499,
"modeled_canonical_cute_ring_vs_reference": 0.89876792287725,
"modeled_canonical_comfy_ring_vs_reference": 0.999423591344273
}
}

78
benchmarks/gb10-cute-p2-ring-capacity-mlp-fc1-2048.json Normal file
View file

@ -0,0 +1,78 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"mnk": [
2048,
28672,
5376
],
"full_activation_rows": 37810,
"modeled_chunk_count": 19,
"ring": {
"row_capacity": 2048,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 5505024,
"sfa_bytes": 688128,
"logical_bytes": 6193152
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"comfy_output_equal": true
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.13199872016906739,
"global_scale_ms": 2.422532806396484,
"cute_consumer_ms": 6.302136840820313,
"modeled_cute_chunk_ms": 6.434135560989381,
"modeled_comfy_chunk_ms": 2.247994508743286,
"actual_into_ring_cute_gemm_ms": 2.0069894409179687,
"comfy_consumer_ms": 2.1159957885742187,
"actual_into_ring_comfy_gemm_ms": 2.2663865661621094,
"reference_vortex_scale_comfy_pack_gemm_ms": 2.26166748046875,
"modeled_canonical_reference_ms": 45.394214935302735,
"modeled_canonical_cute_ring_ms": 40.55533218383789,
"modeled_canonical_comfy_ring_ms": 45.48387756347656,
"actual_ring_vs_reference": 0.8873936855218004,
"comfy_ring_vs_reference": 1.0020865515086157,
"modeled_canonical_cute_ring_vs_reference": 0.8934030964438668,
"modeled_canonical_comfy_ring_vs_reference": 1.0019751994456036
}
}

58
benchmarks/gb10-cute-p2-ring-capacity-summary.json Normal file
View file

@ -0,0 +1,58 @@
{
"date": "2026-08-23",
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"full_activation_rows": 37810,
"global_scale_policy": "one full-activation Vortex scale reused by every ring chunk",
"attention_output_capacity_sweep": {
"128": {
"ring_bytes": 516096,
"cute_ring_vs_reference": 1.0759724609619812
},
"512": {
"ring_bytes": 2064384,
"cute_ring_vs_reference": 0.93377734918835
},
"1024": {
"ring_bytes": 4128768,
"cute_ring_vs_reference": 0.9562389611635855
},
"2048": {
"ring_bytes": 8257536,
"cute_ring_vs_reference": 0.8562034923214508,
"modeled_canonical_vs_reference": 0.8747956842123643
},
"4096": {
"ring_bytes": 16515072,
"cute_ring_vs_reference": 1.148149277409534
}
},
"selected_row_capacity": 2048,
"selected_capacity_results": {
"attn_qkv_proj": {
"ring_bytes": 6193152,
"chunk_improvement_percent": 10.69608479257264,
"modeled_canonical_improvement_percent": 10.123207712275,
"output_equal": true
},
"attn_out_proj": {
"ring_bytes": 8257536,
"chunk_improvement_percent": 14.37965076785492,
"modeled_canonical_improvement_percent": 12.52043157876357,
"output_equal": true
},
"mlp_fc1": {
"ring_bytes": 6193152,
"chunk_improvement_percent": 11.26063144781996,
"modeled_canonical_improvement_percent": 10.65969035561332,
"output_equal": true
}
},
"policy": {
"attn_qkv_proj": "advance to real-block validation",
"attn_out_proj": "advance to real-block validation",
"mlp_fc1": "advance to real-block validation",
"mlp_fc2": "retain Comfy/cuBLAS fallback"
},
"next_gate": "Execute 2048-row chunks across complete real projections in blocks 0, 24, and 49, including the final partial chunk, and compare total projection/block latency and exact outputs."
}

205
benchmarks/gb10-cute-p2-ring-full-block0-attn-out.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 0,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 25.982784271240234,
"projection_ring_total_ms": 14.904607772827148,
"results": [
{
"name": "attn_out_proj",
"mnk": [
37810,
5376,
7168
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 8257536,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": -55638368.0,
"candidate_checksum": -55638368.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 14.904607772827148,
"reference_complete_ms": 25.982784271240234,
"ring_vs_reference": 0.5736339730659553,
"improvement_percent": 42.63660269340447
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 418381824
}
],
"projection_total_improvement_percent": 42.63660269340447
}

205
benchmarks/gb10-cute-p2-ring-full-block0-fc1.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 0,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 217.1246795654297,
"projection_ring_total_ms": 176.6940155029297,
"results": [
{
"name": "mlp_fc1",
"mnk": [
37810,
28672,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": -327274368.0,
"candidate_checksum": -327274368.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 176.6940155029297,
"reference_complete_ms": 217.1246795654297,
"ring_vs_reference": 0.8137905642812181,
"improvement_percent": 18.62094357187819
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 2231369728
}
],
"projection_total_improvement_percent": 18.62094357187819
}

205
benchmarks/gb10-cute-p2-ring-full-block0-qkv.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 0,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 79.74476623535156,
"projection_ring_total_ms": 32.21440124511719,
"results": [
{
"name": "attn_qkv_proj",
"mnk": [
37810,
21504,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": -17431596.0,
"candidate_checksum": -17431596.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 32.21440124511719,
"reference_complete_ms": 79.74476623535156,
"ring_vs_reference": 0.40396884668321037,
"improvement_percent": 59.60311533167897
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 1673527296
}
],
"projection_total_improvement_percent": 59.60311533167897
}

330
benchmarks/gb10-cute-p2-ring-full-block24-attn-out-1024.json Normal file
View file

@ -0,0 +1,330 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"capacity": 1024,
"all_equal": true,
"projection_reference_total_ms": 19.279925028483074,
"projection_ring_total_ms": 21.812459309895832,
"results": [
{
"name": "attn_out_proj",
"mnk": [
37810,
5376,
7168
],
"capacity": 1024,
"chunk_count": 37,
"final_chunk_rows": 946,
"ring_bytes": 4128768,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 81086272.0,
"candidate_checksum": 81086267.875
},
"timing": {
"warmup": 1,
"iterations": 3,
"ring_complete_ms": 21.812459309895832,
"reference_complete_ms": 19.279925028483074,
"ring_vs_reference": 1.1313560233077324,
"improvement_percent": -13.135602330773244
},
"chunks": [
{
"start": 0,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 1024,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 3072,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 5120,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 7168,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 9216,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 11264,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 13312,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 15360,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 17408,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 19456,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 21504,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 23552,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 25600,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 27648,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 29696,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 31744,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 33792,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 35840,
"rows": 1024,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
]
}
],
"projection_total_improvement_percent": -13.135602330773244
}

141
benchmarks/gb10-cute-p2-ring-full-block24-attn-out-4096.json Normal file
View file

@ -0,0 +1,141 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"capacity": 4096,
"all_equal": true,
"projection_reference_total_ms": 16.191519419352215,
"projection_ring_total_ms": 18.844778696695965,
"results": [
{
"name": "attn_out_proj",
"mnk": [
37810,
5376,
7168
],
"capacity": 4096,
"chunk_count": 10,
"final_chunk_rows": 946,
"ring_bytes": 16515072,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 81086272.0,
"candidate_checksum": 81086266.0
},
"timing": {
"warmup": 1,
"iterations": 3,
"ring_complete_ms": 18.844778696695965,
"reference_complete_ms": 16.191519419352215,
"ring_vs_reference": 1.1638672201555438,
"improvement_percent": -16.386722015554376
},
"chunks": [
{
"start": 0,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 4096,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
]
}
],
"projection_total_improvement_percent": -16.386722015554376
}

205
benchmarks/gb10-cute-p2-ring-full-block24-attn-out.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 23.377840042114258,
"projection_ring_total_ms": 24.269023895263672,
"results": [
{
"name": "attn_out_proj",
"mnk": [
37810,
5376,
7168
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 8257536,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 81086272.0,
"candidate_checksum": 81086272.0
},
"timing": {
"warmup": 1,
"iterations": 2,
"ring_complete_ms": 24.269023895263672,
"reference_complete_ms": 23.377840042114258,
"ring_vs_reference": 1.0381208807804305,
"improvement_percent": -3.812088078043052
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 418381824
}
],
"projection_total_improvement_percent": -3.812088078043052
}

205
benchmarks/gb10-cute-p2-ring-full-block24-fc1.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 73.7689437866211,
"projection_ring_total_ms": 43.649391174316406,
"results": [
{
"name": "mlp_fc1",
"mnk": [
37810,
28672,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": -678324672.0,
"candidate_checksum": -678324672.0
},
"timing": {
"warmup": 1,
"iterations": 2,
"ring_complete_ms": 43.649391174316406,
"reference_complete_ms": 73.7689437866211,
"ring_vs_reference": 0.5917041634833975,
"improvement_percent": 40.82958365166025
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 2231369728
}
],
"projection_total_improvement_percent": 40.82958365166025
}

205
benchmarks/gb10-cute-p2-ring-full-block24-qkv.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 53.62971115112305,
"projection_ring_total_ms": 32.13934326171875,
"results": [
{
"name": "attn_qkv_proj",
"mnk": [
37810,
21504,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 44535508.0,
"candidate_checksum": 44535508.0
},
"timing": {
"warmup": 1,
"iterations": 2,
"ring_complete_ms": 32.13934326171875,
"reference_complete_ms": 53.62971115112305,
"ring_vs_reference": 0.5992824233409232,
"improvement_percent": 40.07175766590768
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 1673527296
}
],
"projection_total_improvement_percent": 40.07175766590768
}

205
benchmarks/gb10-cute-p2-ring-full-block49-attn-out.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 49,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 26.224416732788086,
"projection_ring_total_ms": 13.855199813842773,
"results": [
{
"name": "attn_out_proj",
"mnk": [
37810,
5376,
7168
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 8257536,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 1557006080.0,
"candidate_checksum": 1557006080.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 13.855199813842773,
"reference_complete_ms": 26.224416732788086,
"ring_vs_reference": 0.5283320485263558,
"improvement_percent": 47.16679514736442
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 418381824
}
],
"projection_total_improvement_percent": 47.16679514736442
}

205
benchmarks/gb10-cute-p2-ring-full-block49-fc1.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 49,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 242.2157440185547,
"projection_ring_total_ms": 763.5274047851562,
"results": [
{
"name": "mlp_fc1",
"mnk": [
37810,
28672,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": 8475313152.0,
"candidate_checksum": 8475313152.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 763.5274047851562,
"reference_complete_ms": 242.2157440185547,
"ring_vs_reference": 3.1522616660569636,
"improvement_percent": -215.22616660569636
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 2231369728
}
],
"projection_total_improvement_percent": -215.22616660569636
}

205
benchmarks/gb10-cute-p2-ring-full-block49-qkv.json Normal file
View file

@ -0,0 +1,205 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 49,
"capacity": 2048,
"all_equal": true,
"projection_reference_total_ms": 84.6115493774414,
"projection_ring_total_ms": 31.96544075012207,
"results": [
{
"name": "attn_qkv_proj",
"mnk": [
37810,
21504,
5376
],
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"ring_bytes": 6193152,
"parity": {
"all_chunks_equal": true,
"max_abs": 0.0,
"mean_abs_max": 0.0,
"reference_checksum": -40304712.0,
"candidate_checksum": -40304712.0
},
"timing": {
"warmup": 1,
"iterations": 1,
"ring_complete_ms": 31.96544075012207,
"reference_complete_ms": 84.6115493774414,
"ring_vs_reference": 0.37779051424207216,
"improvement_percent": 62.22094857579279
},
"chunks": [
{
"start": 0,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 2048,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 4096,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 6144,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 8192,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 10240,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 12288,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 14336,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 16384,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 18432,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 20480,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 22528,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 24576,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 26624,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 28672,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 30720,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 32768,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 34816,
"rows": 2048,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
{
"start": 36864,
"rows": 946,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
}
],
"output_bytes": 1673527296
}
],
"projection_total_improvement_percent": 62.22094857579279
}

46
benchmarks/gb10-cute-p2-ring-full-projection-summary.json Normal file
View file

@ -0,0 +1,46 @@
{
"date": "2026-08-23",
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"rows": 37810,
"capacity": 2048,
"chunk_count": 19,
"final_chunk_rows": 946,
"parity": {
"blocks": [0, 24, 49],
"roles": ["attn_qkv_proj", "attn_out_proj", "mlp_fc1"],
"all_chunks_equal": true,
"max_abs": 0.0,
"checksums_equal": true
},
"timing_percent_improvement": {
"block_0": {
"attn_qkv_proj": 59.60311533167897,
"attn_out_proj": 42.63660269340447,
"mlp_fc1": 18.62094357187819
},
"block_24": {
"attn_qkv_proj": 40.07175766590768,
"attn_out_proj": -3.812088078043052,
"mlp_fc1": 40.82958365166025
},
"block_49": {
"attn_qkv_proj": 62.22094857579279,
"attn_out_proj": 47.16679514736442,
"mlp_fc1": -215.22616660569636
}
},
"timing_interpretation": {
"attn_qkv_proj": "Consistent 32 ms ring time and positive result across all blocks; advance to in-runtime validation.",
"attn_out_proj": "Exact but timing is mixed; retain as experimental until in-runtime measurement.",
"mlp_fc1": "Exact but duplicate-model unified-memory timing is unstable; no performance approval from this harness."
},
"benchmark_caveat": "The resident service and validator hold separate model copies on unified memory. Multi-gigabyte FC1 outputs caused pressure and one combined validator run stopped the container. Timing must be repeated inside the single resident model before deployment.",
"policy": {
"attn_qkv_proj": "advance",
"attn_out_proj": "experimental",
"mlp_fc1": "parity-approved, performance-pending",
"mlp_fc2": "Comfy/cuBLAS fallback"
},
"next_gate": "Integrate only QKV behind an opt-in runtime flag, benchmark blocks 0, 24, and 49 in the resident model, then decide whether attention output or FC1 should be added."
}

67
benchmarks/gb10-cute-p2-ring-mlp-fc1.json Normal file
View file

@ -0,0 +1,67 @@
{
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"linear": "mlp_fc1",
"mnk": [
128,
28672,
5376
],
"ring": {
"row_capacity": 128,
"producer": "vortex_native_quantize_nvfp4",
"qdata_bytes": 344064,
"sfa_bytes": 43008,
"logical_bytes": 387072
},
"parity": {
"tensor_scale_equal": true,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"max_abs": 0.0,
"mean_abs": 0.0
},
"timing": {
"warmup": 10,
"iterations": 100,
"producer_ms": 0.015716160535812377,
"global_scale_ms": 0.014631999731063843,
"cute_consumer_ms": 0.6108745574951172,
"modeled_ring_ms": 0.6412227177619935,
"actual_into_ring_cute_gemm_ms": 0.6573414611816406,
"reference_vortex_scale_comfy_pack_gemm_ms": 0.5960015869140625,
"modeled_ring_vs_reference": 1.0758741785941779,
"actual_ring_vs_reference": 1.1029189780939672
}
}

49
benchmarks/gb10-cute-p2-ring-summary.json Normal file
View file

@ -0,0 +1,49 @@
{
"date": "2026-08-23",
"device": "NVIDIA GB10",
"cutlass_dsl": "4.6.2",
"row_capacity": 128,
"warmup": 10,
"iterations": 100,
"producer": "vortex_native_quantize_nvfp4_into",
"consumer": "CuTe SM121 prepacked block-scaled GEMM",
"results": {
"attn_qkv_proj": {
"ring_bytes": 387072,
"reference_ms": 0.47588897705078126,
"ring_ms": 0.4637001419067383,
"improvement_percent": 2.56127704818477,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"decision": "experimental; require full-M chunk-size validation"
},
"attn_out_proj": {
"ring_bytes": 516096,
"reference_ms": 0.10898783683776855,
"ring_ms": 0.09652607917785644,
"improvement_percent": 11.43408110618966,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"decision": "advance to chunk-size and real-block validation"
},
"mlp_fc1": {
"ring_bytes": 387072,
"reference_ms": 0.5960015869140625,
"ring_ms": 0.6573414611816406,
"improvement_percent": -10.29189780939672,
"fp4_difference_count": 0,
"block_scale_difference_count": 0,
"output_equal": true,
"decision": "reject current ring consumer; retain reference path"
}
},
"policy": {
"attn_out_proj": "continue",
"attn_qkv_proj": "experimental",
"mlp_fc1": "fallback",
"mlp_fc2": "fallback"
},
"next_gate": "Sweep ring row capacity using one global activation scale, then validate the accepted attention-output path in real blocks 0, 24, and 49."
}

27
benchmarks/gb10-cute-qkv-block0-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 0,
"hidden_shape": [
37810,
5376
],
"iterations": 6,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": 2040895232.0,
"candidate_checksum": 2040895232.0,
"baseline": {
"mean_s": 0.46145327799604274,
"p50_s": 0.46154195550479926,
"min_s": 0.4599941649939865,
"max_s": 0.4628595529939048
},
"ring": {
"mean_s": 0.4653922098320133,
"p50_s": 0.46529679899686016,
"min_s": 0.46371654397808015,
"max_s": 0.4670645549776964
},
"p50_improvement_percent": -0.8135432645455021
}

852
benchmarks/gb10-cute-qkv-block0-baseline.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 0,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.0039003503237230084,
"p50_s": 0.00361842499114573,
"p90_s": 0.004298052983358502,
"p95_s": 0.004383006482385099,
"p99_s": 0.004450969281606376,
"min_s": 0.003614665998611599,
"max_s": 0.0044679599814116955
},
"modulate_msa": {
"count": 3,
"mean_s": 0.011204699655839553,
"p50_s": 0.010762252961285412,
"p90_s": 0.012003415380604565,
"p95_s": 0.01215856068301946,
"p99_s": 0.012282676924951375,
"min_s": 0.010538140020798892,
"max_s": 0.012313705985434353
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 6.149309532095988e-06,
"p50_s": 4.4959597289562225e-06,
"p90_s": 8.566351607441903e-06,
"p95_s": 9.075150592252612e-06,
"p99_s": 9.48218978010118e-06,
"min_s": 4.368019290268421e-06,
"max_s": 9.583949577063322e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.7290663284560043e-05,
"p50_s": 8.816015906631947e-06,
"p90_s": 2.942399587482214e-05,
"p95_s": 3.199999337084591e-05,
"p99_s": 3.406079136766494e-05,
"min_s": 8.4799830801785e-06,
"max_s": 3.457599086686969e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.695992442468802e-06,
"p50_s": 3.6479905247688293e-06,
"p90_s": 3.993557766079903e-06,
"p95_s": 4.036753671243787e-06,
"p99_s": 4.071310395374894e-06,
"min_s": 3.360037226229906e-06,
"max_s": 4.079949576407671e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.001928791015719374,
"p50_s": 0.0017829880234785378,
"p90_s": 0.002139096811879426,
"p95_s": 0.002183610410429537,
"p99_s": 0.0022192212892696264,
"min_s": 0.001775261014699936,
"max_s": 0.0022281240089796484
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 1.439966338997086e-05,
"p50_s": 1.3343989849090576e-05,
"p90_s": 1.871921122074127e-05,
"p95_s": 1.9391113892197608e-05,
"p99_s": 1.9928636029362678e-05,
"min_s": 9.791983757168055e-06,
"max_s": 2.0063016563653946e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.0023788169880087176,
"p50_s": 0.002249772020149976,
"p90_s": 0.0026388911646790802,
"p95_s": 0.0026875310577452183,
"p99_s": 0.002726442972198129,
"min_s": 0.0021505079930648208,
"max_s": 0.0027361709508113563
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 7.8986631706357e-06,
"p50_s": 6.432004738599062e-06,
"p90_s": 1.0156782809644936e-05,
"p95_s": 1.062238006852567e-05,
"p99_s": 1.0994857875630258e-05,
"min_s": 6.176007445901632e-06,
"max_s": 1.1087977327406406e-05
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.03701395931420848,
"p50_s": 0.025930000992957503,
"p90_s": 0.05311313216807321,
"p95_s": 0.05651102356496267,
"p99_s": 0.05922933668247424,
"min_s": 0.025202961987815797,
"max_s": 0.05990891496185213
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 2.0917330402880907e-05,
"p50_s": 5.44002978131175e-06,
"p90_s": 4.272641381248832e-05,
"p95_s": 4.738721181638539e-05,
"p99_s": 5.1115850219503045e-05,
"min_s": 5.2639516070485115e-06,
"max_s": 5.204800982028246e-05
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.041458233628266804,
"p50_s": 0.029947368951980025,
"p90_s": 0.05808691458078101,
"p95_s": 0.061604357784381135,
"p99_s": 0.06441831234726124,
"min_s": 0.02930553094483912,
"max_s": 0.06512180098798126
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 2.4512002710253e-05,
"p50_s": 1.7551996279507875e-05,
"p90_s": 3.592000575736165e-05,
"p95_s": 3.8216006942093366e-05,
"p99_s": 4.005280788987874e-05,
"min_s": 1.547200372442603e-05,
"max_s": 4.0512008126825094e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012107417666508505,
"p50_s": 0.012135433964431286,
"p90_s": 0.012213666830211878,
"p95_s": 0.012223445938434451,
"p99_s": 0.012231269225012512,
"min_s": 0.011953593988437206,
"max_s": 0.012233225046657026
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.00539845967432484,
"p50_s": 0.005506518005859107,
"p90_s": 0.005652962799649686,
"p95_s": 0.005671268398873508,
"p99_s": 0.005685912878252567,
"min_s": 0.004999287019018084,
"max_s": 0.00568957399809733
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.005182119008774559,
"p50_s": 0.005078374990262091,
"p90_s": 0.00533373502548784,
"p95_s": 0.005365655029891059,
"p99_s": 0.005391191033413634,
"min_s": 0.0050704070017673075,
"max_s": 0.005397575034294277
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.005088572021729003,
"p50_s": 0.004968855006154627,
"p90_s": 0.005264303821604699,
"p95_s": 0.005301234923535958,
"p99_s": 0.005330779805080965,
"min_s": 0.004958695033565164,
"max_s": 0.005338166025467217
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2596873336394007,
"p50_s": 0.255025869992096,
"p90_s": 0.26813469716580585,
"p95_s": 0.2697733005625196,
"p99_s": 0.27108418327989053,
"min_s": 0.2526242269668728,
"max_s": 0.2714119039592333
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.004714257995753239,
"p50_s": 0.004697238968219608,
"p90_s": 0.004743498226162046,
"p95_s": 0.004749280633404851,
"p99_s": 0.004753906559199095,
"min_s": 0.004690471978392452,
"max_s": 0.004755063040647656
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.0720021439095335e-06,
"p50_s": 4.799978341907263e-06,
"p90_s": 5.862431135028601e-06,
"p95_s": 5.995237734168768e-06,
"p99_s": 6.101483013480901e-06,
"min_s": 4.2879837565124035e-06,
"max_s": 6.128044333308935e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.8890675467749436e-05,
"p50_s": 1.5839992556720972e-05,
"p90_s": 2.770560095086694e-05,
"p95_s": 2.9188802000135183e-05,
"p99_s": 3.037536283954978e-05,
"min_s": 1.0160030797123909e-05,
"max_s": 3.067200304940343e-05
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.722670953720808e-06,
"p50_s": 3.760040272027254e-06,
"p90_s": 3.7984107621014117e-06,
"p95_s": 3.8032070733606814e-06,
"p99_s": 3.8070441223680975e-06,
"min_s": 3.5999692045152187e-06,
"max_s": 3.8080033846199512e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.002283606658844898,
"p50_s": 0.0022722999565303326,
"p90_s": 0.002299871202558279,
"p95_s": 0.002303317608311772,
"p99_s": 0.002306074732914567,
"min_s": 0.002271756005939096,
"max_s": 0.0023067640140652657
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 1.3381339764843384e-05,
"p50_s": 1.3264012522995472e-05,
"p90_s": 1.6271998174488544e-05,
"p95_s": 1.664799638092518e-05,
"p99_s": 1.6948794946074483e-05,
"min_s": 9.856012184172869e-06,
"max_s": 1.7023994587361813e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.0028784163296222687,
"p50_s": 0.0028789869975298643,
"p90_s": 0.0029005677904933693,
"p95_s": 0.0029032653896138073,
"p99_s": 0.0029054234689101575,
"min_s": 0.0028502990026026964,
"max_s": 0.0029059629887342453
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 8.378662945081791e-06,
"p50_s": 7.951981388032436e-06,
"p90_s": 9.014387615025043e-06,
"p95_s": 9.14718839339912e-06,
"p99_s": 9.253429016098379e-06,
"min_s": 7.90401827543974e-06,
"max_s": 9.279989171773195e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.008133968998057147,
"p50_s": 0.007810881012119353,
"p90_s": 0.008587405015714466,
"p95_s": 0.008684470516163856,
"p99_s": 0.008762122916523367,
"min_s": 0.007809489965438843,
"max_s": 0.008781536016613245
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.957340666403373e-06,
"p50_s": 5.856039933860302e-06,
"p90_s": 6.444775499403477e-06,
"p95_s": 6.518367445096373e-06,
"p99_s": 6.577241001650691e-06,
"min_s": 5.424022674560547e-06,
"max_s": 6.59195939078927e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.013398274313658476,
"p50_s": 0.013059511955361813,
"p90_s": 0.013909942365717142,
"p95_s": 0.014016246167011559,
"p99_s": 0.014101289208047093,
"min_s": 0.013012761017307639,
"max_s": 0.014122549968305975
},
"gate_msa": {
"count": 3,
"mean_s": 0.008729871983329454,
"p50_s": 0.008632095996290445,
"p90_s": 0.00904886396601796,
"p95_s": 0.0091009599622339,
"p99_s": 0.009142636759206652,
"min_s": 0.00840446399524808,
"max_s": 0.00915305595844984
},
"norm2": {
"count": 3,
"mean_s": 0.0036775400318826237,
"p50_s": 0.0036539300344884396,
"p90_s": 0.0037461916450411085,
"p95_s": 0.003757724346360192,
"p99_s": 0.003766950507415459,
"min_s": 0.0036094330134801567,
"max_s": 0.0037692570476792753
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010796177666634321,
"p50_s": 0.010941468004602939,
"p90_s": 0.010954472806770353,
"p95_s": 0.01095609840704128,
"p99_s": 0.010957398887258022,
"min_s": 0.010489340987987816,
"max_s": 0.010957724007312208
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.4533129160602885e-06,
"p50_s": 4.399975296109915e-06,
"p90_s": 4.7199777327477935e-06,
"p95_s": 4.759978037327528e-06,
"p99_s": 4.7919782809913166e-06,
"min_s": 4.159985110163689e-06,
"max_s": 4.799978341907263e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 9.487984546770653e-06,
"p50_s": 9.295996278524399e-06,
"p90_s": 1.0576006025075912e-05,
"p95_s": 1.0736007243394852e-05,
"p99_s": 1.0864008218050004e-05,
"min_s": 8.271948900073767e-06,
"max_s": 1.0896008461713791e-05
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.477325662970543e-06,
"p50_s": 3.471970558166504e-06,
"p90_s": 3.548804670572281e-06,
"p95_s": 3.558408934623003e-06,
"p99_s": 3.5660923458635807e-06,
"min_s": 3.3919932320713997e-06,
"max_s": 3.568013198673725e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017434633142935734,
"p50_s": 0.0017429409781470895,
"p90_s": 0.0017490209778770803,
"p95_s": 0.0017497809778433291,
"p99_s": 0.0017503889778163284,
"min_s": 0.0017369079869240522,
"max_s": 0.0017505409778095782
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 1.0010650536666313e-05,
"p50_s": 9.999959729611874e-06,
"p90_s": 1.058878842741251e-05,
"p95_s": 1.066239201463759e-05,
"p99_s": 1.0721274884417652e-05,
"min_s": 9.295996278524399e-06,
"max_s": 1.0735995601862669e-05
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.0022977933404035866,
"p50_s": 0.002274715981911868,
"p90_s": 0.0024394648033194246,
"p95_s": 0.002460058405995369,
"p99_s": 0.0024765332881361247,
"min_s": 0.0021380120306275785,
"max_s": 0.0024806520086713135
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.543996278196573e-06,
"p50_s": 6.464018952101469e-06,
"p90_s": 6.7327986471354965e-06,
"p95_s": 6.766396109014749e-06,
"p99_s": 6.793274078518152e-06,
"min_s": 6.367976311594248e-06,
"max_s": 6.799993570894003e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.03486676298780367,
"p50_s": 0.03488327999366447,
"p90_s": 0.0348947360063903,
"p95_s": 0.03489616800798103,
"p99_s": 0.03489731360925361,
"min_s": 0.03481940896017477,
"max_s": 0.03489760000957176
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.68533626695474e-06,
"p50_s": 5.6480057537555695e-06,
"p90_s": 5.852803587913513e-06,
"p95_s": 5.878403317183256e-06,
"p99_s": 5.89888310059905e-06,
"min_s": 5.504000000655651e-06,
"max_s": 5.904003046452999e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.038996499322820455,
"p50_s": 0.03899847296997905,
"p90_s": 0.03917223219759762,
"p95_s": 0.03919395210104994,
"p99_s": 0.039211328023811796,
"min_s": 0.03877535299398005,
"max_s": 0.03921567200450227
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.02435958234127611,
"p50_s": 0.02435267501277849,
"p90_s": 0.024421091785188764,
"p95_s": 0.024429643881740046,
"p99_s": 0.024436485558981078,
"min_s": 0.024287876032758504,
"max_s": 0.024438195978291333
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.271666208902995e-06,
"p50_s": 4.208006430417299e-06,
"p90_s": 4.579185042530298e-06,
"p95_s": 4.625582369044423e-06,
"p99_s": 4.662700230255723e-06,
"min_s": 3.935012500733137e-06,
"max_s": 4.671979695558548e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.533359505236149e-06,
"p50_s": 8.704024367034435e-06,
"p90_s": 9.216018952429295e-06,
"p95_s": 9.280018275603652e-06,
"p99_s": 9.331217734143137e-06,
"min_s": 7.552036549896002e-06,
"max_s": 9.34401759877801e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.471989960720142e-06,
"p50_s": 3.5199918784201145e-06,
"p90_s": 3.5199918784201145e-06,
"p95_s": 3.5199918784201145e-06,
"p99_s": 3.5199918784201145e-06,
"min_s": 3.375986125320196e-06,
"max_s": 3.5199918784201145e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.004591644974425435,
"p50_s": 0.004597686987835914,
"p90_s": 0.00461162697756663,
"p95_s": 0.00461336947628297,
"p99_s": 0.00461476347525604,
"min_s": 0.004562135960441083,
"max_s": 0.004615111974999309
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 9.818662268420061e-06,
"p50_s": 9.311945177614689e-06,
"p90_s": 1.0924809612333774e-05,
"p95_s": 1.112641766667366e-05,
"p99_s": 1.1287704110145569e-05,
"min_s": 8.816015906631947e-06,
"max_s": 1.1328025721013546e-05
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.006046714649225275,
"p50_s": 0.006064180983230472,
"p90_s": 0.006119732977822423,
"p95_s": 0.006126676977146417,
"p99_s": 0.006132232176605612,
"min_s": 0.005942341987974942,
"max_s": 0.006133620976470411
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.224009363601605e-06,
"p50_s": 6.208021659404039e-06,
"p90_s": 6.40001380816102e-06,
"p95_s": 6.4240128267556436e-06,
"p99_s": 6.443212041631341e-06,
"min_s": 6.01599458605051e-06,
"max_s": 6.4480118453502655e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.05645514999438698,
"p50_s": 0.05642460100352764,
"p90_s": 0.05665761139243841,
"p95_s": 0.05668673769105226,
"p99_s": 0.05671003872994333,
"min_s": 0.0562249849899672,
"max_s": 0.056715863989666104
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.6533220534523325e-06,
"p50_s": 5.71197597309947e-06,
"p90_s": 5.724781658500433e-06,
"p95_s": 5.726382369175553e-06,
"p99_s": 5.72766293771565e-06,
"min_s": 5.520007107406855e-06,
"max_s": 5.727983079850674e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.06717234667545806,
"p50_s": 0.0670450460165739,
"p90_s": 0.06738878921605647,
"p95_s": 0.06743175711599178,
"p99_s": 0.06746613143594005,
"min_s": 0.06699726899387315,
"max_s": 0.0674747250159271
},
"gate_mlp": {
"count": 3,
"mean_s": 0.018475912996412564,
"p50_s": 0.008704703999683261,
"p90_s": 0.032231163978576664,
"p95_s": 0.035171971475938334,
"p99_s": 0.037524617473827675,
"min_s": 0.008610256016254425,
"max_s": 0.03811277897330001
},
"block_total": {
"count": 3,
"mean_s": 0.5345096856666108,
"p50_s": 0.535079502966255,
"p90_s": 0.5420267941895872,
"p95_s": 0.5428952055925038,
"p99_s": 0.5435899347148371,
"min_s": 0.5246859370381571,
"max_s": 0.5437636169954203
}
},
"module_forward": {
"count": 3,
"mean_s": 0.46528928333039704,
"p50_s": 0.4646527419681661,
"p90_s": 0.46618508918909357,
"p95_s": 0.4663766325917095,
"p99_s": 0.46652986731380225,
"min_s": 0.46464693202869967,
"max_s": 0.46656817599432543
},
"module_forward_checksum": 606380672.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 9093546496
},
"profiler_top_events": [
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1360.0130000000001,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -303104,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1339.1809999999998,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 603.1990000000001,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [21504, 5376], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 527.5030000000006,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 445.02400000000034,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]"
},
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 265.37599999999975,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 222.23999999999978,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 161.48800000000028,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "aten::to",
"count": 12,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 160.35200000000077,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 154.1279999999988,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 148.03099999999995,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "aten::empty",
"count": 34,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 141.58399999999915,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 8009298432,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 138.3199999999997,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 132.09499999999935,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "aten::zeros",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 122.49499999999898,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 112.32000000000062,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 99.2470000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 93.92000000000007,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 81.16800000000057,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]"
},
{
"key": "aten::to",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 79.10399999999981,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], []]"
}
]
}

852
benchmarks/gb10-cute-qkv-block0-ring.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 0,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.0038844093408746025,
"p50_s": 0.003442010027356446,
"p90_s": 0.004506341204978525,
"p95_s": 0.004639382602181286,
"p99_s": 0.004745815719943493,
"min_s": 0.003438793995883316,
"max_s": 0.004772423999384046
},
"modulate_msa": {
"count": 3,
"mean_s": 0.011301896010991186,
"p50_s": 0.010341919027268887,
"p90_s": 0.012674343818798662,
"p95_s": 0.012965896917739883,
"p99_s": 0.01319913939689286,
"min_s": 0.010306318989023566,
"max_s": 0.013257450016681105
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.6800199672579765e-06,
"p50_s": 4.656030796468258e-06,
"p90_s": 7.612793706357479e-06,
"p95_s": 7.982389070093631e-06,
"p99_s": 8.278065361082553e-06,
"min_s": 4.032044671475887e-06,
"max_s": 8.351984433829784e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.581335285057624e-05,
"p50_s": 8.608039934188128e-06,
"p90_s": 2.631038660183549e-05,
"p95_s": 2.8523179935291406e-05,
"p99_s": 3.0293414602056147e-05,
"min_s": 8.096045348793268e-06,
"max_s": 3.073597326874733e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.535998985171318e-06,
"p50_s": 3.535998985171318e-06,
"p90_s": 3.6255922168493273e-06,
"p95_s": 3.636791370809078e-06,
"p99_s": 3.645750693976879e-06,
"min_s": 3.4240074455738068e-06,
"max_s": 3.6479905247688293e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.002059137332253158,
"p50_s": 0.0018270040163770318,
"p90_s": 0.0024354639695957302,
"p95_s": 0.0025115214637480674,
"p99_s": 0.0025723674590699373,
"min_s": 0.0017628290224820375,
"max_s": 0.002587578957900405
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 2.2490644672264654e-05,
"p50_s": 2.297596074640751e-05,
"p90_s": 2.3987190797924995e-05,
"p95_s": 2.411359455436468e-05,
"p99_s": 2.421471755951643e-05,
"min_s": 2.025597495958209e-05,
"max_s": 2.4239998310804367e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002397505333647132,
"p50_s": 0.0022308279876597226,
"p90_s": 0.0026664879755117,
"p95_s": 0.002720945473993197,
"p99_s": 0.002764511472778395,
"min_s": 0.002186285040806979,
"max_s": 0.0027754029724746943
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 7.466665313889583e-06,
"p50_s": 7.599999662488699e-06,
"p90_s": 7.996789645403623e-06,
"p95_s": 8.046388393267989e-06,
"p99_s": 8.086067391559481e-06,
"min_s": 6.704009138047695e-06,
"max_s": 8.095987141132355e-06
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.039801057001265384,
"p50_s": 0.025940386985894293,
"p90_s": 0.059222071792464706,
"p95_s": 0.063382282393286,
"p99_s": 0.06671045087394303,
"min_s": 0.02592029102379456,
"max_s": 0.0675424929941073
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 6.346691710253556e-06,
"p50_s": 5.4720439948141575e-06,
"p90_s": 7.63522693887353e-06,
"p95_s": 7.90562480688095e-06,
"p99_s": 8.121943101286887e-06,
"min_s": 5.39200846105814e-06,
"max_s": 8.176022674888372e-06
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.04436540632741526,
"p50_s": 0.03002823598217219,
"p90_s": 0.06444033759180456,
"p95_s": 0.06874185029300861,
"p99_s": 0.07218306045397184,
"min_s": 0.030024620005860925,
"max_s": 0.07304336299421266
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 1.6122687763224047e-05,
"p50_s": 1.315202098339796e-05,
"p90_s": 2.1305610425770285e-05,
"p95_s": 2.2324809106066822e-05,
"p99_s": 2.3140168050304055e-05,
"min_s": 1.1872034519910812e-05,
"max_s": 2.3344007786363363e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012314411326466749,
"p50_s": 0.012499674980062991,
"p90_s": 0.012545524595770985,
"p95_s": 0.012551255797734485,
"p99_s": 0.012555840759305284,
"min_s": 0.011886571999639273,
"max_s": 0.012556986999697983
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.005020225983268271,
"p50_s": 0.005004551960155368,
"p90_s": 0.005060359183698892,
"p95_s": 0.0050673350866418335,
"p99_s": 0.005072915808996186,
"min_s": 0.004981815000064671,
"max_s": 0.005074310989584774
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.005056855365789185,
"p50_s": 0.005042263015639037,
"p90_s": 0.0050791406421922146,
"p95_s": 0.0050837503455113614,
"p99_s": 0.005087438108166679,
"min_s": 0.005039943032898009,
"max_s": 0.005088360048830509
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.005133500671945512,
"p50_s": 0.005174166988581419,
"p90_s": 0.0052229606080800295,
"p95_s": 0.005229059810517356,
"p99_s": 0.005233939172467217,
"min_s": 0.004991176014300436,
"max_s": 0.005235159012954682
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2589895313140005,
"p50_s": 0.25650634698104113,
"p90_s": 0.26326487499754875,
"p95_s": 0.26410969099961223,
"p99_s": 0.264785543801263,
"min_s": 0.25550773995928466,
"max_s": 0.26495450700167567
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.0046977306677338975,
"p50_s": 0.004640695988200605,
"p90_s": 0.004778039990924299,
"p95_s": 0.0047952079912647605,
"p99_s": 0.00480894239153713,
"min_s": 0.004640120023395866,
"max_s": 0.004812375991605222
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 4.453351721167564e-06,
"p50_s": 4.191999323666096e-06,
"p90_s": 4.870421253144741e-06,
"p95_s": 4.955223994329571e-06,
"p99_s": 5.023066187277436e-06,
"min_s": 4.128029104322195e-06,
"max_s": 5.040026735514402e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.0901324761410555e-05,
"p50_s": 9.343959391117096e-06,
"p90_s": 1.3158377259969712e-05,
"p95_s": 1.3635179493576289e-05,
"p99_s": 1.401662128046155e-05,
"min_s": 9.248033165931702e-06,
"max_s": 1.4111981727182865e-05
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.471989960720142e-06,
"p50_s": 3.3919932320713997e-06,
"p90_s": 3.6095967516303063e-06,
"p95_s": 3.6367971915751694e-06,
"p99_s": 3.6585575435310603e-06,
"min_s": 3.3599790185689926e-06,
"max_s": 3.663997631520033e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.002189169346820563,
"p50_s": 0.0021924120374023914,
"p90_s": 0.002197237592190504,
"p95_s": 0.0021978407865390183,
"p99_s": 0.0021983233420178293,
"min_s": 0.0021766520221717656,
"max_s": 0.0021984439808875322
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 1.1679988044003645e-05,
"p50_s": 1.0255957022309303e-05,
"p90_s": 1.3852771371603011e-05,
"p95_s": 1.4302373165264725e-05,
"p99_s": 1.4662054600194098e-05,
"min_s": 1.0032032150775194e-05,
"max_s": 1.475197495892644e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002833104692399502,
"p50_s": 0.0028447150252759457,
"p90_s": 0.0028486958239227534,
"p95_s": 0.002849193423753604,
"p99_s": 0.002849591503618285,
"min_s": 0.0028049080283381045,
"max_s": 0.0028496910235844553
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 7.119980485488971e-06,
"p50_s": 6.671994924545288e-06,
"p90_s": 7.926393300294876e-06,
"p95_s": 8.083193097263575e-06,
"p99_s": 8.208632934838535e-06,
"min_s": 6.447953637689352e-06,
"max_s": 8.239992894232273e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.008127884687079737,
"p50_s": 0.007850803027395159,
"p90_s": 0.008538391825277358,
"p95_s": 0.008624340425012632,
"p99_s": 0.008693099304800852,
"min_s": 0.007822562009096146,
"max_s": 0.008710289024747908
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.770668697853883e-06,
"p50_s": 5.69602707400918e-06,
"p90_s": 6.054400000721216e-06,
"p95_s": 6.099196616560221e-06,
"p99_s": 6.1350339092314244e-06,
"min_s": 5.471985787153244e-06,
"max_s": 6.143993232399225e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.013236238659980396,
"p50_s": 0.01298735401360318,
"p90_s": 0.01366794117493555,
"p95_s": 0.013753014570102095,
"p99_s": 0.013821073286235334,
"min_s": 0.012883274001069367,
"max_s": 0.013838087965268642
},
"gate_msa": {
"count": 3,
"mean_s": 0.00891906803008169,
"p50_s": 0.008913073048461229,
"p90_s": 0.009082570637110621,
"p95_s": 0.009103757835691794,
"p99_s": 0.009120707594556734,
"min_s": 0.00871918600751087,
"max_s": 0.009124945034272969
},
"norm2": {
"count": 3,
"mean_s": 0.0035535029989356795,
"p50_s": 0.0035576900118030608,
"p90_s": 0.0036055227858014403,
"p95_s": 0.003611501882551238,
"p99_s": 0.003616285159951076,
"min_s": 0.0034853380057029426,
"max_s": 0.0036174809793010354
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010866791980030635,
"p50_s": 0.010817836970090866,
"p90_s": 0.010982598597183823,
"p95_s": 0.011003193800570443,
"p99_s": 0.011019669963279738,
"min_s": 0.010758749966043979,
"max_s": 0.011023789003957063
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.4586680208643275e-06,
"p50_s": 4.352012183517218e-06,
"p90_s": 4.8128305934369566e-06,
"p95_s": 4.870432894676924e-06,
"p99_s": 4.916514735668898e-06,
"min_s": 4.0959566831588745e-06,
"max_s": 4.928035195916891e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 9.039998985826969e-06,
"p50_s": 8.944014552980661e-06,
"p90_s": 9.775965008884668e-06,
"p95_s": 9.879958815872669e-06,
"p99_s": 9.963153861463071e-06,
"min_s": 8.192029781639576e-06,
"max_s": 9.98395262286067e-06
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.536018387724956e-06,
"p50_s": 3.648048732429743e-06,
"p90_s": 3.660807851701975e-06,
"p95_s": 3.662402741611004e-06,
"p99_s": 3.663678653538227e-06,
"min_s": 3.296008799225092e-06,
"max_s": 3.663997631520033e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017847650063534577,
"p50_s": 0.0017553410143591464,
"p90_s": 0.0018440961954183876,
"p95_s": 0.0018551905930507928,
"p99_s": 0.001864066111156717,
"min_s": 0.001732669014018029,
"max_s": 0.001866284990683198
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 1.0303997745116552e-05,
"p50_s": 1.047999830916524e-05,
"p90_s": 1.0748824570327997e-05,
"p95_s": 1.0782427852973343e-05,
"p99_s": 1.0809310479089617e-05,
"min_s": 9.615963790565729e-06,
"max_s": 1.0816031135618687e-05
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.0021633559954352677,
"p50_s": 0.0021477399859577417,
"p90_s": 0.002185807190835476,
"p95_s": 0.0021905655914451925,
"p99_s": 0.0021943723119329664,
"min_s": 0.002147004008293152,
"max_s": 0.0021953239920549095
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.517337169498205e-06,
"p50_s": 6.352027412503958e-06,
"p90_s": 6.799993570894003e-06,
"p95_s": 6.8559893406927586e-06,
"p99_s": 6.900785956531763e-06,
"min_s": 6.287998985499144e-06,
"max_s": 6.911985110491514e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.03497030300786719,
"p50_s": 0.035002851975150406,
"p90_s": 0.035098020010627805,
"p95_s": 0.035109916015062484,
"p99_s": 0.03511943281861022,
"min_s": 0.034786245028954,
"max_s": 0.035121812019497156
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.631998647004366e-06,
"p50_s": 5.536014214158058e-06,
"p90_s": 5.855970084667206e-06,
"p95_s": 5.895964568480849e-06,
"p99_s": 5.927960155531764e-06,
"min_s": 5.424022674560547e-06,
"max_s": 5.935959052294493e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.03900541601857791,
"p50_s": 0.038999997021164745,
"p90_s": 0.03917357780737803,
"p95_s": 0.039195275405654685,
"p99_s": 0.03921263348427601,
"min_s": 0.03879927803063765,
"max_s": 0.039216973003931344
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.0237522956643564,
"p50_s": 0.023783078999258578,
"p90_s": 0.023799194977618756,
"p95_s": 0.023801209474913775,
"p99_s": 0.023802821072749794,
"min_s": 0.023670584021601826,
"max_s": 0.023803223972208798
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.831973152856032e-06,
"p50_s": 5.007954314351082e-06,
"p90_s": 5.251169204711914e-06,
"p95_s": 5.281571066007018e-06,
"p99_s": 5.305892555043101e-06,
"min_s": 4.175992216914892e-06,
"max_s": 5.311972927302122e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.869334124028683e-06,
"p50_s": 8.800008799880743e-06,
"p90_s": 9.478384163230659e-06,
"p95_s": 9.563181083649398e-06,
"p99_s": 9.63101861998439e-06,
"min_s": 8.160015568137169e-06,
"max_s": 9.647978004068136e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.3919932320713997e-06,
"p50_s": 3.3760443329811096e-06,
"p90_s": 3.452785313129425e-06,
"p95_s": 3.4623779356479647e-06,
"p99_s": 3.470052033662796e-06,
"min_s": 3.3279648050665855e-06,
"max_s": 3.471970558166504e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.004335538988622527,
"p50_s": 0.004312007979024202,
"p90_s": 0.004394120781216771,
"p95_s": 0.004404384881490842,
"p99_s": 0.004412596161710098,
"min_s": 0.004279960005078465,
"max_s": 0.004414648981764913
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 1.5887975071867306e-05,
"p50_s": 1.0015966836363077e-05,
"p90_s": 2.4607975501567125e-05,
"p95_s": 2.643197658471763e-05,
"p99_s": 2.7891177451238038e-05,
"min_s": 9.391980711370707e-06,
"max_s": 2.8255977667868137e-05
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.005852811639973273,
"p50_s": 0.005791350966319442,
"p90_s": 0.005944655789062381,
"p95_s": 0.005963818891905248,
"p99_s": 0.005979149374179542,
"min_s": 0.005784101958852261,
"max_s": 0.0059829819947481155
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.6400001135965186e-06,
"p50_s": 6.6400389187037945e-06,
"p90_s": 6.742391269654036e-06,
"p95_s": 6.7551853135228155e-06,
"p99_s": 6.7654205486178394e-06,
"min_s": 6.511982064694166e-06,
"max_s": 6.767979357391596e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.056130341331784926,
"p50_s": 0.0560961120063439,
"p90_s": 0.0565045592142269,
"p95_s": 0.056555615115212274,
"p99_s": 0.056596459836000576,
"min_s": 0.05568824097281322,
"max_s": 0.05660667101619765
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.583996729304393e-06,
"p50_s": 5.599984433501959e-06,
"p90_s": 5.676771979779005e-06,
"p95_s": 5.686370423063636e-06,
"p99_s": 5.694049177691341e-06,
"min_s": 5.456036888062954e-06,
"max_s": 5.695968866348267e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.06640448931527014,
"p50_s": 0.06639227096457034,
"p90_s": 0.06670863500330597,
"p95_s": 0.06674818050814792,
"p99_s": 0.06677981691202148,
"min_s": 0.06603347096825019,
"max_s": 0.06678772601298988
},
"gate_mlp": {
"count": 3,
"mean_s": 0.01474788433794553,
"p50_s": 0.008609394019003958,
"p90_s": 0.023345611605327575,
"p95_s": 0.025187638803618028,
"p99_s": 0.02666126056225039,
"min_s": 0.008604592992924154,
"max_s": 0.02702966600190848
},
"block_total": {
"count": 3,
"mean_s": 0.5313544006785378,
"p50_s": 0.52417680202052,
"p90_s": 0.5474372819997371,
"p95_s": 0.5503448419971392,
"p99_s": 0.5526708899950609,
"min_s": 0.5166339980205521,
"max_s": 0.5532524019945413
}
},
"module_forward": {
"count": 3,
"mean_s": 0.46708817364803207,
"p50_s": 0.4669776560040191,
"p90_s": 0.467878839164041,
"p95_s": 0.4679914870590437,
"p99_s": 0.4680816053750459,
"min_s": 0.4661827299860306,
"max_s": 0.46810413495404646
},
"module_forward_checksum": 606380672.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 7351755264
},
"profiler_top_events": [
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 512.799,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 465.9349999999995,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 369.58299999999963,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 335.5350000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -303104,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 325.1509999999997,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 236.92700000000127,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "aten::to",
"count": 13,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 206.33499999999913,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 194.91200000000026,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [], [], [], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 159.58399999999983,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 136.27199999999993,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "aten::empty",
"count": 29,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 133.95199999999943,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 6267507200,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 133.83899999999994,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 125.10400000000027,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 112.76800000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 111.69599999999991,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "aten::zeros",
"count": 3,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 102.35199999999986,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 87.96799999999985,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]"
},
{
"key": "aten::mul",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 80.92799999999943,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 2048,
"input_shapes": "[[], []]"
},
{
"key": "aten::slice",
"count": 19,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 80.81600000000071,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 65.34299999999985,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
}
]
}

27
benchmarks/gb10-cute-qkv-block24-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"iterations": 10,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -1751403776.0,
"candidate_checksum": -1751403776.0,
"baseline": {
"mean_s": 0.46800651020603257,
"p50_s": 0.4678640030033421,
"min_s": 0.4662869300227612,
"max_s": 0.46959813201101497
},
"ring": {
"mean_s": 0.4705851764010731,
"p50_s": 0.4703063364722766,
"min_s": 0.46895833703456447,
"max_s": 0.472499904979486
},
"p50_improvement_percent": -0.5220178199768499
}

852
benchmarks/gb10-cute-qkv-block24-baseline.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 24,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.0036721593351103365,
"p50_s": 0.0036359680234454572,
"p90_s": 0.00376709041884169,
"p95_s": 0.003783480718266219,
"p99_s": 0.003796592957805842,
"min_s": 0.0035806389641948044,
"max_s": 0.003799871017690748
},
"modulate_msa": {
"count": 3,
"mean_s": 0.010726557637099177,
"p50_s": 0.010602172988001257,
"p90_s": 0.010938596178311855,
"p95_s": 0.010980649077100679,
"p99_s": 0.01101429139613174,
"min_s": 0.010554797947406769,
"max_s": 0.011022701975889504
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.6746648624539375e-06,
"p50_s": 4.847999662160873e-06,
"p90_s": 6.84480182826519e-06,
"p95_s": 7.094402099028229e-06,
"p99_s": 7.294082315638661e-06,
"min_s": 4.83199255540967e-06,
"max_s": 7.344002369791269e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.602132882302006e-05,
"p50_s": 1.003197394311428e-05,
"p90_s": 2.5776028633117675e-05,
"p95_s": 2.7744035469368098e-05,
"p99_s": 2.9318440938368442e-05,
"min_s": 8.319970220327377e-06,
"max_s": 2.9712042305618525e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.562658093869686e-06,
"p50_s": 3.5199918784201145e-06,
"p90_s": 3.68637265637517e-06,
"p95_s": 3.7071702536195515e-06,
"p99_s": 3.7238083314150572e-06,
"min_s": 3.4400145523250103e-06,
"max_s": 3.7279678508639336e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.001872149354312569,
"p50_s": 0.0017504799761809409,
"p90_s": 0.002067510422784835,
"p95_s": 0.0021071392286103217,
"p99_s": 0.0021388422732707114,
"min_s": 0.0017192000523209572,
"max_s": 0.0021467680344358087
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 1.5653010147313278e-05,
"p50_s": 1.8463993910700083e-05,
"p90_s": 1.87960104085505e-05,
"p95_s": 1.8837512470781803e-05,
"p99_s": 1.8870714120566843e-05,
"min_s": 9.616021998226643e-06,
"max_s": 1.8879014533013105e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002274436662749698,
"p50_s": 0.0022630870225839317,
"p90_s": 0.002373551798518747,
"p95_s": 0.002387359895510599,
"p99_s": 0.0023984063731040807,
"min_s": 0.0021590549731627107,
"max_s": 0.002401167992502451
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.943999323993921e-06,
"p50_s": 6.480026058852673e-06,
"p90_s": 7.785600610077381e-06,
"p95_s": 7.948797428980469e-06,
"p99_s": 8.07935488410294e-06,
"min_s": 6.239977665245533e-06,
"max_s": 8.111994247883558e-06
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.02569908833053584,
"p50_s": 0.025752091023605317,
"p90_s": 0.02613150858087465,
"p95_s": 0.02617893577553332,
"p99_s": 0.026216877531260254,
"min_s": 0.025118810997810215,
"max_s": 0.026226362970191985
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 6.245332770049572e-06,
"p50_s": 5.568028427660465e-06,
"p90_s": 7.244781590998172e-06,
"p95_s": 7.454375736415386e-06,
"p99_s": 7.622051052749157e-06,
"min_s": 5.504000000655651e-06,
"max_s": 7.6639698818326e-06
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.029945935336096834,
"p50_s": 0.02986503398278728,
"p90_s": 0.030461424414534123,
"p95_s": 0.030535973218502478,
"p99_s": 0.03059561226167716,
"min_s": 0.029362250003032386,
"max_s": 0.030610522022470832
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 1.9002686409900587e-05,
"p50_s": 2.0368024706840515e-05,
"p90_s": 2.22496222704649e-05,
"p95_s": 2.2484821965917948e-05,
"p99_s": 2.2672981722280386e-05,
"min_s": 1.392001286149025e-05,
"max_s": 2.2720021661370993e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012115544329086939,
"p50_s": 0.012150077964179218,
"p90_s": 0.012263204413466157,
"p95_s": 0.012277345219627023,
"p99_s": 0.012288657864555717,
"min_s": 0.01190506899729371,
"max_s": 0.01229148602578789
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.005235358665231615,
"p50_s": 0.005258975026663393,
"p90_s": 0.005414558190386742,
"p95_s": 0.005434006085852161,
"p99_s": 0.005449564402224496,
"min_s": 0.004993646987713873,
"max_s": 0.00545345398131758
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.005438665684778243,
"p50_s": 0.00541571102803573,
"p90_s": 0.005487903009634465,
"p95_s": 0.005496927007334306,
"p99_s": 0.00550414620549418,
"min_s": 0.005394335021264851,
"max_s": 0.005505951005034149
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.0051273749947237475,
"p50_s": 0.005076878995168954,
"p90_s": 0.0052603541989810765,
"p95_s": 0.005283288599457592,
"p99_s": 0.005301636119838804,
"min_s": 0.00499902298906818,
"max_s": 0.005306222999934107
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2554921513268103,
"p50_s": 0.255616124952212,
"p90_s": 0.25568432337604463,
"p95_s": 0.2556928481790237,
"p99_s": 0.255699668021407,
"min_s": 0.2551589560462162,
"max_s": 0.2557013729820028
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.004698617655473451,
"p50_s": 0.0046985429944470525,
"p90_s": 0.004699272592552007,
"p95_s": 0.004699363792315126,
"p99_s": 0.004699436752125621,
"min_s": 0.004697854979895055,
"max_s": 0.004699454992078245
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 4.762647828708093e-06,
"p50_s": 4.624016582965851e-06,
"p90_s": 5.2511692047119146e-06,
"p95_s": 5.329563282430173e-06,
"p99_s": 5.392278544604778e-06,
"min_s": 4.255969543009996e-06,
"max_s": 5.40795736014843e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.0826683137565851e-05,
"p50_s": 1.0176037903875113e-05,
"p90_s": 1.1814420577138663e-05,
"p95_s": 1.2019218411296606e-05,
"p99_s": 1.218305667862296e-05,
"min_s": 1.0079995263367891e-05,
"max_s": 1.222401624545455e-05
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.573348900924126e-06,
"p50_s": 3.5840203054249287e-06,
"p90_s": 3.6223907954990866e-06,
"p95_s": 3.627187106758356e-06,
"p99_s": 3.6310241557657717e-06,
"min_s": 3.5040429793298244e-06,
"max_s": 3.631983418017626e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.002271252994736036,
"p50_s": 0.002267103991471231,
"p90_s": 0.002277253591455519,
"p95_s": 0.002278522291453555,
"p99_s": 0.002279537251451984,
"min_s": 0.0022668640012852848,
"max_s": 0.0022797909914515913
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 1.067200597996513e-05,
"p50_s": 1.0624004062265158e-05,
"p90_s": 1.105921110138297e-05,
"p95_s": 1.1113611981272697e-05,
"p99_s": 1.1157132685184478e-05,
"min_s": 1.022400101646781e-05,
"max_s": 1.1168012861162424e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.0028463726630434394,
"p50_s": 0.0028423359617590904,
"p90_s": 0.00285377842374146,
"p95_s": 0.002855208731489256,
"p99_s": 0.002856352977687493,
"min_s": 0.0028401429881341755,
"max_s": 0.002856639039237052
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.75199165319403e-06,
"p50_s": 6.671994924545288e-06,
"p90_s": 7.1072019636631005e-06,
"p95_s": 7.1616028435528275e-06,
"p99_s": 7.205123547464608e-06,
"min_s": 6.367976311594248e-06,
"max_s": 7.2160037234425545e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.00781480900089567,
"p50_s": 0.007808863010723144,
"p90_s": 0.007827895798254759,
"p95_s": 0.00783027489669621,
"p99_s": 0.00783217817544937,
"min_s": 0.007802909996826202,
"max_s": 0.007832653995137662
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.680000564704339e-06,
"p50_s": 5.664012860506773e-06,
"p90_s": 5.791976582258939e-06,
"p95_s": 5.80797204747796e-06,
"p99_s": 5.820768419653177e-06,
"min_s": 5.552021320909262e-06,
"max_s": 5.8239675126969814e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.013016370648983866,
"p50_s": 0.013008972979150712,
"p90_s": 0.013036953774280846,
"p95_s": 0.013040451373672112,
"p99_s": 0.013043249453185126,
"min_s": 0.012996189994737506,
"max_s": 0.01304394897306338
},
"gate_msa": {
"count": 3,
"mean_s": 0.008527667979554584,
"p50_s": 0.008532046980690211,
"p90_s": 0.00853673176607117,
"p95_s": 0.00853731736424379,
"p99_s": 0.008537785842781887,
"min_s": 0.00851305399555713,
"max_s": 0.00853790296241641
},
"norm2": {
"count": 3,
"mean_s": 0.0035164790072788796,
"p50_s": 0.0035144950379617512,
"p90_s": 0.003523019806016237,
"p95_s": 0.0035240854020230474,
"p99_s": 0.0035249378788284958,
"min_s": 0.0035097909858450294,
"max_s": 0.003525150998029858
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010573251328120628,
"p50_s": 0.010518733994103968,
"p90_s": 0.010858023609034716,
"p95_s": 0.01090043481090106,
"p99_s": 0.010934363772394136,
"min_s": 0.010258173977490515,
"max_s": 0.010942846012767404
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.634668584913015e-06,
"p50_s": 4.608009476214647e-06,
"p90_s": 5.004799459129572e-06,
"p95_s": 5.054398206993937e-06,
"p99_s": 5.09407720528543e-06,
"min_s": 4.191999323666096e-06,
"max_s": 5.103996954858303e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.896032037834326e-06,
"p50_s": 8.848030120134354e-06,
"p90_s": 9.232014417648316e-06,
"p95_s": 9.28001245483756e-06,
"p99_s": 9.318410884588956e-06,
"min_s": 8.51205550134182e-06,
"max_s": 9.328010492026806e-06
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.4399951497713723e-06,
"p50_s": 3.439956344664097e-06,
"p90_s": 3.516790457069874e-06,
"p95_s": 3.526394721120596e-06,
"p99_s": 3.5340781323611736e-06,
"min_s": 3.3440301194787025e-06,
"max_s": 3.535998985171318e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017615193501114845,
"p50_s": 0.0017409750143997371,
"p90_s": 0.0018063198192976415,
"p95_s": 0.0018144879199098797,
"p99_s": 0.00182102240039967,
"min_s": 0.0017209270154125988,
"max_s": 0.0018226560205221176
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 9.466661140322685e-06,
"p50_s": 9.503972250968218e-06,
"p90_s": 9.542389307171106e-06,
"p95_s": 9.547191439196467e-06,
"p99_s": 9.551033144816756e-06,
"min_s": 9.34401759877801e-06,
"max_s": 9.551993571221828e-06
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.002172234351746738,
"p50_s": 0.0021623510401695967,
"p90_s": 0.0021976286079734564,
"p95_s": 0.002202038303948939,
"p99_s": 0.002205566060729325,
"min_s": 0.002147904015146196,
"max_s": 0.0022064479999244213
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.416017034401496e-06,
"p50_s": 6.399990525096655e-06,
"p90_s": 6.489630322903395e-06,
"p95_s": 6.500835297629237e-06,
"p99_s": 6.509799277409911e-06,
"min_s": 6.336020305752754e-06,
"max_s": 6.51204027235508e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.07115556466548394,
"p50_s": 0.07123408204643056,
"p90_s": 0.07148396357661113,
"p95_s": 0.0715151987678837,
"p99_s": 0.07154018692090176,
"min_s": 0.07068617799086496,
"max_s": 0.07154643395915627
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.6160109428068e-06,
"p50_s": 5.664012860506773e-06,
"p90_s": 5.715189035981893e-06,
"p95_s": 5.721586057916284e-06,
"p99_s": 5.726703675463796e-06,
"min_s": 5.456036888062954e-06,
"max_s": 5.727983079850674e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.07517527565748121,
"p50_s": 0.07530217699240893,
"p90_s": 0.07546080739703029,
"p95_s": 0.07548063619760798,
"p99_s": 0.0754964992380701,
"min_s": 0.07472318498184904,
"max_s": 0.07550046499818563
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.025796906343506027,
"p50_s": 0.025601482018828392,
"p90_s": 0.026077552419155835,
"p95_s": 0.026137061219196768,
"p99_s": 0.02618466825922951,
"min_s": 0.025592666992451996,
"max_s": 0.026196570019237697
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.560026961068313e-06,
"p50_s": 4.56004636362195e-06,
"p90_s": 4.726427141577005e-06,
"p95_s": 4.747224738821387e-06,
"p99_s": 4.763862816616893e-06,
"min_s": 4.352012183517218e-06,
"max_s": 4.768022336065769e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.975989961375793e-06,
"p50_s": 9.119976311922073e-06,
"p90_s": 9.299209341406823e-06,
"p95_s": 9.321613470092414e-06,
"p99_s": 9.33953677304089e-06,
"min_s": 8.463975973427296e-06,
"max_s": 9.34401759877801e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.5733294983704886e-06,
"p50_s": 3.5520060919225216e-06,
"p90_s": 3.6544050090014937e-06,
"p95_s": 3.667204873636365e-06,
"p99_s": 3.677444765344262e-06,
"min_s": 3.4879776649177074e-06,
"max_s": 3.6800047382712364e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.00451310898642987,
"p50_s": 0.004505342978518456,
"p90_s": 0.004527206195052713,
"p95_s": 0.004529939097119496,
"p99_s": 0.004532125418772921,
"min_s": 0.004501311981584877,
"max_s": 0.004532671999186277
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 9.279989171773195e-06,
"p50_s": 9.535986464470625e-06,
"p90_s": 9.753589984029531e-06,
"p95_s": 9.780790423974394e-06,
"p99_s": 9.802550775930286e-06,
"min_s": 8.495990186929703e-06,
"max_s": 9.807990863919258e-06
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.005867129017133266,
"p50_s": 0.005761470005381852,
"p90_s": 0.0060237164259888225,
"p95_s": 0.006056497228564694,
"p99_s": 0.006082721870625391,
"min_s": 0.005750639014877379,
"max_s": 0.006089278031140566
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.517337169498205e-06,
"p50_s": 6.52798917144537e-06,
"p90_s": 6.681610830128194e-06,
"p95_s": 6.700813537463546e-06,
"p99_s": 6.716175703331828e-06,
"min_s": 6.304006092250347e-06,
"max_s": 6.720016244798899e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.0607306759726877,
"p50_s": 0.058016403985675424,
"p90_s": 0.06506514557404444,
"p95_s": 0.06594623827259057,
"p99_s": 0.06665111243142748,
"min_s": 0.05734829296125099,
"max_s": 0.06682733097113669
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.6586771582563715e-06,
"p50_s": 5.567970219999552e-06,
"p90_s": 5.849602166563272e-06,
"p95_s": 5.884806159883737e-06,
"p99_s": 5.912969354540109e-06,
"min_s": 5.488051101565361e-06,
"max_s": 5.920010153204203e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.07119058868071686,
"p50_s": 0.06835102604236454,
"p90_s": 0.07540559160988779,
"p95_s": 0.07628741230582818,
"p99_s": 0.07699286886258051,
"min_s": 0.06805150699801743,
"max_s": 0.07716923300176859
},
"gate_mlp": {
"count": 3,
"mean_s": 0.013778018668138733,
"p50_s": 0.010043838003184646,
"p90_s": 0.02003887001192197,
"p95_s": 0.021288249013014137,
"p99_s": 0.02228775221388787,
"min_s": 0.008752589987125248,
"max_s": 0.022537628014106303
},
"block_total": {
"count": 3,
"mean_s": 0.5541369766773035,
"p50_s": 0.5570170399732888,
"p90_s": 0.5594101664144546,
"p95_s": 0.5597093072196002,
"p99_s": 0.5599486198637169,
"min_s": 0.5453854420338757,
"max_s": 0.560008448024746
}
},
"module_forward": {
"count": 3,
"mean_s": 0.5096157593264555,
"p50_s": 0.5090722020249814,
"p90_s": 0.5104809443932027,
"p95_s": 0.5106570371892303,
"p99_s": 0.5107979114260524,
"min_s": 0.5089419459691271,
"max_s": 0.510833129985258
},
"module_forward_checksum": 296883328.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 9093546496
},
"profiler_top_events": [
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1367.4550000000002,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -303104,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1346.1589999999997,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1125.328,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
},
{
"key": "aten::zeros",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 666.7999999999993,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::zero_",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 591.2480000000005,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37888, 336]]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 588.7200000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [21504, 5376], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 445.0400000000004,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]"
},
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 305.08799999999974,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 261.40800000000036,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 210.1280000000006,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "aten::to",
"count": 12,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 192.3839999999982,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 184.64000000000033,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 173.08799999999974,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 154.47999999999956,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 136.70399999999972,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "aten::empty",
"count": 34,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 132.73599999999925,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 8009298432,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 129.66399999999976,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 109.07200000000012,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 99.28000000000065,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "aten::silu",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 99.15200000000004,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 1084227584,
"input_shapes": "[[37810, 14336]]"
}
]
}

27
benchmarks/gb10-cute-qkv-block24-capacity3072-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"iterations": 8,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -895169280.0,
"candidate_checksum": -895169280.0,
"baseline": {
"mean_s": 0.46958410650404403,
"p50_s": 0.4692447630222887,
"min_s": 0.46773986698826775,
"max_s": 0.47218111396068707
},
"ring": {
"mean_s": 0.4762414453798556,
"p50_s": 0.4739541109884158,
"min_s": 0.47236386401345953,
"max_s": 0.4925510979956016
},
"p50_improvement_percent": -1.0036016035203765
}

27
benchmarks/gb10-cute-qkv-block24-capacity4096-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"iterations": 6,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -168955712.0,
"candidate_checksum": -168955712.0,
"baseline": {
"mean_s": 0.4668582236530104,
"p50_s": 0.46715522548765875,
"min_s": 0.4650274849846028,
"max_s": 0.4681780419778079
},
"ring": {
"mean_s": 0.46820701367687434,
"p50_s": 0.46829428849741817,
"min_s": 0.4667344400077127,
"max_s": 0.46996626403415576
},
"p50_improvement_percent": -0.2438296625217884
}

27
benchmarks/gb10-cute-qkv-block24-capacity8192-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"iterations": 6,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -168955712.0,
"candidate_checksum": -168955712.0,
"baseline": {
"mean_s": 0.46774995284310233,
"p50_s": 0.4674104950099718,
"min_s": 0.4658880060305819,
"max_s": 0.47020024503581226
},
"ring": {
"mean_s": 0.49012390718174476,
"p50_s": 0.4898388920119032,
"min_s": 0.48933237104211,
"max_s": 0.4917365289875306
},
"p50_improvement_percent": -4.798436757705438
}

27
benchmarks/gb10-cute-qkv-block24-full-workspace-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"iterations": 6,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -168955712.0,
"candidate_checksum": -168955712.0,
"baseline": {
"mean_s": 0.46479812582644325,
"p50_s": 0.4648084765067324,
"min_s": 0.46373577596386895,
"max_s": 0.4655176450032741
},
"ring": {
"mean_s": 0.5324759804934729,
"p50_s": 0.5321614120039158,
"min_s": 0.5311248479993083,
"max_s": 0.5355711780139245
},
"p50_improvement_percent": -14.49047057045394
}

852
benchmarks/gb10-cute-qkv-block24-ring.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 24,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.003741738328244537,
"p50_s": 0.003866238985210657,
"p90_s": 0.003901209402829409,
"p95_s": 0.0039055807050317527,
"p99_s": 0.003909077746793627,
"min_s": 0.0034490239922888577,
"max_s": 0.0039099520072340965
},
"modulate_msa": {
"count": 3,
"mean_s": 0.011052168323658407,
"p50_s": 0.010806893988046795,
"p90_s": 0.012067155598197134,
"p95_s": 0.012224688299465925,
"p99_s": 0.012350714460480958,
"min_s": 0.009967389982193708,
"max_s": 0.012382221000734717
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.536014214158058e-06,
"p50_s": 4.143978003412485e-06,
"p90_s": 7.62563431635499e-06,
"p95_s": 8.060841355472804e-06,
"p99_s": 8.409006986767054e-06,
"min_s": 3.968016244471073e-06,
"max_s": 8.496048394590616e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.4229328371584415e-05,
"p50_s": 8.367991540580988e-06,
"p90_s": 2.274239668622613e-05,
"p95_s": 2.4539197329431773e-05,
"p99_s": 2.597663784399629e-05,
"min_s": 7.983995601534843e-06,
"max_s": 2.6335997972637415e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.509320473919312e-06,
"p50_s": 3.4240074455738068e-06,
"p90_s": 3.6927871406078338e-06,
"p95_s": 3.7263846024870874e-06,
"p99_s": 3.7532625719904902e-06,
"min_s": 3.343971911817789e-06,
"max_s": 3.7599820643663406e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.0017316586648424466,
"p50_s": 0.0017092640046030283,
"p90_s": 0.0017795104067772627,
"p95_s": 0.001788291207049042,
"p99_s": 0.0017953158472664653,
"min_s": 0.0016886399826034904,
"max_s": 0.0017970720073208213
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 1.3424005980292955e-05,
"p50_s": 1.1440017260611057e-05,
"p90_s": 1.796798314899206e-05,
"p95_s": 1.8783978885039683e-05,
"p99_s": 1.9436775473877788e-05,
"min_s": 9.232026059180498e-06,
"max_s": 1.9599974621087313e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002303071009616057,
"p50_s": 0.0022418550215661526,
"p90_s": 0.002481432631611824,
"p95_s": 0.002511379832867533,
"p99_s": 0.0025353375938721003,
"min_s": 0.0021260309731587768,
"max_s": 0.002541327034123242
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.789341568946838e-06,
"p50_s": 6.608024705201387e-06,
"p90_s": 7.401604671031237e-06,
"p95_s": 7.500802166759968e-06,
"p99_s": 7.580160163342953e-06,
"min_s": 6.160000339150429e-06,
"max_s": 7.599999662488699e-06
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.029993231679933768,
"p50_s": 0.026411275030113757,
"p90_s": 0.03558764460030943,
"p95_s": 0.03673469079658389,
"p99_s": 0.03765232775360346,
"min_s": 0.025686683016829193,
"max_s": 0.03788173699285835
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 6.229345065852006e-06,
"p50_s": 5.4559786804020405e-06,
"p90_s": 7.32481712475419e-06,
"p95_s": 7.558421930298209e-06,
"p99_s": 7.745305774733424e-06,
"min_s": 5.44002978131175e-06,
"max_s": 7.792026735842228e-06
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.03412259600978965,
"p50_s": 0.030302010010927916,
"p90_s": 0.039898885227739815,
"p95_s": 0.041098494629841296,
"p99_s": 0.04205818215152249,
"min_s": 0.029767673986498266,
"max_s": 0.042298104031942785
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 1.5023998760928711e-05,
"p50_s": 1.2496020644903183e-05,
"p90_s": 1.977919600903988e-05,
"p95_s": 2.0689592929556962e-05,
"p99_s": 2.1417910465970633e-05,
"min_s": 1.0975985787808895e-05,
"max_s": 2.1599989850074053e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012241016685341796,
"p50_s": 0.01193398202303797,
"p90_s": 0.012705834838561714,
"p95_s": 0.012802316440502183,
"p99_s": 0.012879501722054557,
"min_s": 0.011890269990544766,
"max_s": 0.01289879804244265
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.014036514330655336,
"p50_s": 0.0053002709755674005,
"p90_s": 0.026539997407235207,
"p95_s": 0.029194963211193678,
"p99_s": 0.031318935854360457,
"min_s": 0.004959343001246452,
"max_s": 0.031849929015152156
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.007838361353302995,
"p50_s": 0.005219135025981814,
"p90_s": 0.011728727829176933,
"p95_s": 0.012542426929576323,
"p99_s": 0.013193386209895835,
"min_s": 0.00493982300395146,
"max_s": 0.013356126029975712
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.01099991767356793,
"p50_s": 0.005040751013439149,
"p90_s": 0.019409106194507333,
"p95_s": 0.021205150592140853,
"p99_s": 0.02264198611024767,
"min_s": 0.004957807017490268,
"max_s": 0.023001194989774376
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2562586222969306,
"p50_s": 0.25644913397263736,
"p90_s": 0.2567798731615767,
"p95_s": 0.2568212155601941,
"p99_s": 0.25685428947908806,
"min_s": 0.2554641749593429,
"max_s": 0.2568625579588115
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.00469084734019513,
"p50_s": 0.004691840033046901,
"p90_s": 0.004693656810559332,
"p95_s": 0.004693883907748386,
"p99_s": 0.0046940655854996296,
"min_s": 0.0046865909826010466,
"max_s": 0.00469411100493744
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 4.229329836865266e-06,
"p50_s": 4.1120219975709915e-06,
"p90_s": 4.470394924283028e-06,
"p95_s": 4.515191540122033e-06,
"p99_s": 4.551028832793236e-06,
"min_s": 4.01597935706377e-06,
"max_s": 4.559988155961037e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 9.45067343612512e-06,
"p50_s": 9.440002031624317e-06,
"p90_s": 1.010557170957327e-05,
"p95_s": 1.0188767919316889e-05,
"p99_s": 1.0255324887111782e-05,
"min_s": 8.640054147690535e-06,
"max_s": 1.0271964129060507e-05
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.5146561761697135e-06,
"p50_s": 3.4240074455738068e-06,
"p90_s": 3.667175769805908e-06,
"p95_s": 3.697571810334921e-06,
"p99_s": 3.7218886427581313e-06,
"min_s": 3.3919932320713997e-06,
"max_s": 3.7279678508639336e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.002265583665575832,
"p50_s": 0.0022627509897574782,
"p90_s": 0.002275923010893166,
"p95_s": 0.002277569513535127,
"p99_s": 0.0022788867156486956,
"min_s": 0.00225478399079293,
"max_s": 0.002279216016177088
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 1.0005314834415913e-05,
"p50_s": 9.807990863919258e-06,
"p90_s": 1.0319985449314117e-05,
"p95_s": 1.0383984772488475e-05,
"p99_s": 1.0435184231027961e-05,
"min_s": 9.759969543665648e-06,
"max_s": 1.0447984095662832e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002860164638453474,
"p50_s": 0.0028614879702217877,
"p90_s": 0.002871010371018201,
"p95_s": 0.002872200671117753,
"p99_s": 0.0028731529111973944,
"min_s": 0.0028456149739213288,
"max_s": 0.0028733909712173045
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.378686521202326e-06,
"p50_s": 6.432004738599062e-06,
"p90_s": 6.508838851004839e-06,
"p95_s": 6.518443115055561e-06,
"p99_s": 6.526126526296139e-06,
"min_s": 6.176007445901632e-06,
"max_s": 6.528047379106283e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.007843875675462186,
"p50_s": 0.007834365998860449,
"p90_s": 0.007871806004550308,
"p95_s": 0.00787648600526154,
"p99_s": 0.007880230005830527,
"min_s": 0.007816095021553338,
"max_s": 0.007881166005972773
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.5093357029060526e-06,
"p50_s": 5.552021320909262e-06,
"p90_s": 5.6416145525872706e-06,
"p95_s": 5.652813706547021e-06,
"p99_s": 5.661773029714823e-06,
"min_s": 5.311972927302122e-06,
"max_s": 5.664012860506773e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.013048968006235858,
"p50_s": 0.013041116995736957,
"p90_s": 0.013074909010902048,
"p95_s": 0.013079133012797683,
"p99_s": 0.013082512214314192,
"min_s": 0.013022430008277297,
"max_s": 0.01308335701469332
},
"gate_msa": {
"count": 3,
"mean_s": 0.008436798331482956,
"p50_s": 0.008493119035847485,
"p90_s": 0.008522289409302174,
"p95_s": 0.008525935705984011,
"p99_s": 0.00852885274332948,
"min_s": 0.008287693955935538,
"max_s": 0.008529582002665848
},
"norm2": {
"count": 3,
"mean_s": 0.0034889166515010097,
"p50_s": 0.0034742390271276236,
"p90_s": 0.0035121661610901357,
"p95_s": 0.0035169070528354492,
"p99_s": 0.0035206997662317004,
"min_s": 0.0034708629827946424,
"max_s": 0.0035216479445807636
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010772093994698176,
"p50_s": 0.010788750019855797,
"p90_s": 0.010858983569778501,
"p95_s": 0.01086776276351884,
"p99_s": 0.01087478611851111,
"min_s": 0.010650990006979555,
"max_s": 0.010876541957259178
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.058626169959704e-06,
"p50_s": 3.951950930058956e-06,
"p90_s": 4.271953366696835e-06,
"p95_s": 4.3119536712765695e-06,
"p99_s": 4.343953914940357e-06,
"min_s": 3.871973603963852e-06,
"max_s": 4.351953975856304e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.293330514182648e-06,
"p50_s": 8.01600981503725e-06,
"p90_s": 8.860812522470951e-06,
"p95_s": 8.966412860900165e-06,
"p99_s": 9.050893131643534e-06,
"min_s": 7.791968528181314e-06,
"max_s": 9.072013199329376e-06
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.4186717433234057e-06,
"p50_s": 3.456021659076214e-06,
"p90_s": 3.4816330298781395e-06,
"p95_s": 3.48483445122838e-06,
"p99_s": 3.4873955883085728e-06,
"min_s": 3.311957698315382e-06,
"max_s": 3.488035872578621e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017874666567270954,
"p50_s": 0.0017595840035937726,
"p90_s": 0.0018278975854627787,
"p95_s": 0.0018364367831964047,
"p99_s": 0.0018432681413833051,
"min_s": 0.0017578399856574833,
"max_s": 0.0018449759809300303
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 9.296015681078037e-06,
"p50_s": 9.328010492026806e-06,
"p90_s": 9.340816177427769e-06,
"p95_s": 9.342416888102889e-06,
"p99_s": 9.343697456642985e-06,
"min_s": 9.216018952429295e-06,
"max_s": 9.34401759877801e-06
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.002177828651232024,
"p50_s": 0.002151614986360073,
"p90_s": 0.0022210165858268737,
"p95_s": 0.002229691785760224,
"p99_s": 0.002236631945706904,
"min_s": 0.002143503981642425,
"max_s": 0.002238366985693574
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.623993006845315e-06,
"p50_s": 6.607966497540474e-06,
"p90_s": 6.659189239144325e-06,
"p95_s": 6.665592081844807e-06,
"p99_s": 6.670714356005192e-06,
"min_s": 6.592017598450184e-06,
"max_s": 6.671994924545288e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.03496598266065121,
"p50_s": 0.03502911300165579,
"p90_s": 0.03507212101249024,
"p95_s": 0.03507749701384455,
"p99_s": 0.035081797814928,
"min_s": 0.03478596196509898,
"max_s": 0.03508287301519886
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.49866429840525e-06,
"p50_s": 5.4559786804020405e-06,
"p90_s": 5.622406024485826e-06,
"p95_s": 5.643209442496299e-06,
"p99_s": 5.659852176904679e-06,
"min_s": 5.376001354306936e-06,
"max_s": 5.664012860506773e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.03901222701339672,
"p50_s": 0.03901776799466461,
"p90_s": 0.039129883213900034,
"p95_s": 0.03914389761630446,
"p99_s": 0.039155109138228,
"min_s": 0.038861001026816666,
"max_s": 0.039157912018708885
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.02408325932143877,
"p50_s": 0.0241031949990429,
"p90_s": 0.024252827803138644,
"p95_s": 0.02427153190365061,
"p99_s": 0.024286495184060185,
"min_s": 0.02385634696111083,
"max_s": 0.02429023600416258
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.234665539115667e-06,
"p50_s": 4.01597935706377e-06,
"p90_s": 4.617613740265369e-06,
"p95_s": 4.692818038165569e-06,
"p99_s": 4.7529814764857296e-06,
"min_s": 3.9199949242174625e-06,
"max_s": 4.768022336065769e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 7.589328257987897e-06,
"p50_s": 7.71199120208621e-06,
"p90_s": 7.929594721645116e-06,
"p95_s": 7.95679516158998e-06,
"p99_s": 7.978555513545871e-06,
"min_s": 7.071997970342636e-06,
"max_s": 7.983995601534843e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.722670953720808e-06,
"p50_s": 3.503984771668911e-06,
"p90_s": 4.195212386548519e-06,
"p95_s": 4.28161583840847e-06,
"p99_s": 4.3507385998964315e-06,
"min_s": 3.296008799225092e-06,
"max_s": 4.368019290268421e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.004541535027480374,
"p50_s": 0.004574159043841064,
"p90_s": 0.004587253439240158,
"p95_s": 0.004588890238665044,
"p99_s": 0.004590199678204954,
"min_s": 0.004459919000510126,
"max_s": 0.004590527038089931
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 8.880005528529486e-06,
"p50_s": 8.608039934188128e-06,
"p90_s": 9.3247857876122e-06,
"p95_s": 9.414379019290208e-06,
"p99_s": 9.486053604632616e-06,
"min_s": 8.52800440043211e-06,
"max_s": 9.503972250968218e-06
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.005774937667107831,
"p50_s": 0.005734590988140553,
"p90_s": 0.0058928117970936,
"p95_s": 0.005912589398212731,
"p99_s": 0.005928411479108036,
"min_s": 0.005657855013851076,
"max_s": 0.005932366999331862
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.197350254903237e-06,
"p50_s": 6.128044333308935e-06,
"p90_s": 6.396824028342963e-06,
"p95_s": 6.430421490222216e-06,
"p99_s": 6.457299459725618e-06,
"min_s": 5.999987479299307e-06,
"max_s": 6.464018952101469e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.05555110399533684,
"p50_s": 0.05553032597526908,
"p90_s": 0.05586563399992883,
"p95_s": 0.0559075475030113,
"p99_s": 0.055941078305477274,
"min_s": 0.05517352500464767,
"max_s": 0.05594946100609377
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.4613337852060795e-06,
"p50_s": 5.4879928939044476e-06,
"p90_s": 5.6543736718595025e-06,
"p95_s": 5.6751712691038845e-06,
"p99_s": 5.691809346899391e-06,
"min_s": 5.200039595365524e-06,
"max_s": 5.695968866348267e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.06594285130267963,
"p50_s": 0.06600016396259889,
"p90_s": 0.06628018877236172,
"p95_s": 0.06631519187358208,
"p99_s": 0.06634319435455836,
"min_s": 0.06547819497063756,
"max_s": 0.06635019497480243
},
"gate_mlp": {
"count": 3,
"mean_s": 0.013789538682127992,
"p50_s": 0.00966468604747206,
"p90_s": 0.019895839621312916,
"p95_s": 0.02117473381804302,
"p99_s": 0.022197849175427106,
"min_s": 0.009250301984138787,
"max_s": 0.02245362801477313
},
"block_total": {
"count": 3,
"mean_s": 0.5336582703360667,
"p50_s": 0.5072318860329688,
"p90_s": 0.5710783508140594,
"p95_s": 0.5790591589116957,
"p99_s": 0.5854438053898048,
"min_s": 0.5067029579658993,
"max_s": 0.587039967009332
}
},
"module_forward": {
"count": 3,
"mean_s": 0.47701150766806677,
"p50_s": 0.4728726129978895,
"p90_s": 0.48509467700496317,
"p95_s": 0.48662243500584734,
"p99_s": 0.48784464140655476,
"min_s": 0.4700117169995792,
"max_s": 0.48815019300673157
},
"module_forward_checksum": 296883328.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 7351755264
},
"profiler_top_events": [
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 581.328,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 512.4480000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 422.6239999999998,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 247.44000000000028,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "aten::to",
"count": 13,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 218.31999999999857,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 209.37599999999975,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [], [], [], []]"
},
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 187.68000000000043,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -303104,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 175.6159999999994,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 164.1119999999994,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 139.95200000000023,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 123.92000000000007,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "aten::empty",
"count": 29,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 123.24799999999709,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 6267507200,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 121.84000000000015,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "aten::zeros",
"count": 3,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 106.24000000000069,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 100.2559999999994,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 95.00799999999981,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 88.44800000000032,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "aten::mul",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 76.6240000000007,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 2048,
"input_shapes": "[[], []]"
},
{
"key": "aten::silu",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 64.27199999999993,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 1084227584,
"input_shapes": "[[37810, 14336]]"
},
{
"key": "aten::copy_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 63.455999999999904,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1], [], []]"
}
]
}

27
benchmarks/gb10-cute-qkv-block49-alternating.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"block_index": 49,
"hidden_shape": [
37810,
5376
],
"iterations": 6,
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"reference_checksum": -1799476346880.0,
"candidate_checksum": -1799476346880.0,
"baseline": {
"mean_s": 0.4619793013262097,
"p50_s": 0.4623519679880701,
"min_s": 0.46040739299496636,
"max_s": 0.46288129599997774
},
"ring": {
"mean_s": 0.46425590649596415,
"p50_s": 0.46476577199064195,
"min_s": 0.4613973450032063,
"max_s": 0.46552810998400673
},
"p50_improvement_percent": -0.5220706668721542
}

852
benchmarks/gb10-cute-qkv-block49-baseline.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 49,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.0036217596692343554,
"p50_s": 0.00359812198439613,
"p90_s": 0.0037101987865753474,
"p95_s": 0.0037242083868477493,
"p99_s": 0.0037354160670656712,
"min_s": 0.0035289390361867845,
"max_s": 0.0037382179871201515
},
"modulate_msa": {
"count": 3,
"mean_s": 0.01067996733278657,
"p50_s": 0.010696462995838374,
"p90_s": 0.011010011786129327,
"p95_s": 0.011049205384915696,
"p99_s": 0.011080560263944791,
"min_s": 0.010255040018819273,
"max_s": 0.011088398983702064
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.525342809657256e-06,
"p50_s": 4.208006430417299e-06,
"p90_s": 7.382419425994158e-06,
"p95_s": 7.779221050441265e-06,
"p99_s": 8.096662349998951e-06,
"min_s": 4.191999323666096e-06,
"max_s": 8.176022674888372e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.5418666104475658e-05,
"p50_s": 8.544011507183313e-06,
"p90_s": 2.5260786060243845e-05,
"p95_s": 2.7350382879376408e-05,
"p99_s": 2.9022060334682465e-05,
"min_s": 8.27200710773468e-06,
"max_s": 2.9439979698508978e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.674688438574473e-06,
"p50_s": 3.6479905247688293e-06,
"p90_s": 3.827223554253578e-06,
"p95_s": 3.849627682939172e-06,
"p99_s": 3.867550985887647e-06,
"min_s": 3.5040429793298244e-06,
"max_s": 3.872031811624765e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.0017329036879042785,
"p50_s": 0.0017396770417690277,
"p90_s": 0.0017431330401450396,
"p95_s": 0.001743565039942041,
"p99_s": 0.0017439106397796422,
"min_s": 0.001715036982204765,
"max_s": 0.0017439970397390425
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 1.3610658546288809e-05,
"p50_s": 1.2303993571549654e-05,
"p90_s": 1.84479751624167e-05,
"p95_s": 1.9215972861275078e-05,
"p99_s": 1.983037102036178e-05,
"min_s": 8.544011507183313e-06,
"max_s": 1.9983970560133457e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.0022982683343191943,
"p50_s": 0.0023595320526510477,
"p90_s": 0.0024083896074444055,
"p95_s": 0.0024144968017935755,
"p99_s": 0.002419382557272911,
"min_s": 0.0021146689541637897,
"max_s": 0.002420603996142745
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 7.045338861644268e-06,
"p50_s": 6.511982064694166e-06,
"p90_s": 7.868779357522726e-06,
"p95_s": 8.038379019126295e-06,
"p99_s": 8.174058748409153e-06,
"min_s": 6.416055839508772e-06,
"max_s": 8.207978680729866e-06
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.02569509532380228,
"p50_s": 0.02566098398528993,
"p90_s": 0.026279260776937007,
"p95_s": 0.026356545375892895,
"p99_s": 0.0264183730550576,
"min_s": 0.02499047201126814,
"max_s": 0.026433829974848777
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 6.298670389999946e-06,
"p50_s": 5.504000000655651e-06,
"p90_s": 7.46243167668581e-06,
"p95_s": 7.70723563618958e-06,
"p99_s": 7.903078803792597e-06,
"min_s": 5.439971573650837e-06,
"max_s": 7.95203959569335e-06
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.029824565979652107,
"p50_s": 0.02990254497854039,
"p90_s": 0.030253315379377456,
"p95_s": 0.030297161679482085,
"p99_s": 0.030332238719565793,
"min_s": 0.02923014498082921,
"max_s": 0.03034100797958672
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 1.4346345172574123e-05,
"p50_s": 1.1824013199657202e-05,
"p90_s": 1.82872056029737e-05,
"p95_s": 1.909510465338826e-05,
"p99_s": 1.9741423893719913e-05,
"min_s": 1.1312018614262342e-05,
"max_s": 1.9903003703802824e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012300620651027808,
"p50_s": 0.012445547967217863,
"p90_s": 0.012557216011919082,
"p95_s": 0.012571174517506735,
"p99_s": 0.012582341321976855,
"min_s": 0.011871180962771177,
"max_s": 0.012585133023094386
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.005062365322373807,
"p50_s": 0.005007288011256605,
"p90_s": 0.005225092789623886,
"p95_s": 0.005252318386919796,
"p99_s": 0.005274098864756525,
"min_s": 0.00490026397164911,
"max_s": 0.005279543984215707
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.00505693598339955,
"p50_s": 0.00496911199297756,
"p90_s": 0.005203018360771239,
"p95_s": 0.005232256656745448,
"p99_s": 0.005255647293524816,
"min_s": 0.004940201004501432,
"max_s": 0.005261494952719659
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.00494270701892674,
"p50_s": 0.00493679300416261,
"p90_s": 0.004955710633657872,
"p95_s": 0.004958075337344781,
"p99_s": 0.004959967100294307,
"min_s": 0.004930888011585921,
"max_s": 0.0049604400410316885
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2531811236597908,
"p50_s": 0.2535298540024087,
"p90_s": 0.25376300599891694,
"p95_s": 0.25379214999848043,
"p99_s": 0.2538154651981313,
"min_s": 0.25219222297891974,
"max_s": 0.253821293998044
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.0048227393223593635,
"p50_s": 0.004690104979090393,
"p90_s": 0.005012331414036453,
"p95_s": 0.00505260971840471,
"p99_s": 0.005084832361899316,
"min_s": 0.004685224965214729,
"max_s": 0.005092888022772968
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 4.410685505717993e-06,
"p50_s": 4.272034857422113e-06,
"p90_s": 4.822446499019861e-06,
"p95_s": 4.89124795421958e-06,
"p99_s": 4.946289118379355e-06,
"min_s": 3.999972250312567e-06,
"max_s": 4.960049409419298e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 9.290330732862154e-06,
"p50_s": 9.215029422193766e-06,
"p90_s": 9.74057475104928e-06,
"p95_s": 9.806267917156219e-06,
"p99_s": 9.858822450041771e-06,
"min_s": 8.78400169312954e-06,
"max_s": 9.871961083263159e-06
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.445330852021774e-06,
"p50_s": 3.4240074455738068e-06,
"p90_s": 3.5136006772518156e-06,
"p95_s": 3.5247998312115673e-06,
"p99_s": 3.533759154379368e-06,
"min_s": 3.375986125320196e-06,
"max_s": 3.535998985171318e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.0022702256489234665,
"p50_s": 0.0022497729514725506,
"p90_s": 0.0023021497880108656,
"p95_s": 0.0023086968925781545,
"p99_s": 0.0023139345762319863,
"min_s": 0.002245659998152405,
"max_s": 0.002315243997145444
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 9.96800372377038e-06,
"p50_s": 9.695999324321747e-06,
"p90_s": 1.0348809882998465e-05,
"p95_s": 1.0430411202833055e-05,
"p99_s": 1.0495692258700728e-05,
"min_s": 9.695999324321747e-06,
"max_s": 1.0512012522667646e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002868635676956425,
"p50_s": 0.0028517399914562702,
"p90_s": 0.002900878433138132,
"p95_s": 0.002907020738348365,
"p99_s": 0.0029119345825165508,
"min_s": 0.0028410039958544075,
"max_s": 0.0029131630435585976
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.453308742493391e-06,
"p50_s": 6.511982064694166e-06,
"p90_s": 6.588769610971213e-06,
"p95_s": 6.5983680542558435e-06,
"p99_s": 6.606046808883548e-06,
"min_s": 6.239977665245533e-06,
"max_s": 6.607966497540474e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.008090866671409458,
"p50_s": 0.007834755000658333,
"p90_s": 0.008461467805318534,
"p95_s": 0.00853980690590106,
"p99_s": 0.008602478186367078,
"min_s": 0.007819699007086456,
"max_s": 0.008618146006483585
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.562673322856426e-06,
"p50_s": 5.4879928939044476e-06,
"p90_s": 5.795189645141363e-06,
"p95_s": 5.833589239045977e-06,
"p99_s": 5.864308914169669e-06,
"min_s": 5.328038241714239e-06,
"max_s": 5.871988832950592e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.013308762339875102,
"p50_s": 0.013069211039692163,
"p90_s": 0.013700198195874692,
"p95_s": 0.013779071590397507,
"p99_s": 0.013842170306015759,
"min_s": 0.01299913099501282,
"max_s": 0.013857944984920323
},
"gate_msa": {
"count": 3,
"mean_s": 0.008683037323256334,
"p50_s": 0.008554771018680185,
"p90_s": 0.008906488574575632,
"p95_s": 0.008950453269062564,
"p99_s": 0.008985625024652109,
"min_s": 0.008499922987539321,
"max_s": 0.008994417963549495
},
"norm2": {
"count": 3,
"mean_s": 0.003726057999301702,
"p50_s": 0.003699098015204072,
"p90_s": 0.0038113028276711702,
"p95_s": 0.0038253284292295575,
"p99_s": 0.0038365489104762673,
"min_s": 0.0036397219519130886,
"max_s": 0.003839354030787945
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010754756338428706,
"p50_s": 0.010686943016480654,
"p90_s": 0.010987435781862587,
"p95_s": 0.011024997377535328,
"p99_s": 0.011055046654073522,
"min_s": 0.010514767025597394,
"max_s": 0.01106255897320807
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.288003159066041e-06,
"p50_s": 4.031986463814974e-06,
"p90_s": 4.684797022491694e-06,
"p95_s": 4.766398342326284e-06,
"p99_s": 4.831679398193956e-06,
"min_s": 3.984023351222277e-06,
"max_s": 4.847999662160873e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.207998083283504e-06,
"p50_s": 7.983995601534843e-06,
"p90_s": 8.828798308968544e-06,
"p95_s": 8.934398647397758e-06,
"p99_s": 9.018878918141127e-06,
"min_s": 7.599999662488699e-06,
"max_s": 9.039998985826969e-06
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.503984771668911e-06,
"p50_s": 3.4879776649177074e-06,
"p90_s": 3.5775708965957167e-06,
"p95_s": 3.5887700505554675e-06,
"p99_s": 3.5977293737232683e-06,
"min_s": 3.4240074455738068e-06,
"max_s": 3.5999692045152187e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017449889952937763,
"p50_s": 0.0017389570130035281,
"p90_s": 0.0017614722019061446,
"p95_s": 0.0017642866005189717,
"p99_s": 0.0017665381194092334,
"min_s": 0.0017289089737460017,
"max_s": 0.0017671009991317987
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 1.4506649070729813e-05,
"p50_s": 9.312003385275602e-06,
"p90_s": 2.1907163318246604e-05,
"p95_s": 2.3481558309867977e-05,
"p99_s": 2.4741074303165077e-05,
"min_s": 9.15199052542448e-06,
"max_s": 2.5055953301489353e-05
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.002230454992968589,
"p50_s": 0.002145772974472493,
"p90_s": 0.002364140178542584,
"p95_s": 0.0023914360790513456,
"p99_s": 0.0024132727994583547,
"min_s": 0.0021268600248731673,
"max_s": 0.002418731979560107
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.261320474247138e-06,
"p50_s": 6.27199187874794e-06,
"p90_s": 6.425566971302032e-06,
"p95_s": 6.444763857871294e-06,
"p99_s": 6.460121367126704e-06,
"min_s": 6.0480087995529175e-06,
"max_s": 6.4639607444405556e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.03514027533431848,
"p50_s": 0.03508026502095163,
"p90_s": 0.03532921141013503,
"p95_s": 0.03536032970878296,
"p99_s": 0.03538522434770129,
"min_s": 0.03494911297457293,
"max_s": 0.03539144800743088
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.6320180495580035e-06,
"p50_s": 5.631998647004366e-06,
"p90_s": 5.657610017806292e-06,
"p95_s": 5.660811439156532e-06,
"p99_s": 5.663372576236725e-06,
"min_s": 5.600042641162872e-06,
"max_s": 5.664012860506773e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.039202652658180646,
"p50_s": 0.03935540997190401,
"p90_s": 0.039365841995459054,
"p95_s": 0.039367145998403436,
"p99_s": 0.039368189200758935,
"min_s": 0.03888409800129011,
"max_s": 0.03936845000134781
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.024196713328516733,
"p50_s": 0.024238202022388577,
"p90_s": 0.024260946782305837,
"p95_s": 0.024263789877295494,
"p99_s": 0.02426606435328722,
"min_s": 0.024085304990876466,
"max_s": 0.02426663297228515
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.272015454868476e-06,
"p50_s": 4.384026397019625e-06,
"p90_s": 4.435202572494745e-06,
"p95_s": 4.441599594429136e-06,
"p99_s": 4.446717211976647e-06,
"min_s": 3.984023351222277e-06,
"max_s": 4.447996616363525e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.181338974585136e-06,
"p50_s": 8.447968866676092e-06,
"p90_s": 8.460821118205785e-06,
"p95_s": 8.462427649646997e-06,
"p99_s": 8.463712874799967e-06,
"min_s": 7.632013875991106e-06,
"max_s": 8.464034181088209e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.4186717433234057e-06,
"p50_s": 3.4240074455738068e-06,
"p90_s": 3.4240074455738068e-06,
"p95_s": 3.4240074455738068e-06,
"p99_s": 3.4240074455738068e-06,
"min_s": 3.4080003388226032e-06,
"max_s": 3.4240074455738068e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.004601721011567861,
"p50_s": 0.0045881689875386655,
"p90_s": 0.004628655395936221,
"p95_s": 0.004633716196985916,
"p99_s": 0.004637764837825671,
"min_s": 0.004578217049129307,
"max_s": 0.00463877699803561
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 1.3728005190690359e-05,
"p50_s": 1.2575997970998287e-05,
"p90_s": 1.8028798513114454e-05,
"p95_s": 1.8710398580878974e-05,
"p99_s": 1.925567863509059e-05,
"min_s": 9.216018952429295e-06,
"max_s": 1.9391998648643494e-05
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.005966662662103772,
"p50_s": 0.005981990019790828,
"p90_s": 0.0060327299637719985,
"p95_s": 0.006039072456769645,
"p99_s": 0.006044146451167762,
"min_s": 0.005872583016753197,
"max_s": 0.0060454149497672915
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.144012634952863e-06,
"p50_s": 6.176007445901632e-06,
"p90_s": 6.30401773378253e-06,
"p95_s": 6.320019019767642e-06,
"p99_s": 6.332820048555731e-06,
"min_s": 5.920010153204203e-06,
"max_s": 6.336020305752754e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.05552383201817671,
"p50_s": 0.05537804099731147,
"p90_s": 0.055793770635500554,
"p95_s": 0.055845736840274184,
"p99_s": 0.055887309804093094,
"min_s": 0.05529575201217085,
"max_s": 0.05589770304504782
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.546646813551585e-06,
"p50_s": 5.504000000655651e-06,
"p90_s": 5.619158037006855e-06,
"p95_s": 5.633552791550755e-06,
"p99_s": 5.645068595185876e-06,
"min_s": 5.4879928939044476e-06,
"max_s": 5.647947546094656e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.06617354867436613,
"p50_s": 0.06602008803747594,
"p90_s": 0.06640077200718224,
"p95_s": 0.06644835750339553,
"p99_s": 0.06648642590036616,
"min_s": 0.06600461498601362,
"max_s": 0.06649594299960881
},
"gate_mlp": {
"count": 3,
"mean_s": 0.014910061300421754,
"p50_s": 0.008941200969275087,
"p90_s": 0.023517421761061995,
"p95_s": 0.02533944936003536,
"p99_s": 0.02679707143921405,
"min_s": 0.008627505972981453,
"max_s": 0.027161476959008723
},
"block_total": {
"count": 3,
"mean_s": 0.5105496386337715,
"p50_s": 0.5060367499827407,
"p90_s": 0.5189121219678782,
"p95_s": 0.5205215434660204,
"p99_s": 0.5218090806645341,
"min_s": 0.5034812009544112,
"max_s": 0.5221309649641626
}
},
"module_forward": {
"count": 3,
"mean_s": 0.4635661483431856,
"p50_s": 0.4635968530201353,
"p90_s": 0.4642748417914845,
"p95_s": 0.46435959038790314,
"p99_s": 0.46442738926503807,
"min_s": 0.46265725302509964,
"max_s": 0.4644443389843218
},
"module_forward_checksum": -778077863936.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 9093546496
},
"profiler_top_events": [
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1357.5660000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -303104,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 1349.8219999999997,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 380.12699999999995,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [21504, 5376], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 333.52000000000044,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 282.47900000000027,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [21504, 2688], [], [], [37888, 336], [21504, 336], [], [], []]"
},
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 191.07200000000012,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 165.34399999999982,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 150.27199999999993,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 131.98399999999992,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 127.2159999999999,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "aten::to",
"count": 12,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 112.27199999999971,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 111.93600000000015,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 110.38299999999981,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 109.64799999999968,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "aten::empty",
"count": 34,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 104.27200000000116,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 8009298432,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 95.03900000000067,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "aten::zeros",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 88.20800000000008,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 71.7430000000004,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "aten::to",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 66.44799999999987,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 62.83199999999988,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [], [], [], []]"
}
]
}

852
benchmarks/gb10-cute-qkv-block49-ring.json Normal file
View file

@ -0,0 +1,852 @@
{
"prompt": "A playful orange tabby cat starts in an ordinary cozy living room in a normal house, afternoon light, sofa and rug. The cat crouches, jumps, and does one clean athletic backflip in slow motion. As the backflip completes there is a sharp cinematic cut: the cat lands perfectly on a glowing neon disco dance floor wearing oversized black sunglasses. Mirror ball reflections, colorful lights, joyful party energy, stylish and funny, clear before-and-after transformation.",
"model_path": "/models/minimax_h3_fl2va_pruned_nvfp4.safetensors",
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"block_index": 49,
"attention": "sage2",
"warmup": 2,
"iterations": 3,
"hidden_shape": [
37810,
5376
],
"output_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"timings": {
"norm1": {
"count": 3,
"mean_s": 0.003877114000109335,
"p50_s": 0.003574107016902417,
"p90_s": 0.004309529426973313,
"p95_s": 0.0044014572282321755,
"p99_s": 0.004474999469239265,
"min_s": 0.0035638499539345503,
"max_s": 0.004493385029491037
},
"modulate_msa": {
"count": 3,
"mean_s": 0.011260746008095643,
"p50_s": 0.010931248019915074,
"p90_s": 0.012151918408926576,
"p95_s": 0.012304502207553015,
"p99_s": 0.012426569246454165,
"min_s": 0.0103939039981924,
"max_s": 0.012457086006179452
},
"linear.attn_qkv_proj.flatten_contiguous": {
"count": 3,
"mean_s": 5.386633953700463e-06,
"p50_s": 4.335946869105101e-06,
"p90_s": 7.0239766500890255e-06,
"p95_s": 7.359980372712016e-06,
"p99_s": 7.628783350810409e-06,
"min_s": 4.127970896661282e-06,
"max_s": 7.695984095335007e-06
},
"linear.attn_qkv_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_qkv_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.5333333673576515e-05,
"p50_s": 8.84797191247344e-06,
"p90_s": 2.498881658539176e-05,
"p95_s": 2.700642216950655e-05,
"p99_s": 2.862050663679838e-05,
"min_s": 8.128001354634762e-06,
"max_s": 2.902402775362134e-05
},
"linear.attn_qkv_proj.bias_cast": {
"count": 3,
"mean_s": 3.562658093869686e-06,
"p50_s": 3.4239492379128933e-06,
"p90_s": 3.782415296882391e-06,
"p95_s": 3.827223554253578e-06,
"p99_s": 3.8630701601505284e-06,
"min_s": 3.3919932320713997e-06,
"max_s": 3.872031811624765e-06
},
"linear.attn_qkv_proj.activation_scale": {
"count": 3,
"mean_s": 0.0019726533403930566,
"p50_s": 0.001826349995099008,
"p90_s": 0.002202194812707603,
"p95_s": 0.0022491754149086777,
"p99_s": 0.0022867598966695366,
"min_s": 0.0017954540089704096,
"max_s": 0.0022961560171097517
},
"linear.attn_qkv_proj.scale_to_device": {
"count": 3,
"mean_s": 1.755197687695424e-05,
"p50_s": 1.7903978005051613e-05,
"p90_s": 2.3907190188765527e-05,
"p95_s": 2.4657591711729766e-05,
"p99_s": 2.5257912930101157e-05,
"min_s": 9.343959391117096e-06,
"max_s": 2.5407993234694004e-05
},
"linear.attn_qkv_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002370401343796402,
"p50_s": 0.0021540760062634945,
"p90_s": 0.0026746008079499006,
"p95_s": 0.0027396664081607014,
"p99_s": 0.0027917188883293423,
"min_s": 0.00215239601675421,
"max_s": 0.002804732008371502
},
"linear.attn_qkv_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 7.146678399294615e-06,
"p50_s": 6.608024705201387e-06,
"p90_s": 7.977581117302179e-06,
"p95_s": 8.148775668814778e-06,
"p99_s": 8.285731310024857e-06,
"min_s": 6.51204027235508e-06,
"max_s": 8.319970220327377e-06
},
"linear.attn_qkv_proj.gemm": {
"count": 3,
"mean_s": 0.03147544667202359,
"p50_s": 0.026172921003308147,
"p90_s": 0.039067967410665004,
"p95_s": 0.040679848211584616,
"p99_s": 0.0419693528523203,
"min_s": 0.025961690000258386,
"max_s": 0.04229172901250422
},
"linear.attn_qkv_proj.slice_reshape": {
"count": 3,
"mean_s": 6.479987253745397e-06,
"p50_s": 5.583977326750755e-06,
"p90_s": 7.811188697814942e-06,
"p95_s": 8.089590119197965e-06,
"p99_s": 8.312311256304384e-06,
"min_s": 5.4879928939044476e-06,
"max_s": 8.367991540580988e-06
},
"attn_qkv_proj": {
"count": 3,
"mean_s": 0.03591976532091697,
"p50_s": 0.030206882976926863,
"p90_s": 0.04406598538625986,
"p95_s": 0.04579837318742648,
"p99_s": 0.04718428342835978,
"min_s": 0.030021651997230947,
"max_s": 0.0475307609885931
},
"attn_qkv_split_view": {
"count": 3,
"mean_s": 1.6997335478663445e-05,
"p50_s": 1.4527991879731417e-05,
"p90_s": 2.1683203522115948e-05,
"p95_s": 2.257760497741401e-05,
"p99_s": 2.3293126141652467e-05,
"min_s": 1.2992008123546839e-05,
"max_s": 2.3472006432712078e-05
},
"attn_qk_rms_rope": {
"count": 3,
"mean_s": 0.012158526010656109,
"p50_s": 0.012062589987181127,
"p90_s": 0.012462397222407162,
"p95_s": 0.012512373126810417,
"p99_s": 0.012552353850333021,
"min_s": 0.011850639013573527,
"max_s": 0.012562349031213671
},
"attn_q_transpose_contiguous": {
"count": 3,
"mean_s": 0.007270698668435216,
"p50_s": 0.005053576023783535,
"p90_s": 0.010375015193130822,
"p95_s": 0.01104019508929923,
"p99_s": 0.01157233900623396,
"min_s": 0.0050531449960544705,
"max_s": 0.011705374985467643
},
"attn_k_transpose_contiguous": {
"count": 3,
"mean_s": 0.00565119698876515,
"p50_s": 0.005379768030252308,
"p90_s": 0.006368463987018912,
"p95_s": 0.006492050981614738,
"p99_s": 0.006590920577291399,
"min_s": 0.004958184959832579,
"max_s": 0.006615637976210564
},
"attn_v_transpose_contiguous": {
"count": 3,
"mean_s": 0.005205602676142007,
"p50_s": 0.005224872031249106,
"p90_s": 0.005333428806625307,
"p95_s": 0.005346998403547332,
"p99_s": 0.005357854081084952,
"min_s": 0.005031367996707559,
"max_s": 0.005360568000469357
},
"attention_kernel": {
"count": 3,
"mean_s": 0.2570537893722455,
"p50_s": 0.25454208703013137,
"p90_s": 0.26159199584508314,
"p95_s": 0.26247323444695214,
"p99_s": 0.2631782253284473,
"min_s": 0.25326480803778395,
"max_s": 0.2633544730488211
},
"attn_output_reshape": {
"count": 3,
"mean_s": 0.004732787667308003,
"p50_s": 0.00468229700345546,
"p90_s": 0.004807135392911732,
"p95_s": 0.004822740191593766,
"p99_s": 0.004835224030539393,
"min_s": 0.004677721008192748,
"max_s": 0.0048383449902758
},
"linear.attn_out_proj.flatten_contiguous": {
"count": 3,
"mean_s": 4.373335589965184e-06,
"p50_s": 4.224013537168503e-06,
"p90_s": 4.607997834682465e-06,
"p95_s": 4.6559958718717095e-06,
"p99_s": 4.694394301623106e-06,
"min_s": 4.191999323666096e-06,
"max_s": 4.703993909060955e-06
},
"linear.attn_out_proj.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.attn_out_proj.packed_weight_wrapper": {
"count": 3,
"mean_s": 1.0650682573517164e-05,
"p50_s": 1.0048039257526398e-05,
"p90_s": 1.1801626533269882e-05,
"p95_s": 1.2020824942737818e-05,
"p99_s": 1.2196183670312166e-05,
"min_s": 9.66398511081934e-06,
"max_s": 1.2240023352205753e-05
},
"linear.attn_out_proj.bias_cast": {
"count": 3,
"mean_s": 3.53597958261768e-06,
"p50_s": 3.503984771668911e-06,
"p90_s": 3.6448007449507716e-06,
"p95_s": 3.662402741611004e-06,
"p99_s": 3.6764843389391897e-06,
"min_s": 3.4239492379128933e-06,
"max_s": 3.6800047382712364e-06
},
"linear.attn_out_proj.activation_scale": {
"count": 3,
"mean_s": 0.0022405670024454594,
"p50_s": 0.0022424450144171715,
"p90_s": 0.002249599387869239,
"p95_s": 0.0022504936845507474,
"p99_s": 0.002251209121895954,
"min_s": 0.002227868011686951,
"max_s": 0.0022513879812322557
},
"linear.attn_out_proj.scale_to_device": {
"count": 3,
"mean_s": 1.0266647829363743e-05,
"p50_s": 1.0079995263367891e-05,
"p90_s": 1.0655971709638835e-05,
"p95_s": 1.0727968765422702e-05,
"p99_s": 1.0785566410049796e-05,
"min_s": 9.91998240351677e-06,
"max_s": 1.079996582120657e-05
},
"linear.attn_out_proj.activation_quant_pack": {
"count": 3,
"mean_s": 0.002841467969119549,
"p50_s": 0.002840427972842008,
"p90_s": 0.002845663169864565,
"p95_s": 0.002846317569492385,
"p99_s": 0.0028468410891946405,
"min_s": 0.002837003965396434,
"max_s": 0.0028469719691202044
},
"linear.attn_out_proj.activation_quant_wrap": {
"count": 3,
"mean_s": 6.853311788290739e-06,
"p50_s": 6.783986464142799e-06,
"p90_s": 7.1551650762557985e-06,
"p95_s": 7.201562402769923e-06,
"p99_s": 7.238680263981223e-06,
"min_s": 6.52798917144537e-06,
"max_s": 7.247959729284048e-06
},
"linear.attn_out_proj.gemm": {
"count": 3,
"mean_s": 0.007955518667586148,
"p50_s": 0.007876515970565379,
"p90_s": 0.008085207198746502,
"p95_s": 0.008111293602269143,
"p99_s": 0.008132162725087256,
"min_s": 0.007852660026401281,
"max_s": 0.008137380005791783
},
"linear.attn_out_proj.slice_reshape": {
"count": 3,
"mean_s": 5.562673322856426e-06,
"p50_s": 5.4879928939044476e-06,
"p90_s": 5.718402098864317e-06,
"p95_s": 5.7472032494843004e-06,
"p99_s": 5.770244169980288e-06,
"min_s": 5.424022674560547e-06,
"max_s": 5.776004400104284e-06
},
"attn_out_proj": {
"count": 3,
"mean_s": 0.01312111063937967,
"p50_s": 0.013056731957476586,
"p90_s": 0.013249423156958075,
"p95_s": 0.01327350955689326,
"p99_s": 0.013292778676841407,
"min_s": 0.01300900400383398,
"max_s": 0.013297595956828445
},
"gate_msa": {
"count": 3,
"mean_s": 0.008771182018487403,
"p50_s": 0.008706916007213295,
"p90_s": 0.008864918421022595,
"p95_s": 0.008884668722748757,
"p99_s": 0.008900468964129686,
"min_s": 0.008702211023773998,
"max_s": 0.008904419024474919
},
"norm2": {
"count": 3,
"mean_s": 0.003586383982716749,
"p50_s": 0.0035896420013159513,
"p90_s": 0.0036038891877979043,
"p95_s": 0.003605670086108148,
"p99_s": 0.0036070948047563435,
"min_s": 0.003562058962415904,
"max_s": 0.003607450984418392
},
"modulate_mlp": {
"count": 3,
"mean_s": 0.010889343665136645,
"p50_s": 0.01104473602026701,
"p90_s": 0.011215244000777603,
"p95_s": 0.011236557498341427,
"p99_s": 0.011253608296392485,
"min_s": 0.010365423979237676,
"max_s": 0.01125787099590525
},
"linear.mlp_fc1.flatten_contiguous": {
"count": 3,
"mean_s": 4.560007558514674e-06,
"p50_s": 4.415982402861118e-06,
"p90_s": 4.851189441978932e-06,
"p95_s": 4.905590321868658e-06,
"p99_s": 4.9491110257804395e-06,
"min_s": 4.3040490709245205e-06,
"max_s": 4.959991201758385e-06
},
"linear.mlp_fc1.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc1.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.78933739537994e-06,
"p50_s": 8.608039934188128e-06,
"p90_s": 9.23519255593419e-06,
"p95_s": 9.313586633652448e-06,
"p99_s": 9.376301895827056e-06,
"min_s": 8.367991540580988e-06,
"max_s": 9.391980711370707e-06
},
"linear.mlp_fc1.bias_cast": {
"count": 3,
"mean_s": 3.509359279026588e-06,
"p50_s": 3.503984771668911e-06,
"p90_s": 3.6064302548766136e-06,
"p95_s": 3.6192359402775764e-06,
"p99_s": 3.6294804885983467e-06,
"min_s": 3.392051439732313e-06,
"max_s": 3.6320416256785393e-06
},
"linear.mlp_fc1.activation_scale": {
"count": 3,
"mean_s": 0.0017761682975105941,
"p50_s": 0.0017401569639332592,
"p90_s": 0.0018271209788508712,
"p95_s": 0.0018379914807155728,
"p99_s": 0.001846687882207334,
"min_s": 0.0017394859460182488,
"max_s": 0.0018488619825802743
},
"linear.mlp_fc1.scale_to_device": {
"count": 3,
"mean_s": 1.0149359392623106e-05,
"p50_s": 9.776034858077765e-06,
"p90_s": 1.0953645687550305e-05,
"p95_s": 1.1100847041234373e-05,
"p99_s": 1.1218608124181627e-05,
"min_s": 9.423994924873114e-06,
"max_s": 1.1248048394918442e-05
},
"linear.mlp_fc1.activation_quant_pack": {
"count": 3,
"mean_s": 0.0022371909581124783,
"p50_s": 0.002281755965668708,
"p90_s": 0.0022871831548400222,
"p95_s": 0.0022878615534864368,
"p99_s": 0.002288404272403568,
"min_s": 0.002141276956535876,
"max_s": 0.002288539952132851
},
"linear.mlp_fc1.activation_quant_wrap": {
"count": 3,
"mean_s": 6.458683249851068e-06,
"p50_s": 6.336020305752754e-06,
"p90_s": 6.720004603266716e-06,
"p95_s": 6.768002640455961e-06,
"p99_s": 6.8064010702073575e-06,
"min_s": 6.224028766155243e-06,
"max_s": 6.8160006776452065e-06
},
"linear.mlp_fc1.gemm": {
"count": 3,
"mean_s": 0.03520335033924008,
"p50_s": 0.03511902003083378,
"p90_s": 0.03545432800892741,
"p95_s": 0.0354962415061891,
"p99_s": 0.035529772303998466,
"min_s": 0.03495287598343566,
"max_s": 0.03553815500345081
},
"linear.mlp_fc1.slice_reshape": {
"count": 3,
"mean_s": 5.594648731251557e-06,
"p50_s": 5.6800199672579765e-06,
"p90_s": 5.79517800360918e-06,
"p95_s": 5.809572758153081e-06,
"p99_s": 5.821088561788201e-06,
"min_s": 5.279958713799715e-06,
"max_s": 5.8239675126969814e-06
},
"mlp_fc1": {
"count": 3,
"mean_s": 0.039302336323695876,
"p50_s": 0.03923152602510527,
"p90_s": 0.039652977988589556,
"p95_s": 0.03970565948402509,
"p99_s": 0.03974780468037352,
"min_s": 0.03891714196652174,
"max_s": 0.03975834097946063
},
"mlp_swiglu": {
"count": 3,
"mean_s": 0.02406409801915288,
"p50_s": 0.023905293026473373,
"p90_s": 0.024343602627050133,
"p95_s": 0.024398391327122227,
"p99_s": 0.0244422222871799,
"min_s": 0.023833821003790945,
"max_s": 0.02445318002719432
},
"linear.mlp_fc2.flatten_contiguous": {
"count": 3,
"mean_s": 4.298674563566844e-06,
"p50_s": 4.336005076766014e-06,
"p90_s": 4.591979086399079e-06,
"p95_s": 4.6239758376032116e-06,
"p99_s": 4.649573238566518e-06,
"min_s": 3.9040460251271725e-06,
"max_s": 4.6559725888073444e-06
},
"linear.mlp_fc2.pre_quant_scale": {
"count": 3,
"mean_s": 0.0,
"p50_s": 0.0,
"p90_s": 0.0,
"p95_s": 0.0,
"p99_s": 0.0,
"min_s": 0.0,
"max_s": 0.0
},
"linear.mlp_fc2.packed_weight_wrapper": {
"count": 3,
"mean_s": 8.506661591430506e-06,
"p50_s": 8.463975973427296e-06,
"p90_s": 8.899183012545108e-06,
"p95_s": 8.953583892434835e-06,
"p99_s": 8.997104596346618e-06,
"min_s": 8.048024028539658e-06,
"max_s": 9.007984772324562e-06
},
"linear.mlp_fc2.bias_cast": {
"count": 3,
"mean_s": 3.5466509871184826e-06,
"p50_s": 3.4879776649177074e-06,
"p90_s": 3.641599323600531e-06,
"p95_s": 3.6608020309358834e-06,
"p99_s": 3.6761641968041657e-06,
"min_s": 3.471970558166504e-06,
"max_s": 3.6800047382712364e-06
},
"linear.mlp_fc2.activation_scale": {
"count": 3,
"mean_s": 0.004390478308778256,
"p50_s": 0.004361385013908148,
"p90_s": 0.004447554564103484,
"p95_s": 0.004458325757877901,
"p99_s": 0.004466942712897435,
"min_s": 0.0043409529607743025,
"max_s": 0.004469096951652318
},
"linear.mlp_fc2.scale_to_device": {
"count": 3,
"mean_s": 1.036798736701409e-05,
"p50_s": 9.519979357719421e-06,
"p90_s": 1.2169592082500458e-05,
"p95_s": 1.2500793673098087e-05,
"p99_s": 1.2765754945576191e-05,
"min_s": 8.751987479627132e-06,
"max_s": 1.2831995263695717e-05
},
"linear.mlp_fc2.activation_quant_pack": {
"count": 3,
"mean_s": 0.0059309463443545,
"p50_s": 0.00591752800391987,
"p90_s": 0.0060289128334261475,
"p95_s": 0.006042835937114432,
"p99_s": 0.00605397442006506,
"min_s": 0.005818551988340914,
"max_s": 0.006056759040802717
},
"linear.mlp_fc2.activation_quant_wrap": {
"count": 3,
"mean_s": 6.405326227347056e-06,
"p50_s": 6.52798917144537e-06,
"p90_s": 6.566406227648258e-06,
"p95_s": 6.571208359673619e-06,
"p99_s": 6.575050065293908e-06,
"min_s": 6.111979018896818e-06,
"max_s": 6.57601049169898e-06
},
"linear.mlp_fc2.gemm": {
"count": 3,
"mean_s": 0.05613339397435387,
"p50_s": 0.056144060974474996,
"p90_s": 0.05626750335795805,
"p95_s": 0.05628293365589343,
"p99_s": 0.056295277894241734,
"min_s": 0.0559577569947578,
"max_s": 0.05629836395382881
},
"linear.mlp_fc2.slice_reshape": {
"count": 3,
"mean_s": 5.423983869453271e-06,
"p50_s": 5.423964466899633e-06,
"p90_s": 5.462381523102522e-06,
"p95_s": 5.467183655127883e-06,
"p99_s": 5.471025360748172e-06,
"min_s": 5.376001354306936e-06,
"max_s": 5.471985787153244e-06
},
"mlp_fc2": {
"count": 3,
"mean_s": 0.06653381067250545,
"p50_s": 0.06663658900652081,
"p90_s": 0.06664222099352628,
"p95_s": 0.06664292499190197,
"p99_s": 0.06664348819060252,
"min_s": 0.06632121402071789,
"max_s": 0.06664362899027765
},
"gate_mlp": {
"count": 3,
"mean_s": 0.014599877002183348,
"p50_s": 0.008974290976766497,
"p90_s": 0.022819262195844205,
"p95_s": 0.024549883598228916,
"p99_s": 0.025934380720136686,
"min_s": 0.008544835029169917,
"max_s": 0.02628050500061363
},
"block_total": {
"count": 3,
"mean_s": 0.5241046730079688,
"p50_s": 0.5232612389954738,
"p90_s": 0.5255022222059779,
"p95_s": 0.5257823451072908,
"p99_s": 0.5260064434283414,
"min_s": 0.5229903120198287,
"max_s": 0.5260624680086039
}
},
"module_forward": {
"count": 3,
"mean_s": 0.4765262039921557,
"p50_s": 0.4681359010282904,
"p90_s": 0.4901721953880042,
"p95_s": 0.4929267321829684,
"p99_s": 0.4951303616189398,
"min_s": 0.46576144197024405,
"max_s": 0.49568126897793263
},
"module_forward_checksum": -778077863936.0,
"fused_elementwise": true,
"profiler_summary": {
"profiled_iterations": 1,
"runtime_kernel_launches": 0,
"runtime_kernel_launches_per_block": 0.0,
"positive_self_device_allocated_bytes": 7351755264
},
"profiler_top_events": [
{
"key": "comfy_kitchen::rms_rope_split_half_",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 500.23900000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 37810, 1, 48, 2, 2], [128], [128], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 416.1590000000001,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [5376, 7168], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 323.05600000000004,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 3584], [5376, 3584], [], [], [37888, 448], [5376, 448], [], [], []]"
},
{
"key": "aten::_to_copy",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 217.82299999999896,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], [], [], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 201.44000000000005,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 7168], [], [], [], []]"
},
{
"key": "aten::to",
"count": 13,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 188.896000000002,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 186.01600000000002,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::_fused_rms_norm",
"count": 2,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 175.42400000000006,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [5376], []]"
},
{
"key": "aten::copy_",
"count": 8,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 147.2470000000003,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 133.7439999999997,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [28672, 5376], []]"
},
{
"key": "aten::linear",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 129.29600000000028,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 14336], [5376, 14336], []]"
},
{
"key": "aten::empty",
"count": 29,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 120.14400000000344,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 6267507200,
"input_shapes": "[[], [], [], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 112.3680000000004,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 7168], [5376, 7168], [], [], [37888, 896], [5376, 896], [], [], []]"
},
{
"key": "comfy_kitchen::scaled_mm_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 109.55199999999968,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": -512,
"input_shapes": "[[37824, 2688], [28672, 2688], [], [], [37888, 336], [28672, 336], [], [], []]"
},
{
"key": "aten::zeros",
"count": 3,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 107.92000000000007,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[], [], [], [], []]"
},
{
"key": "aten::mean",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 86.096,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 14336,
"input_shapes": "[[1, 37810, 56, 128], [], [], []]"
},
{
"key": "sageattention_sm89::qk_int8_sv_f8_accum_f16_fuse_v_scale_attn_inst_buf",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 77.87199999999984,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[1, 37810, 56, 128], [1, 37810, 56, 128], [1, 128, 56, 37824], [1, 37810, 56, 128], [1, 56, 1184], [1, 56, 591], [1, 56, 128], [], [], [], [], []]"
},
{
"key": "aten::mul",
"count": 4,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 70.12799999999925,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 2048,
"input_shapes": "[[], []]"
},
{
"key": "comfy_kitchen::quantize_nvfp4",
"count": 1,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 65.21600000000035,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
},
{
"key": "aten::slice",
"count": 19,
"device_type": "DeviceType.CPU",
"cpu_time_total_us": 63.07199999999966,
"device_time_total_us": 0.0,
"self_device_time_total_us": 0.0,
"self_device_memory_usage_bytes": 0,
"input_shapes": "[[37810, 5376], [], [], [], []]"
}
]
}

38
benchmarks/gb10-cute-qkv-runtime-block-gate-summary.json Normal file
View file

@ -0,0 +1,38 @@
{
"device": "NVIDIA GB10",
"workload": {
"width": 1344,
"height": 768,
"frames": 124,
"tokens": 37810,
"seed": 440420
},
"method": "Alternating baseline and ring execution inside one loaded block",
"capacity_2048": {
"block_0": {
"equal": true,
"baseline_p50_ms": 461.54195550479926,
"ring_p50_ms": 465.29679899686016,
"improvement_percent": -0.8135432645455021
},
"block_24": {
"equal": true,
"baseline_p50_ms": 467.8640030033421,
"ring_p50_ms": 470.3063364722766,
"improvement_percent": -0.5220178199768499
},
"block_49": {
"equal": true,
"baseline_p50_ms": 462.3519679880701,
"ring_p50_ms": 464.76577199064195,
"improvement_percent": -0.5220706668721542
}
},
"block_24_capacity_sweep": {
"3072": -1.0036016035203765,
"4096": -0.2438296625217884,
"8192": -4.798436757705438,
"37888": -14.49047057045394
},
"decision": "Reject runtime dispatch. Keep opt-in and disabled until launch fusion or a different persistent scheduler passes the alternating block gate. Skip trajectory validation."
}

42
benchmarks/gb10-cute-qkv-runtime-block24-2048.json Normal file
View file

@ -0,0 +1,42 @@
{
"device": "NVIDIA GB10",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"rows": 2048,
"role": "h3_attn_qkv",
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"ring_ms": 1.6192415237426758,
"reference_ms": 1.859343910217285,
"improvement_percent": 12.913285441989618
}

42
benchmarks/gb10-cute-qkv-runtime-block24-full.json Normal file
View file

@ -0,0 +1,42 @@
{
"device": "NVIDIA GB10",
"metadata": {
"width": 1344,
"height": 768,
"frames": 124,
"steps": 12,
"sampler_step": 1,
"seed": 440420,
"text_tokens": 100,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
]
},
"block_index": 24,
"rows": 37810,
"role": "h3_attn_qkv",
"equal": true,
"max_abs": 0.0,
"mean_abs": 0.0,
"ring_ms": 31.741008758544922,
"reference_ms": 46.7579345703125,
"improvement_percent": 32.11631555107678
}

1392
benchmarks/gb10-deployed-modulate-fusion-block24-profile.json Normal file
View file

File diff suppressed because it is too large Load diff

95
benchmarks/gb10-four-gemm-nvfp4-roofline.json Normal file
View file

@ -0,0 +1,95 @@
{
"status": "complete",
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"workload": {
"resolution": [1344, 768],
"frames": 124,
"tokens": 37810,
"block": 24,
"seed": 440420
},
"method": {
"flops": "2*M*N*K dense-equivalent operations",
"l2_traffic": "Nsight Compute lts__t_bytes.sum with cache control disabled",
"off_chip_proxy": "32 bytes times warm-cache L2 read plus write lookup-miss sectors",
"practical_compute_ceiling_tflops_per_s": 321.455895,
"theoretical_memory_bandwidth_gb_per_s": 273.0,
"practical_crossover_flops_per_byte": 1177.49412,
"note": "GB10 exposes no direct physical DRAM-byte counter; classification uses the documented bandwidth and the warm-cache off-chip request proxy."
},
"gemms": [
{
"role": "attention_qkv",
"shape_mnk": [37810, 21504, 5376],
"kernel": "standard",
"duration_ms": 27.195296,
"dense_equivalent_tflops": 8.742088,
"achieved_tflops_per_s": 321.455895,
"l2_requested_gb": 40.129071,
"off_chip_proxy_gb": 4.368326,
"off_chip_arithmetic_intensity": 2001.244307,
"ideal_unique_bytes_gb": 1.805498,
"off_chip_amplification": 2.419458,
"classification": "compute-bound"
},
{
"role": "attention_output",
"shape_mnk": [37810, 5376, 7168],
"kernel": "standard",
"duration_ms": 9.25072,
"dense_equivalent_tflops": 2.914029,
"achieved_tflops_per_s": 315.005686,
"l2_requested_gb": 13.24078,
"off_chip_proxy_gb": 1.337147,
"off_chip_arithmetic_intensity": 2179.28899,
"ideal_unique_bytes_gb": 0.580659,
"off_chip_amplification": 2.302809,
"classification": "compute-bound"
},
{
"role": "mlp_fc1",
"shape_mnk": [37810, 28672, 5376],
"kernel": "standard",
"duration_ms": 37.024512,
"dense_equivalent_tflops": 11.656118,
"achieved_tflops_per_s": 314.821641,
"l2_requested_gb": 53.507764,
"off_chip_proxy_gb": 5.823759,
"off_chip_arithmetic_intensity": 2001.476751,
"ideal_unique_bytes_gb": 2.369218,
"off_chip_amplification": 2.458093,
"classification": "compute-bound"
},
{
"role": "mlp_fc2",
"shape_mnk": [37810, 5376, 14336],
"kernel": "stream-k",
"duration_ms": 54.973312,
"dense_equivalent_tflops": 5.828059,
"achieved_tflops_per_s": 106.016148,
"l2_requested_gb": 26.07999,
"off_chip_proxy_gb": 12.161131,
"off_chip_arithmetic_intensity": 479.236573,
"ideal_unique_bytes_gb": 0.754785,
"off_chip_amplification": 16.112046,
"classification": "bandwidth-bound",
"roofline_at_273_gb_per_s_ms": 44.546268,
"measured_bandwidth_efficiency_percent": 81.032534,
"traffic_required_to_reach_practical_compute_ridge_gb": 4.949544,
"required_traffic_reduction_percent": 59.299,
"practical_compute_floor_ms": 18.130197
}
],
"decision": {
"primary_target": "mlp_fc2",
"do_not_target_first": ["attention_qkv", "attention_output", "mlp_fc1"],
"reason": "QKV, attention output, and FC1 cluster at 315-321 dense-equivalent TFLOP/s and lie above the practical ridge point. FC2 uses stream-K, reaches only 106 TFLOP/s, and generates 12.16 GB of off-chip request traffic, 16.1x its ideal unique bytes.",
"next_experiment": "Benchmark an exact FC2 schedule that reduces K-split/stream-K rereads and reuses the fused SwiGLU-produced activation tiles. Require bit-exact BF16 output and at least 3% complete-block improvement before integration."
},
"source_artifacts": [
"benchmarks/gb10-fully-fused-block24-fresh-l2.csv",
"benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv",
"benchmarks/gb10-fully-fused-fresh-nsight-summary.json"
]
}

31
benchmarks/gb10-fully-fused-block24-fresh-capture.json Normal file
View file

@ -0,0 +1,31 @@
{
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"fused_elementwise": true,
"fused_nvfp4_modulation": true,
"fused_nvfp4_swiglu": true,
"nvfp4_scale_backend": "vortex",
"sage_qkv_layout": "strided_nhd",
"module_forward_checksum": 273326784.0,
"capture": "one warmed block between cudaProfilerStart/Stop"
}

47
benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv Normal file
View file

File diff suppressed because one or more lines are too long

46
benchmarks/gb10-fully-fused-block24-fresh-l2.csv Normal file
View file

File diff suppressed because one or more lines are too long

31
benchmarks/gb10-fully-fused-block24-fresh-ncu-capture.json Normal file
View file

@ -0,0 +1,31 @@
{
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"fused_elementwise": true,
"fused_nvfp4_modulation": true,
"fused_nvfp4_swiglu": true,
"nvfp4_scale_backend": "vortex",
"sage_qkv_layout": "strided_nhd",
"module_forward_checksum": 273326784.0,
"capture": "one warmed block between cudaProfilerStart/Stop"
}

31
benchmarks/gb10-fully-fused-block24-fresh-ncu-misses-warm-capture.json Normal file
View file

@ -0,0 +1,31 @@
{
"block_index": 24,
"hidden_shape": [
37810,
5376
],
"segments": [
[
0,
100,
1
],
[
100,
514,
2
],
[
514,
37810,
0
]
],
"fused_elementwise": true,
"fused_nvfp4_modulation": true,
"fused_nvfp4_swiglu": true,
"nvfp4_scale_backend": "vortex",
"sage_qkv_layout": "strided_nhd",
"module_forward_checksum": 273326784.0,
"capture": "one warmed block between cudaProfilerStart/Stop"
}

1292
benchmarks/gb10-fully-fused-block24-fresh-torch.json Normal file
View file

File diff suppressed because it is too large Load diff

BIN
benchmarks/gb10-fully-fused-block24-fresh.nsys-rep Normal file
View file

Binary file not shown.

1292
benchmarks/gb10-fully-fused-block24-profile.json Normal file
View file

File diff suppressed because it is too large Load diff

125
benchmarks/gb10-fully-fused-fresh-nsight-summary.json Normal file
View file

@ -0,0 +1,125 @@
{
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"workload": {
"resolution": [1344, 768],
"frames": 124,
"packed_tokens": 37810,
"block": 24,
"attention": "sage2",
"seed": 440420
},
"configuration": {
"H3_NVFP4_SCALE_BACKEND": "vortex",
"H3_NVFP4_SCALE_VERSION": "1",
"H3_FUSED_ELEMENTWISE": "1",
"H3_NVFP4_MODULATE_FUSION": "1",
"H3_NVFP4_SWIGLU_FUSION": "1",
"H3_SAGE_QKV_LAYOUT": "strided_nhd"
},
"block_24": {
"uninstrumented_module_p50_ms": 458.7751985236537,
"nsight_gpu_span_ms": 465.779616,
"kernel_time_ms": 465.695712,
"kernel_launches": 41,
"inter_kernel_idle_ms": 0.083904,
"average_inter_kernel_idle_us": 2.098,
"maximum_inter_kernel_idle_us": 3.36,
"components": {
"sage2": {"milliseconds": 268.224256, "percent": 57.59646247290334},
"nvfp4_gemms": {"milliseconds": 125.113088, "percent": 26.865844966165376},
"nvfp4_packing": {"milliseconds": 41.526688, "percent": 8.917129131736562},
"norm_and_rope": {"milliseconds": 19.863552, "percent": 4.265349988878575},
"remaining_gate_add": {"milliseconds": 10.940192, "percent": 2.349214673464719},
"other": {"milliseconds": 0.027936, "percent": 0.005998732851428}
},
"producer_detail": {
"modulated_qkv_and_fc1_ms": 13.041792,
"swiglu_fc2_ms": 23.043104,
"attention_output_pack_ms": 5.126112,
"scale_finalize_and_zero_fill_ms": 0.31568,
"standalone_modulation_kernels": 0,
"standalone_swiglu_kernels": 0,
"gate_add_kernels": 2
},
"allocation_and_conversion": {
"positive_self_allocated_bytes_per_block": 8009578496,
"aten_empty_calls_per_block": 44,
"large_layout_copy_kernels": 0,
"explicit_cuda_memcpy_ops": 0,
"remaining_contiguous_calls_per_block": 4,
"remaining_contiguous_shape": [3, 5376],
"remaining_to_copy_calls_per_block": 2,
"note": "Remaining contiguous and to-copy operations are small AdaLN/scalar conversions, not full activation layout materializations."
}
},
"one_warmed_sampling_step": {
"elapsed_seconds": 23.707971603027545,
"gpu_span_seconds": 23.706977664,
"kernel_time_seconds": 23.693982688,
"kernel_launches": 2694,
"inter_kernel_idle_ms": 12.994976,
"average_inter_kernel_idle_us": 4.825,
"maximum_inter_kernel_idle_ms": 6.395168,
"peak_allocated_bytes": 15570401280,
"peak_reserved_bytes": 18538823680,
"explicit_cuda_memcpy_ops": 9,
"explicit_cuda_memcpy_bytes": 15282118,
"explicit_cuda_memcpy_ms": 0.169856,
"components": {
"sage2": {"milliseconds": 13760.566304, "percent": 58.07620645797613},
"nvfp4_gemms": {"milliseconds": 6269.804512, "percent": 26.46158982455656},
"nvfp4_packing": {"milliseconds": 2093.052672, "percent": 8.833688702997335},
"norm_and_rope": {"milliseconds": 987.609696, "percent": 4.168187801117043},
"remaining_gate_add": {"milliseconds": 545.150976, "percent": 2.3007992500817345},
"other": {"milliseconds": 37.798528, "percent": 0.159528965271231}
},
"gpu_metrics": {
"gr_active_average_percent": 99.57,
"sms_active_average_percent": 99.27,
"sm_issue_average_percent": 25.63,
"tensor_active_average_percent": 53.72
}
},
"memory_traffic": {
"measurement_note": "GB10 Nsight exposes no direct physical DRAM-byte counter. L2 requested bytes are measured with lts__t_bytes.sum. The off-chip proxy is 32 bytes times warm-cache L2 read/write miss sectors with Nsight Compute cache control disabled.",
"l2_requested_bytes": {
"total": 320998821280,
"sage2": 166126147104,
"nvfp4_gemms": 132957604576,
"nvfp4_packing": 14638661888,
"norm_and_rope": 4802883392,
"remaining_gate_add": 2472452288
},
"off_chip_request_proxy_bytes": {
"total": 42116618176,
"sage2": 6357343328,
"nvfp4_gemms": 23690363008,
"nvfp4_packing": 7685899232,
"norm_and_rope": 2753536064,
"remaining_gate_add": 1629476544
},
"off_chip_request_proxy_percent": {
"sage2": 15.094619661610697,
"nvfp4_gemms": 56.249442699793654,
"nvfp4_packing": 18.249089230957726,
"norm_and_rope": 6.537885004188424,
"remaining_gate_add": 3.868963403449499
}
},
"decision": {
"time_bottleneck": "Sage2, led by the 246.629 ms attention mainloop in block 24.",
"memory_traffic_bottleneck": "The four NVFP4 GEMMs, with 56.25% of the warm-cache off-chip request proxy.",
"cpu_launch_bottleneck": false,
"next_kernel": "No implementation target is selected solely from the old profile. Any next target must use this fresh split between Sage2 compute time and NVFP4 GEMM memory traffic."
},
"artifacts": [
"benchmarks/gb10-fully-fused-block24-fresh.nsys-rep",
"benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep",
"benchmarks/gb10-fully-fused-block24-fresh-capture.json",
"benchmarks/gb10-fully-fused-block24-fresh-torch.json",
"benchmarks/gb10-fully-fused-one-step-fresh.json",
"benchmarks/gb10-fully-fused-block24-fresh-l2.csv",
"benchmarks/gb10-fully-fused-block24-fresh-l2-misses-warm.csv"
]
}

24
benchmarks/gb10-fully-fused-one-step-fresh.json Normal file
View file

@ -0,0 +1,24 @@
{
"device": "NVIDIA GB10",
"torch": "2.9.1+cu130",
"attention": "sage2",
"resolution": [
1344,
768
],
"frames": 124,
"steps": 1,
"seed": 440420,
"text_tokens": 100,
"warmup_runs": 1,
"cuda_profiler_capture": true,
"elapsed_seconds": 23.707971603027545,
"stage_trace": [],
"checksums": [
-197605.3125,
528.5980224609375
],
"peak_allocated_bytes": 15570401280,
"peak_reserved_bytes": 18538823680,
"measurement_policy": "uninstrumented production sampling wall time inside a whole-run NVTX range"
}

BIN
benchmarks/gb10-fully-fused-one-step-fresh.nsys-rep Normal file
View file

Binary file not shown.

1532
benchmarks/gb10-fused-block-residual-profile-component2-1344x768-124f-seed440420.json Normal file
View file

File diff suppressed because it is too large Load diff

77
benchmarks/gb10-h3-fused-elementwise-1344x768-124f-seed440420.json Normal file
View file

@ -0,0 +1,77 @@
{
"device": "NVIDIA GB10",
"compute_capability": "SM121",
"torch": "2.9.1+cu130",
"attention": "sage2",
"resolution": [
1344,
768
],
"frames": 124,
"packed_tokens": 37810,
"steps": 12,
"seed": 440420,
"feature_flag": "H3_FUSED_ELEMENTWISE=1",
"isolated_cuda_tests": {
"modulation_bit_exact": true,
"gate_add_bit_exact": true
},
"real_block_results": [
{
"block": 0,
"eager_p50_seconds": 0.5083600514917634,
"fused_p50_seconds": 0.4901100995193701,
"speedup": 1.0372364331816264,
"latency_reduction": 0.03589965796651318,
"max_abs_difference": 0.0,
"checksum": -8279139.0
},
{
"block": 24,
"eager_p50_seconds": 0.5089578525221441,
"fused_p50_seconds": 0.4891547074948903,
"speedup": 1.040484420826024,
"latency_reduction": 0.03890920422804989,
"max_abs_difference": 0.0,
"checksum": 89071408.0
},
{
"block": 49,
"eager_p50_seconds": 0.5089952205016743,
"fused_p50_seconds": 0.4891957634827122,
"speedup": 1.0404734842305352,
"latency_reduction": 0.03889910203763303,
"max_abs_difference": 0.0,
"checksum": -40617144320.0
}
],
"sampling_12_step": {
"eager_seconds": 310.109976747015,
"fused_seconds": 301.04568115097936,
"speedup": 1.0301093693202321,
"latency_reduction": 0.029229293720628102,
"eager_checksums": [
-91481.7890625,
678.401611328125
],
"fused_checksums": [
-91481.7890625,
678.401611328125
]
},
"sampling_2_step_memory": {
"eager_peak_allocated_bytes": 19278355968,
"fused_peak_allocated_bytes": 18869530112,
"allocated_bytes_saved": 408825856,
"eager_peak_reserved_bytes": 21976055808,
"fused_peak_reserved_bytes": 21846032384,
"reserved_bytes_saved": 130023424
},
"validation": {
"deployed_tests": "32 passed",
"block_parity": "torch.equal at blocks 0, 24, and 49",
"sampling_parity": "identical video and audio checksums after 12 steps",
"generic_default": "eager path remains default unless H3_FUSED_ELEMENTWISE is enabled",
"spark_default": "enabled in Spark compose files"
}
}

9100
benchmarks/gb10-nvfp4-component2-profile-1344x768-124f-seed440420.json Normal file
View file

File diff suppressed because it is too large Load diff

27
benchmarks/gb10-nvfp4-lora-producer-fusion-smoke.json Normal file
View file

@ -0,0 +1,27 @@
{
"device": "NVIDIA GB10",
"resolution": [
320,
192
],
"frames": 22,
"steps": 1,
"feature": "lora_producer_fusion",
"seed": 440420,
"baseline_seconds": 3.3671130429720506,
"candidate_seconds": 0.32701997098047286,
"improvement_percent": 90.28782322402155,
"video_equal": true,
"audio_equal": true,
"video_max_abs": 0.0,
"audio_max_abs": 0.0,
"reference_checksums": [
911.742431640625,
45.64133834838867
],
"candidate_checksums": [
911.742431640625,
45.64133834838867
],
"equal": true
}

Some files were not shown because too many files have changed in this diff Show more