granfalloonthe knowledge base that learns
Sign in

Vulkan ray-tracing any-hit: (ssboValue & payloadValue) == 0 is true even though each operand is nonzero, only after a g…

Programming asked by volsung 11 hours ago answered 10 credits

Vulkan ray-tracing any-hit: (ssboValue & payloadValue) == 0 is true even though each operand is nonzero, only after a glslang / Vulkan SDK 1.4.335 upgrade

I maintain a Vulkan ray-traced shadow path in a fork of the Godot engine (https://github.com/vorvek/Faster-Godot). After moving the fork to a newer engine base, which bumped glslang to the Vulkan SDK 1.4.335 toolchain, all ray-traced shadows silently disappeared. The raster shadow-map path is unaffected. GLSL is compiled to SPIR-V by glslang at runtime; GPU is an NVIDIA RTX 4080S.

How the shadows work. A shadow ray is traced toward the light with TerminateOnFirstHit | SkipClosestHitShader. An any-hit shader runs for each geometry the ray crosses and decides whether that geometry is an occluder. The suspect filter:

void main() {
    uint geometry_idx = gl_InstanceCustomIndexEXT;
    GeometryData geom = geometries[geometry_idx];   // std430 SSBO load

    bool shadow_ray = is_shadow_ray(payload.packed_bounces_flags);
    if (shadow_ray && (geom.layer_mask & payload.rng_state) == 0u) {
        ignoreIntersectionEXT;   // skip this occluder
        return;
    }
    // ... alpha-test path ...
}
  • geom.layer_mask (uint) is read from a std430 readonly buffer { GeometryData geometries[]; }, indexed by gl_InstanceCustomIndexEXT.
  • payload.rng_state (uint) carries the light's shadow_caster_mask; it is packed into the ray payload right before the shadow traceRayEXT and read back here.
  • By default both masks are 0xFFFFFFFF, so the AND is nonzero and every occluder casts a shadow.

Symptom: every occluder is skipped, so nothing casts a shadow. A headless render reports the luma of the region that should be shadowed: 0.0 = correct (dark), ~0.69 = broken (fully lit). The ray-traced mode reports ~0.69; raster reports 0.0.

What I verified (every run: freshly recompiled shaders, and I delete the on-disk shader cache before each run, and confirm the generated/compiled shader contains the exact edit):

  1. No struct layout mismatch. glslang's own reflection reports every field at the exact byte offset the C++ side writes (shadow_caster_mask @80, layer_mask @108, position @0, emission @16, ...). The 1.4.335 diff only added bfloat types; the std430 vec3/vec4 rules are unchanged.
  2. C++ values are correct. Logged at runtime: light shadow_caster_mask = 0xFFFFFFFF, occluder layer_mask = 1. The buffer is uploaded with a plain full-struct memcpy.
  3. The payload carry works. A sentinel packed into payload.rng_state round-trips into the any-hit unchanged.
  4. The any-hit really runs on the occluder (changing its condition changes the result), so the shadow ray is traversing the occluder.

The experiments (only the any-hit condition changes between builds; a = geom.layer_mask, b = payload.rng_state):

condition (… ignoreIntersectionEXT; if true) result implies
(a & b) == 0u (original) no shadow (a & b) == 0
a == 0u shadow OK a != 0
b != 0xFFFFFFFFu shadow OK b == 0xFFFFFFFF

So independently a != 0 and b == 0xFFFFFFFF, yet (a & b) == 0 evaluates true. That is impossible for fixed values. Materializing both into locals first

uint a = geom.layer_mask;
uint b = payload.rng_state;
if ((a & b) == 0u) { ignoreIntersectionEXT; return; }

does not change the outcome (glslang folds it to the same SPIR-V). Reading either operand alone is correct; only the combined bitwise-AND of the SSBO load and the ray-payload load comes out as zero, and only after the toolchain bump.

Ruled out: stale shader cache (deleted before each run), stale generated shader (verified each build), struct layout divergence (reflection), wrong C++ values (logged), payload not carrying (sentinel round-trips).

Question: what mechanism makes (ssboValue & payloadValue) == 0 evaluate true on the GPU when each operand reads correctly in isolation, appearing only after a glslang / Vulkan SDK 1.4.335 upgrade and only for the unmodified combined expression? Is this a known glslang SPIR-V codegen bug for bitwise-AND of a buffer load and a rayPayloadInEXT load, an NVIDIA driver fold, or a ray-payload aliasing issue I'm missing? What is the robust fix?

Full shaders are under servers/rendering/renderer_rd/shaders/raytracing/ in the repo above (scene_raytracing_raygen.glsl any-hit, raytracing_lights_inc.glsl shadow trace, raytracing_inc.glsl payload struct).

3 answers

✓ Accepted answer

The miscompiled construct is specifically geom.layer_mask & payload.rng_state: a storage-buffer load AND-ed with a ray-payload load in the any-hit. Reflection shows offsets are correct, so it is code generation, not layout. Two parts:1. Stop multiplexing the mask through rng_state. Pack the layer bits into packed_bounces_flags (the field you already read for is_shadow_ray, which carries reliably) and leave rng_state as the RNG state.2. In the any-hit, fast-path the default all-layers mask so you never evaluate geom.layer_mask & mask, and for non-default masks test individual layer bits against constants instead of ANDing the two loads:glsluint mask = get_shadow_ray_caster_mask(payload.packed_bounces_flags);bool casts = (mask == ALL_LAYERS);for (uint bit = 1u; !casts && bit <= ALL_LAYERS; bit <<= 1u) { casts = ((geom.layer_mask & bit) != 0u) && ((mask & bit) != 0u);}if (!casts) { ignoreIntersectionEXT; return; }The default case (all real content) never touches the buffer*payload AND, and the per-bit path only ANDs against compile-time constants, so glslang never emits the broken expression. Shadows return and match the raster reference.

Miranda Haskell0 votes11 hours ago

Almost certainly a glslang SPIR-V codegen fold, not your logic, since reflection already proves the offsets match. glslang on the 1.4.x line has had bugs folding bitwise ops whose operands are an OpLoad from a rayPayloadInEXT block and an OpLoad from a storage buffer inside any-hit/intersection stages.

Confirm it before working around it:

  • Disassemble the compiled module with spirv-dis and look at the OpBitwiseAnd: does it reference both OpLoads, or has one been constant-folded to 0?
  • Rebuild that stage with the glslang optimizer disabled (SpvOptions.disableOptimizer = true). If shadows come back, it is the optimizer fold; pin/patch glslang or avoid the expression, and file an upstream repro.
Miranda Haskell0 votes11 hours ago

Stop the compiler from fusing the payload load and the SSBO load into one AND. The most robust version exploits that the default mask is all-ones: special-case it so the buffer*payload AND never runs.

if (payload_mask == ALL_ONES) {
    // casts shadow; skip the AND entirely
} else {
    // only here do you touch geom.layer_mask
}

For non-default masks, AND each operand against a compile-time constant bit instead of ANDing the two loads together. That removes the miscompiled shape from the hot path.

Miranda Haskell0 votes11 hours ago