vkd3d-shader/ir: Make FOG and PSIZE writemasks 0x1 on IO normalization.

This fixes a (sneaky) regression that Gio found on one of his shaders. It is the manifestation of an older bug, I narrowed it down as follows:

Consider the following shader:

// fogger.hlsl
void main(float4 pos : position, out float fog : fog, out float4 out_pos : sv_position)
{
    fog = 1.0;
    out_pos = pos;
}

If we compile this shader using the native compiler and the vs_2_0, and then try to translate that into SPIR-V using current master:

$ fxc10 -T vs_2_0 fogger.hlsl -Fo fogger.d3dbc
$ vkd3d-compiler -x d3dbc -b spirv-binary fogger.d3dbc -o fogger.spv

This will fail because of the validation introduced in f616e6c1.

This is because the native compiler generates instructions with a 0xf dst writemask for oFog and oPts registers (note that these RASTOUT type registers are only relevant in SM 2.0, since SM 3.0 uses OUTPUT registers with dcl instructions instead):

mov oFog, c0.x

even though the fog signature element is scalar. This results in unused writemask elements which we are not allowing on normalized vsir.

We are not noticing this when compiling from HLSL directly:

$ vkd3d-compiler -p vs_2_0 -b spirv-binary fogger.hlsl -o fogger.spv

because we are not emitting these instructions with the 0xf writemask as the native compiler does, this is a bug fixed in 2/3.

Additionally, even after fixing this, this bug doesn't manifest when we use the shader_runner, because there a varying map is used, and in this case the vsir_program_remap_output_signature() pass, specifically the remove_unread_output_components() function, fixes the write_mask for us.

3/3 introduces the proper fix.

While working on this, I made the tests in 1/3. For the reasons above, they are not very good to see the progress that these fixes make, but at least show that there is missing work on validating the usage of the FOG and PSIZE semantics for certain shader models and explain why the fix in 2/3 makes sense.

Merge request reports

Loading