tree 1a338ed7c85f29df419eb647b366fd288c7820a4
parent 3549479dc4ccd92ef3b9e179080dcec1a430b01d
author Nicolas Capens <capn@google.com> 1610491480 -0500
committer Nicolas Capens <nicolascapens@google.com> 1610561364 +0000

Fix back-face culling for vertices near w=0

When the homogeneous w coordinate is near 0, the projected x or y
coordinates (after viewport and subpixel scaling) may be larger than
what's representable by 32-bit signed integers. On x86 this produces
the "indefinite" integer value 0x80000000. This negative value is
returned even for positive inputs. This causes the back-face culling
calculations to reject triangles which are in fact visible.

This change addresses that by introducing the rr::RoundIntClamped()
function, which guarantees that very large positive floating-point
values are converted to a positive integer value near INT_MAX.

Note that the Vulkan spec states that "[the determination whether the
triangle is back-facing or front-facing] is made based on the sign of
the (clipped or unclipped) polygon’s area computed in framebuffer
coordinates." Thus the alternative of performing culling using the
unprojected coordinates would have to take negative viewport dimensions
into account, and may not produce the same results as sub-pixel
snapped coordinates.

Adjust back-face culling of wireframe triangles to use the same
calculations as solid triangles.

Bug: b/177382194
Change-Id: Ice8493d4cfd164638f091bf5a39b5396d65458e2
Reviewed-on: https://swiftshader-review.googlesource.com/c/SwiftShader/+/51648
Tested-by: Nicolas Capens <nicolascapens@google.com>
Reviewed-by: Alexis Hétu <sugoi@google.com>