Fix integer overflow in rasterization.
Rasterization of very large triangles was causing signed 32-bit
integer overflow due to multiplying two unsigned 12.4 fixed-point
coordinates. The equations have been reworked to only require
multiplication of 12.4 by 0.4 fixed-point.
Bug b/34078120
Change-Id: I227b81254559af04baf50fbfec6a7f123bd230e3
Reviewed-on: https://swiftshader-review.googlesource.com/8370
Tested-by: Nicolas Capens <capn@google.com>
Reviewed-by: Alexis Hétu <sugoi@google.com>
Reviewed-by: Nicolas Capens <capn@google.com>
diff --git a/src/Shader/SetupRoutine.cpp b/src/Shader/SetupRoutine.cpp
index ac65afa..2fd1883 100644
--- a/src/Shader/SetupRoutine.cpp
+++ b/src/Shader/SetupRoutine.cpp
@@ -585,10 +585,10 @@
Int FDX12 = DX12 << 4;
Int FDY12 = DY12 << 4;
- Int X = DX12 * ((y1 << 4) - Y1) + X1 * DY12;
- Int x = X / FDY12; // Edge
- Int d = X % FDY12; // Error-term
- Int ceil = -d >> 31; // Ceiling division: remainder <= 0
+ Int X = DX12 * ((y1 << 4) - Y1) + (X1 & 0x0000000F) * DY12;
+ Int x = (X1 >> 4) + X / FDY12; // Edge
+ Int d = X % FDY12; // Error-term
+ Int ceil = -d >> 31; // Ceiling division: remainder <= 0
x -= ceil;
d -= ceil & FDY12;
