tree 150ffeee071936a667574fc0609e06c629308a62
parent 391532fef197ffae899b2819d98fc40c8a394679
author Nicolas Capens <capn@google.com> 1570036165 -0400
committer Nicolas Capens <nicolascapens@google.com> 1570050431 +0000

Fix integer overflow in image size calculation

GLsizei is historically defined as a 32-bit signed integer, which is
not allowed to be negative. But doing arithmetic with them can lead to
overflows producing undefined values, including negative ones.

Use size_t instead for these image data size calculations. It's defined
as an unsigned integer large enough to fit valid memory buffer sizes.
In practice that's 32-bit on 32-bit platforms, and 64-bit on 64-bit
ones, which is large enough to avoid overflow in the OutOfMemory test.

Bug: b/140421004
Change-Id: Id05dd64d470f579455fdc9925b0ab303ecaf37fb
Reviewed-on: https://swiftshader-review.googlesource.com/c/SwiftShader/+/36968
Tested-by: Nicolas Capens <nicolascapens@google.com>
Reviewed-by: Alexis Hétu <sugoi@google.com>