Prevent 32-bit numeric overflow on image allocation.
We assume the pixels of an image can be addressed with a signed 32-bit
offset, including any padding. For a 3D image it's possible to exceed
this without exceeding the per-dimension limits. Lowering the per-
dimension limit so the allocation is always less than 2 GiB makes them
unreasonably small, so instead we must check the total size.
Use 1 GiB as the soft limit in OpenGL.
Bug chromium:835299
Change-Id: I9c5184002c1710e3923b549f8c21e7f6a516e1c7
Reviewed-on: https://swiftshader-review.googlesource.com/18869
Reviewed-by: Nicolas Capens <nicolascapens@google.com>
Tested-by: Nicolas Capens <nicolascapens@google.com>
diff --git a/src/Renderer/Surface.cpp b/src/Renderer/Surface.cpp
index 788ca63..83412a3 100644
--- a/src/Renderer/Surface.cpp
+++ b/src/Renderer/Surface.cpp
@@ -2648,14 +2648,23 @@
size_t Surface::size(int width, int height, int depth, int border, int samples, Format format)
{
+ samples = max(1, samples);
+
switch(format)
{
default:
- // FIXME: Unpacking byte4 to short4 in the sampler currently involves reading 8 bytes,
- // and stencil operations also read 8 bytes per four 8-bit stencil values,
- // so we have to allocate 4 extra bytes to avoid buffer overruns.
- return (size_t)sliceB(width, height, border, format, true) * depth * samples + 4;
+ {
+ uint64_t size = (uint64_t)sliceB(width, height, border, format, true) * depth * samples;
+ // FIXME: Unpacking byte4 to short4 in the sampler currently involves reading 8 bytes,
+ // and stencil operations also read 8 bytes per four 8-bit stencil values,
+ // so we have to allocate 4 extra bytes to avoid buffer overruns.
+ size += 4;
+
+ // We can only sample buffers smaller than 2 GiB.
+ // Force an out-of-memory if larger, or let the caller report an error.
+ return size < 0x80000000u ? (size_t)size : std::numeric_limits<size_t>::max();
+ }
case FORMAT_YV12_BT601:
case FORMAT_YV12_BT709:
case FORMAT_YV12_JFIF:
