fix incorrect overflow check for allocation in fmemopen

when a null buffer pointer is passed to fmemopen, requesting it
allocate its own memory buffer, extremely large size arguments near
SIZE_MAX could overflow and result in underallocation. this results
from omission of the size of the cookie structure in the overflow
check but inclusion of it in the calloc call.

instead of accounting for individual small contributions to the total
allocation size needed, simply reject sizes larger than PTRDIFF_MAX,
which will necessarily fail anyway. then adding arbitrary fixed-size
structures is safe without matching up the expressions in the
comparison and the allocation.

diff --git a/src/stdio/fmemopen.c b/src/stdio/fmemopen.c
index 7c193a5..2ce43d3 100644 (file)
--- a/src/stdio/fmemopen.c
+++ b/src/stdio/fmemopen.c
@@ -81,7 +81,7 @@ FILE *fmemopen(void *restrict buf, size_t size, const char *restrict mode)
                return 0;
        }
 
-       if (!buf && size > SIZE_MAX-sizeof(FILE)-BUFSIZ-UNGET) {
+       if (!buf && size > PTRDIFF_MAX) {
                errno = ENOMEM;
                return 0;
        }