fix integer overflow in float printf needed-precision computation

if the requested precision is close to INT_MAX, adding
LDBL_MANT_DIG/3+8 overflows. in practice the resulting undefined
behavior manifests as a large negative result, which is then used to
compute the new end pointer (z) with a wildly out-of-bounds value
(more overflow, more undefined behavior). the end result is at least
incorrect output and character count (return value); worse things do
not seem to happen, but detailed analysis has not been done.

this patch fixes the overflow by performing the intermediate
computation as unsigned; after division by 9, the final result
necessarily fits in int.

diff --git a/src/stdio/vfprintf.c b/src/stdio/vfprintf.c
index e439a07..cd17ad7 100644 (file)
--- a/src/stdio/vfprintf.c
+++ b/src/stdio/vfprintf.c
@@ -312,7 +312,7 @@ static int fmt_fp(FILE *f, long double y, int w, int p, int fl, int t)
        }
        while (e2<0) {
                uint32_t carry=0, *b;
-               int sh=MIN(9,-e2), need=1+(p+LDBL_MANT_DIG/3+8)/9;
+               int sh=MIN(9,-e2), need=1+(p+LDBL_MANT_DIG/3U+8)/9;
                for (d=a; d<z; d++) {
                        uint32_t rm = *d & (1<<sh)-1;
                        *d = (*d>>sh) + carry;