fix integer overflows and uncaught EOVERFLOW in printf core

this patch fixes a large number of missed internal signed-overflow
checks and errors in determining when the return value (output length)
would exceed INT_MAX, which should result in EOVERFLOW. some of the
issues fixed were reported by Alexander Cherepanov; others were found
in subsequent review of the code.

aside from the signed overflows being undefined behavior, the
following specific bugs were found to exist in practice:

- overflows computing length of floating point formats with huge
  explicit precisions, integer formats with prefix characters and huge
  explicit precisions, or string arguments or format strings longer
  than INT_MAX, resulted in wrong return value and wrong %n results.

- literal width and precision values outside the range of int were
  misinterpreted, yielding wrong behavior in at least one well-defined
  case: string formats with precision greater than INT_MAX were
  sometimes truncated.

- in cases where EOVERFLOW is produced, incorrect values could be
  written for %n specifiers past the point of exceeding INT_MAX.

in addition to fixing these bugs, we now stop producing output
immediately when output length would exceed INT_MAX, rather than
continuing and returning an error only at the end.