fix various bugs in new integer parser framework

1. my interpretation of subject sequence definition was wrong. adjust
parser to conform to the standard.

2. some code for handling tail overflow case was missing (forgot to
finish writing it).

3. typo (= instead of ==) caused ERANGE to wrongly behave like EINVAL