mbrtowc: do not leave mbstate_t in permanent-fail state after EILSEQ the standard is clear that the old behavior is conforming: "In this case, [EILSEQ] shall be stored in errno and the conversion state is undefined." however, the specification of mbrtowc has one peculiarity when the source argument is a null pointer: in this case, it's required to behave as mbrtowc(NULL, "", 1, ps). no motivation is provided for this requirement, but the natural one that comes to mind is that the intent is to reset the mbstate_t object. for stateful encodings, such behavior is actually specified: "If the corresponding wide character is the null wide character, the resulting state described shall be the initial conversion state." but in the case of UTF-8 where the mbstate_t object contains a partially-decoded character rather than a shift state, a subsequent '\0' byte indicates that the previous partial character is incomplete and thus an illegal sequence. naturally, applications using their own mbstate_t object should clear it themselves after an error, but the standard presently provides no way to clear the builtin mbstate_t object used when the ps argument is a null pointer. I suspect this issue may be addressed in the future by specifying that a null source argument resets the state, as this seems to have been the intent all along. for what it's worth, this change also slightly reduces code size.
implement mbtowc directly, not as a wrapper for mbrtowc the interface contract for mbtowc admits a much faster implementation than mbrtowc can achieve; wrapping mbrtowc with an extra call frame only made the situation worse. since the regex implementation uses mbtowc already, this change should improve regex performance too. it may be possible to improve performance in other places internally by switching from mbrtowc to mbtowc.
optimize mbrtowc this simple change, in my measurements, makes about a 7% performance improvement. at first glance this change would seem like a compiler-specific hack, since the modified code is not even used. however, I suspect the reason is that I'm eliminating a second path into the main body of the code, allowing the compiler more flexibility to optimize the normal (hot) path into the main body. so even if it weren't for the measurable (and quite notable) difference in performance, I think the change makes sense.
fix out-of-bounds access in UTF-8 decoding SA and SB are used as the lowest and highest valid starter bytes, but the value of SB was one-past the last valid starter. this caused access past the end of the state table when the illegal byte '\xf5' was encountered in a starter position. the error did not show up in full-character decoding tests, since the bogus state read from just past the table was unlikely to admit any continuation bytes as valid, but would have shown up had we tested feeding '\xf5' to the byte-at-a-time decoding in mbrtowc: it would cause the funtion to wrongly return -2 rather than -1. I may eventually go back and remove all references to SA and SB, replacing them with the values; this would make the code more transparent, I think. the original motivation for using macros was to allow misguided users of the code to redefine them for the purpose of enlarging the set of accepted sequences past the end of Unicode...
overhaul mbsrtowcs these changes fix at least two bugs: - misaligned access to the input as uint32_t for vectorized ASCII test - incorrect src pointer after stopping on EILSEQ in addition, the text of the standard makes it unclear whether the mbstate_t object is to be modified when the destination pointer is null; previously it was cleared either way; now, it's only cleared when the destination is non-null. this change may need revisiting, but it should not affect most applications, since calling mbsrtowcs with non-zero state can only happen when the head of the string was already processed with mbrtowc. finally, these changes shave about 20% size off the function and seem to improve performance by 1-5%.
use restrict everywhere it's required by c99 and/or posix 2008 to deal with the fact that the public headers may be used with pre-c99 compilers, __restrict is used in place of restrict, and defined appropriately for any supported compiler. we also avoid the form [restrict] since older versions of gcc rejected it due to a bug in the original c99 standard, and instead use the form *restrict.
fix longstanding exit logic bugs in mbsnrtowcs and wcsnrtombs these are POSIX 2008 (previously GNU extension) functions that are rarely used. apparently they had never been tested before, since the end-of-string logic was completely missing. mbsnrtowcs is used by modern versions of bash for its glob implementation, and and this bug was causing tab completion to hang in an infinite loop.
cleanup and work around visibility bug in gcc 3 that affects x86_64 in gcc 3, the visibility attribute must be placed on both the declaration and on the definition. if it's omitted from the definition, the compiler fails to emit the ".hidden" directive in the assembly, and the linker will either generate textrels (if supported, such as on i386) or refuse to link (on targets where certain types of textrels are forbidden or impossible without further assumptions about memory layout, such as on x86_64). this patch also unifies the decision about when to use visibility into libc.h and makes the visibility in the utf-8 state machine tables based on libc.h rather than a duplicate test.
fix all implicit conversion between signed/unsigned pointers sadly the C language does not specify any such implicit conversion, so this is not a matter of just fixing warnings (as gcc treats it) but actual errors. i would like to revisit a number of these changes and possibly revise the types used to reduce the number of casts required.
cleanup utf-8 multibyte code, use visibility if possible this code was written independently of musl, with support for a the backwards, nonstandard "31-bit unicode" some libraries/apps might want. unfortunately the extra code (inside #ifdef) makes the source harder to read and makes code that should be simple look complex, so i'm removing it. anyone who wants to use the old code can find it in the history or from elsewhere. also, change the visibility of the __fsmu8 state machine table to hidden, if supported. this should improve performance slightly in shared-library builds.