math: tgammal.c fixes

this is not a full rewrite just fixes to the special case logic:
+-0 and non-integer x<INT_MIN inputs incorrectly raised invalid
exception and for +-0 the return value was wrong

so integer test and odd/even test for negative inputs are changed
and a useless overflow test was removed