math: fix atanh (overflow and underflow issues)

math: fix atanh (overflow and underflow issues)

in atanh exception handling was left to the called log functions,
but the argument to those functions could underflow or overflow.

use double_t and float_t to avoid some useless stores on x86