math: fix rare underflow issue in fma