math: fix long double ulperror calculation