fix incorrect overflow check for allocation in fmemopen