incorrect values for float calculus