Incorrect calculation

calc issue

When comparing 2 numbers (as per the attached screenshot) 153.9 vs (153.89+0.01), studio claims the first one is higher than the second, which is nonsense, as they are equal.

Can you add greater than or equal to, see if that makes a difference?

Hi,

It’s Double type accuracy matter.
For now, can you check the following?

In Double type, 153.9 is precisely 153.90000000000001,
and 153.89+0.01 is preciously 153.89999999999998, because of its accuracy.

We can check it using G specifier.

If we need to prevent this matter, we can use Decimal type.

image

The above D is literal for Decimal. Please see the following in details.

Regards,

I believe Math.round(153.9,2) > math.round(math.round(153.89,2)+0.01,2) leads to same result.
Both 153.9 and 153.89 come into my formula as in arguments, so the above can be easier to implement, as opposed to adding ‘D’ after argument name, am I right?

It doesn’t as it is ‘or’ so one condition is enough to show True.

Hi,

Yes, it will be also good because Round method remove difference of 16 or 17th decimal place value.

Regards,

1 Like

Thank you Yoichi.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.