Quantcast
Channel: VBForums - Visual Basic 6 and Earlier
Viewing all articles
Browse latest Browse all 21273

[RESOLVED] Precision problem (single vs integer data types)

$
0
0
At a certain stage during program run, (single) variables A and B have values:

A = 4.95
B = 5

At these statement

VB Code:
  1. If B-A > 0.05 Then
  2.     Do This
  3. Else
  4.     Do That
  5. End If

B-A is 0.05 so 'That' should be done. Instead, 'This' is done. I checked the values of A and B and they were the above. However, the value of B-A turned out to be 0.050002 and that's why.

So in order to circumvent this problem I could work with integers, for example 100 times larger so the test would be:

If B-A > 5

for integer values A = 495 and B = 500.

When at a later stage the values are used for some intended purpose, they are simply multiplied by 0.01. But this would mean a great deal of changes in the code and I'd rather use some other trick if available and easy enough.

Viewing all articles
Browse latest Browse all 21273

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>