Enterprise Client Community

Hi,

there seems to be a problem with values defined as double (format dec1).
The converting in class DGVColumnFilterFromTo of from and to value works but the value itself has a precision problem, e.g. 0.8 becomes sthg. like 8.0000000019797

I added a screenshot of the debugger!

By the way I'm using version 20221031

regards
Markus

We will check - thanks for the message!

...screenshot? I do not see any... ;-(
Please note: attachments are only allowed with a size < 100 kBytes in this forum.

Thanks! - Björn

Hi Markus,

...never use float/double when you want to have numeric precision ;-)

Code:

                 float f = 0.8f;
                 System.out.println("Float f: " + f);
                 double d1 = 0.8d;
                 System.out.println("Double d1: " + d1);
                 double d2= (double)f;
                 System.out.println("Double d2: " + d2);
 
 Result:
 
 Float f: 0.8
 Double d1: 0.8
 Double d2: 0.800000011920929

Unfortunately we internally use Java-class DecimalFormat and this class internally casts float values to double values, so is using exactly what is done with the d2-variale in the code above...

We now introduced a correction factor when comparing from-to in the grid of "0.0000002". The correction factor is only applied when using float or double - it is not applied when using BigDecimal.

Changes are applied in version 20221114 (just on its way to be released).

Thanks for your hint! Björn