TDbf removes valid digits from float field for some files.
Original Reporter info from Mantis: wp @wpam
-
Reporter name:
Original Reporter info from Mantis: wp @wpam
- Reporter name:
Description:
This report is the result of a discussion in the forum https://forum.lazarus.freepascal.org/index.php/topic,54987.msg408763.html.
There exist legacy files in which float fields have an internal Size larger than expected by TDbf. Delphi & BDE refuse to read these files, LibreOffice Base as well as third-party dBase viewing tools, however, display them correctly. TDbf does open them but displays incorrect values.
I think this behaviour of TDbf is not acceptable. Rather than silently displaying a wrong value it should either reject the file completely or relax internal size requirements for file input and display the correct values.
Steps to reproduce:
File "SampleDbf.dbf" was presented in the above-mentioned forum discussion as an example of such files. It contains two float fields, the first one named "Amt2" is type "N(24,2)", i.e. a float field with Size 24 and 2 decimal places; the other one named "AMOUNT" has type "N(25,3)" with Size 25 and 3 decimals (is is not clear how the file was created). In the first record, "Amt2" contains the value 12345.67 but is displayed in a DBGrid as 1.2E3 (too small by 1 order of magnitude, accepting the loss in decimals the displayed value should be 1.2E4), and the field "AMOUNT" contains the value 201.14 but is displayed as 2 (too small by two orders of magnitude).
The attached demo analyzes the issue and compares various ways to extract the field values; field.AsFloat, field.AsString, field.DisplayText and field.Text. This is the output for the first record with the current version of TDbf in FPC-trunk:
amt2 AMOUNT
RecNo: 1
AsFloat: 1234.00000 2.00000
AsString: 1234 2
DisplayText: 1,2E3 2 <--- WRONG
EditText: 1,2E3 2 <--- WRONG
field.AsFloat and field.AsString do return the correct values, while field.DisplayText and field.Text return the wrong value as described. These two methods call TDbfFieldDef.CheckSizePrecision in unit dbf_fields. For the case of float fields the method truncates the Size to 20 if it is larger than that:
case FNativeFieldType of
....
'N','F':
begin
// floating point
if FSize < 1 then FSize := 1;
if FSize >= 20 then FSize := 20; // <--- here !!!!!
if FPrecision > FSize-2 then FPrecision := FSize-2;
if FPrecision < 0 then FPrecision := 0;
end;
This is problematic if for some reason the Size used in the file is larger than 20. Dbf stores field content as string and needs the Size value in order to advance from field to field for reading. When the currently used Size is different from the value used when writing the file the reading process gets out of sync and produces wrong results.
When I remove the line "if FSize >= 20 then FSize := 20;", the faulty file is read correctly, this is the output of the test program obtained with the patched file (apart from the exponential format used - but this is different issue):
amt2 AMOUNT
RecNo: 1
AsFloat: 12345.67000 201.14000
AsString: 12345,67 201,14
DisplayText: 1,2E4 201 <--- correct
EditText: 1,2E4 201 <--- correct
I also did not see a negative effect with other (non-faulty) files. Therefore, I'd propose to remove this line and thus give up the restriction that float fields in dbf files must not have a Size larger than 20. This fixes the issue.
Additional information:
The other option to raise an exception in such a case and to abort reading appears too harsh to me in view of the easy way to make the file readable.
Mantis conversion info:
- Mantis ID: 39009
- Fixed in version: 3.3.1
- Fixed in revision: 49507 (#f3e7d960)
- Target version: 4.0.0