View Issue Details

IDProjectCategoryView StatusLast Update
0031445FPCDatabasepublic2017-03-09 22:26
ReporterArtem Proskurnev Assigned ToLacaK  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
OSLinux 
Product Version3.1.1 
Fixed in Version3.1.1 
Summary0031445: Field size not correct for UTF8
DescriptionWhen the database returns UTF8 string of Russian letters, they calculated wrong and the letters are cut off. For example, the database field is specified the size of a single symbol and there is the Russian letter "М" or "Ж". Lazarus creates a field size of 1 and half the letter is cut off, because it actually is two characters. Temporarily I did to fix this issue (Very bad. I known):
packages/fcl-db/src/base/fields.inc
constructor TFieldDef.Create(AOwner: TFieldDefs; const AName: string;
  ADataType: TFieldType; ASize: Integer; ARequired: Boolean; AFieldNo: Longint;
  ACodePage: TSystemCodePage);

begin
{$ifdef dsdebug }
  Writeln('TFieldDef.Create : ',Aname,'(',AFieldNo,')');
{$endif}
  Inherited Create(AOwner);
  Name:=Aname;
  FDatatype:=ADatatype;
  FSize:=ASize;
  FRequired:=ARequired;
  FPrecision:=-1;
  FFieldNo:=AFieldNo;
  case FDataType of
    ftString, ftFixedChar, ftMemo:
//===============================================
    begin
      FCodePage := CP_UTF8;
      FSize:=ASize*2;
    end;
//===============================================
    ftWideString, ftFixedWideChar, ftWideMemo:
      FCodePage := CP_UTF16;
    else
      FCodePage := 0;
  end;
end;
TagsNo tags attached.
Fixed in Revision35544
FPCOldBugId
FPCTarget
Attached Files

Activities

Artem Proskurnev

2017-02-25 12:48

reporter  

s1.tar.gz (22,433 bytes)

Artem Proskurnev

2017-02-25 13:05

reporter   ~0098452

Base MySQL MariaDB

Thaddy de Koning

2017-02-25 13:56

reporter   ~0098453

Last edited: 2017-02-25 14:01

View 2 revisions

This piece of your code looks wrong:
//===============================================
    begin
      FCodePage := CP_UTF8;
      FSize:=ASize*2;
    end;
It should be ASize * 4 for UTF8 (the maximum codepoint size in UTF8 is 4! not 2)

Artem Proskurnev

2017-02-25 16:30

reporter   ~0098457

Thank you :-)

Thaddy de Koning

2017-02-25 20:19

reporter   ~0098461

Last edited: 2017-02-25 20:21

View 2 revisions

Solved? ;-)

Artem Proskurnev

2017-02-26 02:57

reporter   ~0098465

Last edited: 2017-02-26 02:58

View 2 revisions

I'm afraid that this problem will appear in lazarus version 1.8 and next version fpc :-(

Thaddy de Koning

2017-02-26 11:05

reporter   ~0098469

Last edited: 2017-02-26 11:10

View 3 revisions

I am almost sure that Lacak fixed this already in trunk a couple of months ago for UTF8 field lengths in Databases and database interfaces like TDataset. They use four bytes per char to ensure they can be read/written correctly for any given UTF8 char as reserved storage space.
It may be a Lazarus issue?

Lacak?

Thaddy de Koning

2017-02-26 13:07

reporter   ~0098472

Last edited: 2017-02-26 13:36

View 5 revisions

0031078. It is resolved by Lacak. See also 0017376.

{$ifdef dsdebug }
  Writeln('TFieldDef.Create : ',Aname,'(',AFieldNo,')');
{$endif}
  Inherited Create(AOwner);
  Name:=Aname;
  FDatatype:=ADatatype;
  FSize:=ASize;
  FRequired:=ARequired;
  FPrecision:=-1;
  FFieldNo:=AFieldNo;
  case FDataType of
    ftString, ftFixedChar, ftMemo:
      FCodePage := ACodePage;
    ftWideString, ftFixedWideChar, ftWideMemo:
      FCodePage := CP_UTF16;
    else
      FCodePage := 0;
  end;
end;

You are testing with old code. All occurrences have been corrected to 4 bytes.
Plz ask to close...

LacaK

2017-02-27 08:12

developer   ~0098485

Please be aware of that "Size" is display (character) size (in your case 1), while "DataSize" is byte size (in your case 4).
As Thaddy say try current trunk and report back ...

Artem Proskurnev

2017-03-01 21:00

reporter  

a1.png (8,879 bytes)   
a1.png (8,879 bytes)   

Artem Proskurnev

2017-03-01 21:01

reporter   ~0098555

Screenshot today trunc.
UTF8
p=varchar(1)
a=varchar(4)

Artem Proskurnev

2017-03-01 21:11

reporter  

a2.png (114,247 bytes)   
a2.png (114,247 bytes)   

Artem Proskurnev

2017-03-01 21:11

reporter  

a3.png (38,820 bytes)   
a3.png (38,820 bytes)   

Artem Proskurnev

2017-03-01 21:11

reporter  

a5.png (73,804 bytes)   
a5.png (73,804 bytes)   

Artem Proskurnev

2017-03-01 21:12

reporter  

a4.png (73,731 bytes)   
a4.png (73,731 bytes)   

Artem Proskurnev

2017-03-01 21:12

reporter  

a6.png (55,702 bytes)   
a6.png (55,702 bytes)   

Artem Proskurnev

2017-03-01 21:16

reporter   ~0098556

Screenshot s34 - lazarus 1.6.2 fpc 3.0.0 all ok
cpol=varchar(1)

Artem Proskurnev

2017-03-01 21:16

reporter  

s34.png (146,215 bytes)   
s34.png (146,215 bytes)   

LacaK

2017-03-02 08:06

developer   ~0098557

Can you please prepare test case with:
1. SQL script with CREATE TABLE statement + INSERT test values
2. Small compilable project which will show errors

Artem Proskurnev

2017-03-02 09:26

reporter   ~0098558

CREATE TABLE `test`.`new_table` (
  `idnew_table` INT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT '',
  `a` VARCHAR(1) CHARACTER SET 'utf8' COLLATE 'utf8_general_ci' NULL COMMENT '',
  `b` VARCHAR(4) CHARACTER SET 'utf8' COLLATE 'utf8_general_ci' NULL COMMENT '',
  PRIMARY KEY (`idnew_table`) COMMENT '');

INSERT INTO `test`.`new_table` (`a`, `b`) VALUES ('М', 'ЙЦУК');
INSERT INTO `test`.`new_table` (`a`, `b`) VALUES ('Ж', 'ЯЁЭ');

Artem Proskurnev

2017-03-02 09:27

reporter  

test.tar.gz (129,220 bytes)

Artem Proskurnev

2017-03-02 09:31

reporter   ~0098559

Trunc from:
http://svn.freepascal.org/svn/lazarus/trunk
and
http://svn.freepascal.org/svn/fpc/trunk

Zeljan Rikalo

2017-03-02 10:15

reporter   ~0098560

@Artem, if your db is UTF-8 varchar(1) is insufficient for 'Ж'. Mysql (mariadb) does not raise exception if insert is bigger than field, and in return you have only one char. eg Postgresql raises exception in this case.
eg. inserting croatian latin letter Ž into postgresql utf-8 db varchar(1) raises error.

Artem Proskurnev

2017-03-02 14:52

reporter   ~0098566

Last edited: 2017-03-02 14:57

View 2 revisions

@Zeljan Rikalo
Screenshot s34 - lazarus 1.6.2 fpc 3.0.0 all ok
cpol=varchar(1)
Mysql (mariadb)
Screenshot a4 - fpc 3.1.1 (lazarus 1.6.2 or 1.7) bad
p=varchar(1)
Mysql (mariadb)

LacaK

2017-03-09 10:33

developer  

bug31445.lpr (1,799 bytes)

LacaK

2017-03-09 10:37

developer   ~0098776

Please test and close if okay

Artem Proskurnev

2017-03-09 22:26

reporter   ~0098783

Great! All work! :-)

Issue History

Date Modified Username Field Change
2017-02-25 12:48 Artem Proskurnev New Issue
2017-02-25 12:48 Artem Proskurnev File Added: s1.tar.gz
2017-02-25 13:05 Artem Proskurnev Note Added: 0098452
2017-02-25 13:56 Thaddy de Koning Note Added: 0098453
2017-02-25 14:01 Thaddy de Koning Note Edited: 0098453 View Revisions
2017-02-25 16:30 Artem Proskurnev Note Added: 0098457
2017-02-25 20:19 Thaddy de Koning Note Added: 0098461
2017-02-25 20:21 Thaddy de Koning Note Edited: 0098461 View Revisions
2017-02-26 02:57 Artem Proskurnev Note Added: 0098465
2017-02-26 02:58 Artem Proskurnev Note Edited: 0098465 View Revisions
2017-02-26 11:05 Thaddy de Koning Note Added: 0098469
2017-02-26 11:08 Thaddy de Koning Note Edited: 0098469 View Revisions
2017-02-26 11:10 Thaddy de Koning Note Edited: 0098469 View Revisions
2017-02-26 13:07 Thaddy de Koning Note Added: 0098472
2017-02-26 13:09 Thaddy de Koning Note Edited: 0098472 View Revisions
2017-02-26 13:17 Thaddy de Koning Note Edited: 0098472 View Revisions
2017-02-26 13:18 Thaddy de Koning Note Edited: 0098472 View Revisions
2017-02-26 13:36 Thaddy de Koning Note Edited: 0098472 View Revisions
2017-02-27 08:03 LacaK Assigned To => LacaK
2017-02-27 08:03 LacaK Status new => assigned
2017-02-27 08:12 LacaK Note Added: 0098485
2017-02-27 08:12 LacaK Status assigned => feedback
2017-03-01 21:00 Artem Proskurnev File Added: a1.png
2017-03-01 21:01 Artem Proskurnev Note Added: 0098555
2017-03-01 21:01 Artem Proskurnev Status feedback => assigned
2017-03-01 21:11 Artem Proskurnev File Added: a2.png
2017-03-01 21:11 Artem Proskurnev File Added: a3.png
2017-03-01 21:11 Artem Proskurnev File Added: a5.png
2017-03-01 21:12 Artem Proskurnev File Added: a4.png
2017-03-01 21:12 Artem Proskurnev File Added: a6.png
2017-03-01 21:16 Artem Proskurnev Note Added: 0098556
2017-03-01 21:16 Artem Proskurnev File Added: s34.png
2017-03-02 08:06 LacaK Note Added: 0098557
2017-03-02 08:06 LacaK Status assigned => feedback
2017-03-02 09:26 Artem Proskurnev Note Added: 0098558
2017-03-02 09:26 Artem Proskurnev Status feedback => assigned
2017-03-02 09:27 Artem Proskurnev File Added: test.tar.gz
2017-03-02 09:31 Artem Proskurnev Note Added: 0098559
2017-03-02 10:15 Zeljan Rikalo Note Added: 0098560
2017-03-02 14:52 Artem Proskurnev Note Added: 0098566
2017-03-02 14:57 Artem Proskurnev Note Edited: 0098566 View Revisions
2017-03-09 10:33 LacaK File Added: bug31445.lpr
2017-03-09 10:37 LacaK Fixed in Revision => 35544
2017-03-09 10:37 LacaK Note Added: 0098776
2017-03-09 10:37 LacaK Status assigned => resolved
2017-03-09 10:37 LacaK Fixed in Version => 3.1.1
2017-03-09 10:37 LacaK Resolution open => fixed
2017-03-09 22:26 Artem Proskurnev Note Added: 0098783
2017-03-09 22:26 Artem Proskurnev Status resolved => closed