View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0022310FPCFCLpublic2012-06-24 20:152012-07-12 04:17
ReporterLuiz Americo 
Assigned ToMichael Van Canneyt 
PrioritynormalSeverityminorReproducibilityhave not tried
StatusclosedResolutionfixed 
PlatformOSOS Version
Product VersionProduct Build 
Target Version2.6.1Fixed in Version2.7.1 
Summary0022310: Wrong conversion of unicode code points in fpjsonscanner
DescriptionHi, converting unicode code points out of basic latin, like CJK, leads to wrong encoded value.

See the attached example. It should output the below ( see http://www.fileformat.info/info/unicode/char/4E01/index.htm [^] )

E4
B8
81

The problem was pointed by Reinier Olislagers. The suggested fix by him fix the problem. See below

Then, adapted line jsonscannerutf8.pp line 244 (FPC 2.7.1 jsonscanner.pp
line 238/239), function DoFetchToken
to convert incoming json string data to UTF8 instead of system codepage:
e.g.
from:
// Takes care of conversion...
S:=WideChar(StrToInt('$'+S));
to:
// Convert from Unicode codepoint in hex to UTF8... via UTF16:
S:=Utf8Encode(WideString(WideChar(StrToInt('$'+S))));
TagsNo tags attached.
FPCOldBugId0
Fixed in Revision21837
Attached Fileszip file icon JSONUnicode.zip [^] (587 bytes) 2012-06-24 20:15

- Relationships

-  Notes
(0060930)
Michael Van Canneyt (administrator)
2012-07-10 08:58

Changed it in a different way to allow for backward compatibility.
The constructor now has an extra argument 'UseUTF8'.
It is also a property of the scanner and parser.

Change your code to
    Parser := TJSONParser.Create(Stream,True);

And all will work well.

- Issue History
Date Modified Username Field Change
2012-06-24 20:15 Luiz Americo New Issue
2012-06-24 20:15 Luiz Americo File Added: JSONUnicode.zip
2012-06-24 20:15 Luiz Americo FPCOldBugId => 0
2012-06-26 19:02 Jonas Maebe Status new => assigned
2012-06-26 19:02 Jonas Maebe Assigned To => Michael Van Canneyt
2012-07-10 08:58 Michael Van Canneyt Fixed in Revision => 21837
2012-07-10 08:58 Michael Van Canneyt Status assigned => resolved
2012-07-10 08:58 Michael Van Canneyt Fixed in Version => 2.7.1
2012-07-10 08:58 Michael Van Canneyt Resolution open => fixed
2012-07-10 08:58 Michael Van Canneyt Note Added: 0060930
2012-07-10 08:58 Michael Van Canneyt Target Version => 2.6.1
2012-07-12 04:17 Luiz Americo Status resolved => closed



MantisBT 1.2.12[^]
Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker