View Issue Details

IDProjectCategoryView StatusLast Update
0038968FPCRTLpublic2021-06-16 10:07
Reporterwp Assigned ToMichael Van Canneyt  
PrioritynormalSeverityminorReproducibilityalways
Status assignedResolutionopen 
Fixed in Version3.3.1 
Summary0038968: ISO8601ToDate issue with fractional seconds
DescriptionThe function ISO8601ToDate of unit DateUtils converts an ISO8601 formatted date/time string to a TDateTime variable. According to specification (https://en.wikipedia.org/wiki/ISO_8601), "there is no limit on the number of decimal places for the decimal fraction" and the decimal mark can be "either a comma or a dot".

However, FPC accepts only 3 decimals places, and the decimal separator can only be a dot. Otherwise the function raises an exception.
Delphi accepts any count of decimals, but the decimal separatur must be a dot as well.

The attached patch fixes the issue.
Steps To ReproduceRun the attached demo. It varies the number of decimal places and switches the decimal separator between dot and comma. If successful the converted TDateTime number is displayed, otherwise the word "ERROR".

Unpatched FPC displays "ERROR" except for the case with three decimal places and decimal dot. Delphi XE10.3 displays "ERROR" only in the decimal comma cases.

After applying the patch "ERROR" is not displayed any more.
Additional InformationSee also forum discussion https://forum.lazarus.freepascal.org/index.php/topic,54890.0.html

According to the cited article it is also possible to specify date and time strings without the date and time separators (i.e. '20210606' instead of '2021-06-06', or '1310' instead of '13:10') - these variants will crash the conversion, too, but are not covered by the present patch.
TagsNo tags attached.
Fixed in Revision49485
FPCOldBugId
FPCTarget4.0.0
Attached Files

Activities

wp

2021-06-06 11:20

reporter  

dateutil.inc.patch (2,480 bytes)   
Index: packages/rtl-objpas/src/inc/dateutil.inc
===================================================================
--- packages/rtl-objpas/src/inc/dateutil.inc	(revision 49000)
+++ packages/rtl-objpas/src/inc/dateutil.inc	(working copy)
@@ -2769,7 +2769,9 @@
 
 function TryISOStrToTime(const aString: string; Out outTime: TDateTime): Boolean;
 var
-  xHour, xMinute, xSecond, xMillisecond, xLength: LongInt;
+  xHour, xMinute, xSecond, xLength, res: LongInt;
+  xFractionalSecond: Extended;
+  tmp: String;
 begin
   Result := True;
   xLength := Length(aString);
@@ -2829,24 +2831,31 @@
           (aString[6] = ':') and
           TryStrToInt(Copy(aString, 7, 2), xSecond) and
           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
-    10: Result :=
-          TryStrToInt(Copy(aString, 1, 2), xHour) and
-          TryStrToInt(Copy(aString, 3, 2), xMinute) and
-          TryStrToInt(Copy(aString, 5, 2), xSecond) and
-          (aString[7] = '.') and
-          TryStrToInt(Copy(aString, 8, 3), xMillisecond) and
-          TryEncodeTime(xHour, xMinute, xSecond, xMillisecond, outTime);
-    12: Result :=
-          TryStrToInt(Copy(aString, 1, 2), xHour) and
-          (aString[3] = ':') and
-          TryStrToInt(Copy(aString, 4, 2), xMinute) and
-          (aString[6] = ':') and
-          TryStrToInt(Copy(aString, 7, 2), xSecond) and
-          (aString[9] = '.') and
-          TryStrToInt(Copy(aString, 10, 3), xMillisecond) and
-          TryEncodeTime(xHour, xMinute, xSecond, xMillisecond, outTime);
-  else
-    Result := False;
+    else
+        if xLength >= 9 then
+        begin
+          Result := 
+            TryStrToInt(Copy(aString, 1, 2), xHour) and
+            (aString[3] = ':') and
+            TryStrToInt(Copy(aString, 4, 2), xMinute) and
+            (aString[6] = ':') and
+            TryStrToInt(Copy(aString, 7, 2), xSecond) and
+            ((aString[9] = '.') or (aString[9] = ',')) and
+            TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
+          if Result then
+          begin
+            tmp := Copy(aString, 9, xLength-8);
+            if tmp <> '' then
+            begin
+              tmp[1] := '.';
+              val(tmp, xFractionalSecond, res);
+              Result := res = 0;
+              if Result then
+                outTime := outTime + xFractionalSecond * OneSecond;
+            end;
+          end;
+        end else
+          Result := false;
   end;
 
   if not Result then
dateutil.inc.patch (2,480 bytes)   

Michael Van Canneyt

2021-06-06 12:50

administrator   ~0131183

Applied patch, added test in testsuite. Thank you very much !

wp

2021-06-06 14:00

reporter   ~0131184

Last edited: 2021-06-06 14:01

View 2 revisions

Thanks - that was fast. But why FPC-Target 4.0.0? It clearly is a bug and should go into fixes, v3.2.4.

wp

2021-06-08 17:52

reporter   ~0131204

Last edited: 2021-06-08 22:51

View 2 revisions

Sorry - but my patch broke the case without separators which was working correctly in Laz 3.2.

The new patch works correctly (*) with and without date/time separators - see new test program.

(*) There are still issues in some cases not related to the patched TryISOStrToTime(): Strings with a date part only, or with a date part and a time part containing only hours are not detected.
38968_dateutil.inc-v2.patch (3,011 bytes)   
Index: packages/rtl-objpas/src/inc/dateutil.inc
===================================================================
--- packages/rtl-objpas/src/inc/dateutil.inc	(revision 49492)
+++ packages/rtl-objpas/src/inc/dateutil.inc	(working copy)
@@ -2824,38 +2824,40 @@
           TryStrToInt(Copy(aString, 3, 2), xMinute) and
           TryStrToInt(Copy(aString, 5, 2), xSecond) and
           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
-    8: Result :=
-          TryStrToInt(Copy(aString, 1, 2), xHour) and
-          (aString[3] = ':') and
-          TryStrToInt(Copy(aString, 4, 2), xMinute) and
-          (aString[6] = ':') and
-          TryStrToInt(Copy(aString, 7, 2), xSecond) and
-          TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
     else
-        if xLength >= 9 then
-        begin
-          Result := 
-            TryStrToInt(Copy(aString, 1, 2), xHour) and
-            (aString[3] = ':') and
-            TryStrToInt(Copy(aString, 4, 2), xMinute) and
-            (aString[6] = ':') and
-            TryStrToInt(Copy(aString, 7, 2), xSecond) and
-            ((aString[9] = '.') or (aString[9] = ',')) and
-            TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
-          if Result then
-          begin
-            tmp := Copy(aString, 9, xLength-8);
-            if tmp <> '' then
-            begin
-              tmp[1] := '.';
-              val(tmp, xFractionalSecond, res);
-              Result := res = 0;
-              if Result then
-                outTime := outTime + xFractionalSecond * OneSecond;
-            end;
-          end;
-        end else
-          Result := false;
+       if (xLength >= 8) and (aString[3] = ':') and (aString[6] = ':') then
+       begin
+         Result :=
+           TryStrToInt(Copy(aString, 1, 2), xHour) and
+           TryStrToInt(Copy(aString, 4, 2), xMinute) and
+           TryStrToInt(Copy(aString, 7, 2), xSecond) and
+           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
+         if Result and (xLength >= 9) then
+         begin
+           tmp := copy(aString, 10, xLength-9);
+           val('.' + tmp, xFractionalSecond, res);
+           Result := (res = 0);
+           if Result then
+             outTime := outTime + xFractionalSecond * OneSecond;
+         end;
+       end else
+       if (xLength >= 7) and (aString[7] in ['.', ',']) then
+       begin
+         Result :=
+           TryStrToInt(Copy(aString, 1, 2), xHour) and
+           TryStrToInt(Copy(aString, 3, 2), xMinute) and
+           TryStrToInt(Copy(aString, 5, 2), xSecond) and
+           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
+         tmp := copy(aString, 8, xLength-7);
+         if Result and (tmp <> '') then
+         begin
+           val('.'+tmp, xFractionalSecond, res);
+           Result := res = 0;
+           if Result then
+             outTime := outTime + xFractionalSecond * OneSecond;
+         end;
+       end else
+         Result := false;
   end;
 
   if not Result then
38968_dateutil.inc-v2.patch (3,011 bytes)   
38968 test2.zip (1,335 bytes)

wp

2021-06-09 13:05

reporter   ~0131214

This is the last patch of this series (hopefully...). It now covers all the cases that came up to my mind - see attached test project. (Please ignore the patch -v2 of note 0141204.).
38968-dateutil.inc-v3.patch (7,136 bytes)   
Index: packages/rtl-objpas/src/inc/dateutil.inc
===================================================================
--- packages/rtl-objpas/src/inc/dateutil.inc	(revision 49492)
+++ packages/rtl-objpas/src/inc/dateutil.inc	(working copy)
@@ -2749,12 +2749,23 @@
   xYear, xMonth, xDay: LongInt;
 begin
   case Length(aString) of
-    8: Result :=
+    4: Result :=                                        // YYYY
+          TryStrToInt(aString, xYear) and
+          TryEncodeDate(xYear, 1, 1, outDate);
+    6: Result :=                                        // YYYYMM
           TryStrToInt(Copy(aString, 1, 4), xYear) and
           TryStrToInt(Copy(aString, 5, 2), xMonth) and
+          TryEncodeDate(xYear, xMonth, 1, outDate);
+    7: Result :=                                        // YYYY-MM
+          TryStrToInt(Copy(aString, 1, 4), xYear) and
+          TryStrToInt(Copy(aString, 6, 2), xMonth) and
+          TryEncodeDate(xYear, xMonth, 1, outDate);
+    8: Result :=                                        // YYYYMMDD
+          TryStrToInt(Copy(aString, 1, 4), xYear) and
+          TryStrToInt(Copy(aString, 5, 2), xMonth) and
           TryStrToInt(Copy(aString, 7, 2), xDay) and
           TryEncodeDate(xYear, xMonth, xDay, outDate);
-    10: Result :=
+    10: Result :=                                       //YYYY-MM-DD
           TryStrToInt(Copy(aString, 1, 4), xYear) and
           TryStrToInt(Copy(aString, 6, 2), xMonth) and
           TryStrToInt(Copy(aString, 9, 2), xDay) and
@@ -2807,55 +2818,57 @@
   end;
 
   case xLength of
-    2: Result :=
+    2: Result :=                                          // HH
           TryStrToInt(aString, xHour) and
           TryEncodeTime(xHour, 0, 0, 0, outTime);
-    4: Result :=
+    4: Result :=                                          // HHNN
           TryStrToInt(Copy(aString, 1, 2), xHour) and
           TryStrToInt(Copy(aString, 3, 2), xMinute) and
           TryEncodeTime(xHour, xMinute, 0, 0, outTime);
-    5: Result :=
+    5: Result :=                                          // HH:NN
           TryStrToInt(Copy(aString, 1, 2), xHour) and
           (aString[3] = ':') and
           TryStrToInt(Copy(aString, 4, 2), xMinute) and
           TryEncodeTime(xHour, xMinute, 0, 0, outTime);
-    6: Result :=
+    6: Result :=                                          // HHNNSS
           TryStrToInt(Copy(aString, 1, 2), xHour) and
           TryStrToInt(Copy(aString, 3, 2), xMinute) and
           TryStrToInt(Copy(aString, 5, 2), xSecond) and
           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
-    8: Result :=
-          TryStrToInt(Copy(aString, 1, 2), xHour) and
-          (aString[3] = ':') and
-          TryStrToInt(Copy(aString, 4, 2), xMinute) and
-          (aString[6] = ':') and
-          TryStrToInt(Copy(aString, 7, 2), xSecond) and
-          TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
     else
-        if xLength >= 9 then
-        begin
-          Result := 
-            TryStrToInt(Copy(aString, 1, 2), xHour) and
-            (aString[3] = ':') and
-            TryStrToInt(Copy(aString, 4, 2), xMinute) and
-            (aString[6] = ':') and
-            TryStrToInt(Copy(aString, 7, 2), xSecond) and
-            ((aString[9] = '.') or (aString[9] = ',')) and
-            TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
-          if Result then
-          begin
-            tmp := Copy(aString, 9, xLength-8);
-            if tmp <> '' then
-            begin
-              tmp[1] := '.';
-              val(tmp, xFractionalSecond, res);
-              Result := res = 0;
-              if Result then
-                outTime := outTime + xFractionalSecond * OneSecond;
-            end;
-          end;
-        end else
-          Result := false;
+       if (xLength >= 8) and (aString[3] = ':') and (aString[6] = ':') then
+       begin
+         Result :=                            // HH:NN:SS
+           TryStrToInt(Copy(aString, 1, 2), xHour) and
+           TryStrToInt(Copy(aString, 4, 2), xMinute) and
+           TryStrToInt(Copy(aString, 7, 2), xSecond) and
+           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
+         if Result and (xLength >= 9) then    // HH:NN:SS.[z] (0 or several z)
+         begin
+           tmp := copy(aString, 10, xLength-9);
+           val('.' + tmp, xFractionalSecond, res);
+           Result := (res = 0);
+           if Result then
+             outTime := outTime + xFractionalSecond * OneSecond;
+         end;
+       end else
+       if (xLength >= 7) and (aString[7] in ['.', ',']) then
+       begin
+         Result :=                         // HHNNSS
+           TryStrToInt(Copy(aString, 1, 2), xHour) and
+           TryStrToInt(Copy(aString, 3, 2), xMinute) and
+           TryStrToInt(Copy(aString, 5, 2), xSecond) and
+           TryEncodeTime(xHour, xMinute, xSecond, 0, outTime);
+         tmp := copy(aString, 8, xLength-7);
+         if Result and (tmp <> '') then
+         begin                           // HHNNSS.[z] (0 or several z)
+           val('.'+tmp, xFractionalSecond, res);
+           Result := res = 0;
+           if Result then
+             outTime := outTime + xFractionalSecond * OneSecond;
+         end;
+       end else
+         Result := false;
   end;
 
   if not Result then
@@ -2870,12 +2883,27 @@
 
 begin
   xLength := Length(aString);
-  if (xLength>11) and CharInSet(aString[11], [' ', 'T']) then
+  if (xLength = 0) then
+    exit(false);
+
+  if (aString[1]='T') then
     begin
+    Result := TryISOStrToTime(copy(aString, 2, Length(aString)-1), outDateTime);
+    exit;
+  end;
+
+  if (xLength in [4 {YYYY}, 6 {YYYYMM}, 7 {YYYY-MM}, 8 {YYYYMMDD}, 10 {YYYY-MM-DD}]) then
+    begin
+    Result := TryISOStrToDate(aString, outDateTime);
+    exit;
+    end;
+
+  if (xLength>11) and CharInSet(aString[11], [' ', 'T']) then   // YYYY-MM-DDT...
+    begin
     sDate:=Copy(aString, 1, 10);
     sTime:=Copy(aString, 12, Length(aString))
     end
-  else if (xLength>9) and CharInSet(aString[9], [' ', 'T']) then
+  else if (xLength>9) and CharInSet(aString[9], [' ', 'T']) then    // YYYYMMDDT...
     begin
     sDate:=Copy(aString, 1, 8);
     sTime:=Copy(aString, 10, Length(aString));
@@ -2945,21 +2973,25 @@
     TZ:='Z';
     S:=Copy(S,1,L-1);
     end
-  else If (L>2) and (S[L-2] in ['+','-']) then
+  else if ((L>11) and ((S[11] in ['T',' ']) or (S[9] in ['T',' ']))) or // make sure that we dont't have date-only
+          (S[1]='T') then
+  begin
+    If (S[L-2] in ['+','-']) then
     begin
     TZ:=Copy(S,L-2,3);
     S:=Copy(S,1,L-3);
     end
-  else If (L>4) and (S[L-4] in ['+','-']) then
+  else If (S[L-4] in ['+','-']) then
     begin
     TZ:=Copy(S,L-4,5);
     S:=Copy(S,1,L-5);
     end
-  else If (L>5) and (S[L-5] in ['+','-']) then
+  else If (S[L-5] in ['+','-']) and ((L > 13) or (S[1]='T')) then  // do not confuse with '2021-05-21T13'
     begin
     TZ:=Copy(S,L-5,6);
     S:=Copy(S,1,L-6);
     end;
+  end;
   Result:=TryIsoStrToDateTime(S,aDateTime) and TryISOTZStrToTZOffset(TZ,TZOffset);
   if not Result then
     exit;
38968-dateutil.inc-v3.patch (7,136 bytes)   
38968 test3.zip (1,784 bytes)

wp

2021-06-16 09:52

reporter   ~0131327

Any problems with this one?

Michael Van Canneyt

2021-06-16 10:07

administrator   ~0131329

nono, simply a matter of time. FPC/laz Migration to gitlab takes a lot of time currently.
I will look at it later today.
Feel free to mail/remind me if you didn't see a close of this bug.

Issue History

Date Modified Username Field Change
2021-06-06 11:20 wp New Issue
2021-06-06 11:20 wp File Added: dateutil.inc.patch
2021-06-06 11:20 wp File Added: iso8601 fractional seconds.zip
2021-06-06 12:50 Michael Van Canneyt Assigned To => Michael Van Canneyt
2021-06-06 12:50 Michael Van Canneyt Status new => resolved
2021-06-06 12:50 Michael Van Canneyt Resolution open => fixed
2021-06-06 12:50 Michael Van Canneyt Fixed in Version => 3.3.1
2021-06-06 12:50 Michael Van Canneyt Fixed in Revision => 49485
2021-06-06 12:50 Michael Van Canneyt FPCTarget => 4.0.0
2021-06-06 12:50 Michael Van Canneyt Note Added: 0131183
2021-06-06 14:00 wp Status resolved => feedback
2021-06-06 14:00 wp Resolution fixed => open
2021-06-06 14:00 wp Note Added: 0131184
2021-06-06 14:01 wp Note Edited: 0131184 View Revisions
2021-06-08 17:52 wp Note Added: 0131204
2021-06-08 17:52 wp File Added: 38968_dateutil.inc-v2.patch
2021-06-08 17:52 wp File Added: 38968 test2.zip
2021-06-08 17:52 wp Status feedback => assigned
2021-06-08 22:51 wp Note Edited: 0131204 View Revisions
2021-06-09 13:05 wp Note Added: 0131214
2021-06-09 13:05 wp File Added: 38968-dateutil.inc-v3.patch
2021-06-09 13:05 wp File Added: 38968 test3.zip
2021-06-16 09:52 wp Note Added: 0131327
2021-06-16 10:07 Michael Van Canneyt Note Added: 0131329