View Issue Details

IDProjectCategoryView StatusLast Update
0034824LazarusFCLpublic2020-05-14 12:38
ReporterDenis Golovan Assigned ToMattias Gaertner  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionduplicate 
Product Version2.1 (SVN) 
Summary0034824: Code navigation fails with generics
DescriptionHi

Lazarus code navigation stops on following code.
Error in "Messages" window is:
Error: expected >, but TVI found

...

type
  generic TVI<T> = record
    v:T;
    ix:NativeInt;
    class operator > (const a:TVI; const b:TVI): boolean;inline;
    class operator = (const a:TVI; const b:TVI): boolean;inline;
    class operator <= (const a:TVI; const b:TVI): boolean;inline;
    class operator >= (const a:TVI; const b:TVI): boolean;inline;
  end;
  generic TVISet<T> = class(specialize TheBTreeSet<specialize TVI<T>>);
...

Looks like last line is not correctly parsed (but compiled ok).
Additional InformationTested under Linux x64, Lazarus svn. rev 59970
TagsNo tags attached.
Fixed in Revisionr62693, r63137, 63148, 63149.
LazTarget-
WidgetsetGTK 2
Attached Files

Relationships

duplicate of 0033788 resolvedJuha Manninen codetools disabled (generics) 
related to 0023903 closedMattias Gaertner Complete code error 
related to 0036494 resolvedJuha Manninen codetools - issues with generics when it is used for a record 

Activities

Denis Golovan

2019-06-07 20:52

reporter   ~0116615

Still fails.
Tested in Lazarus svn. 61340

Juha Manninen

2019-12-30 14:24

developer   ~0120151

Last edited: 2019-12-30 15:05

View 2 revisions

Denis, your code does not compile. I tested with FPC trunk from ~ 2 months ago, using {$mode objfpc}{$H+}.
Does it compile with some other compiler version?
The message is:
 unit1.pas(44,5) Fatal: Syntax error, "END" expected but "CLASS" found

Please provide compilable code, then we can look at codetools problems.

Denis Golovan

2019-12-30 14:37

reporter   ~0120153

Last edited: 2019-12-30 14:49

View 2 revisions

Small example attached.
Tested under Lazarus svn rev.61603 and FPC svn rev.42987

Juha Manninen

2019-12-30 15:24

developer   ~0120155

Last edited: 2019-12-31 12:08

View 3 revisions

Ok, {$modeswitch advancedrecords} was the essential thing. Now it compiles and I get the same error from codetools.
I must detach myself. I don't know how to fix this one.

[Edit]
The syntax is strange. Why does FPC accept it? There are 2 nested "specialize" clauses but the inner one "specialize TVI<T>" does not really specialize anything. It still has the "T" there.
Maybe I just don't understand generics enough...

soerensen3

2020-03-02 22:31

reporter   ~0121324

Last edited: 2020-03-02 22:34

View 2 revisions

generic TVISet<T> = class(specialize TheBTreeSet<specialize TVI<T>>);
@Juha Manninen
if you specialized the generic above the nested generic is also specialized and resolves to this:
specialize TVISet< Integer >;
resolves to
class(specialize TheBTreeSet<specialize TVI<Integer>>);

I think the error is in PascalParserTool somewhere in lines 6070+, after checking for unit name.
Maybe it needs to check for '<' after the identifier or specialize keyword as first identifier (depending on mode delphi or objfpc). There it might be necessary to recursively call TPascalParserTool.ReadSpecialize again. Sorry I don't understand CodeTools enough to make the change myself.

Please note that in mode objfpc the optional unit identifier comes before the specialize keyword.
In Mode Delphi the nested specialization goes like this:
type TypeName = UnitNameCanBeDotted.TypeName < UnitNameCanBeDotted.TypeName < UnitNameCanBeDotted.TypeName >>;
In Mode Objfpc the equivalent would be:
type TypeName = UnitNameCanBeDotted. specialize TypeName < UnitNameCanBeDotted. specialize TypeName < UnitNameCanBeDotted.TypeName >>;

I would be very happy if this was fixed! Thanks in advance!

soerensen3

2020-03-03 00:01

reporter   ~0121329

It seems that the attached changes fixed the problem for me.

Explanation: A generic specialization parameter is basically a type reference so no need to reinvent the wheel.

I did not notice any negative side effects yet but please someone review this patch before pulling it into trunk.
fix-nested-generics-specialization.patch (774 bytes)   
Index: pascalparsertool.pas
===================================================================
--- pascalparsertool.pas	(revision 62333)
+++ pascalparsertool.pas	(working copy)
@@ -6052,7 +6052,8 @@
   repeat
     // read identifier (a parameter of the generic type)
     Next;
-    AtomIsIdentifierSaveE(20180411194303);
+    ReadTypeReference( CreateChildNodes );
+    {AtomIsIdentifierSaveE(20180411194303);
     if CreateChildNodes then begin
       CreateChildNode;
       CurNode.Desc:=ctnSpecializeParam;
@@ -6068,7 +6069,7 @@
       Next;
     end;
     if CreateChildNodes then
-      EndChildNode; // close ctnSpecializeParam
+      EndChildNode;} // close ctnSpecializeParam
     if AtomIsChar('>') then
       break
     else if CurPos.Flag=cafComma then begin

Juha Manninen

2020-03-05 11:55

developer   ~0121393

Last edited: 2020-03-05 12:03

View 3 revisions

Yes the patch fixes the issue and does not break anything. I applied it.
Thank you soerensen3! I guess you studied the Codetools for a while.

Notes about patches for the future:

1. Please remove old code instead of commenting it out. Old code can be found in commit history if needed.
I removed it before the commit. The code became shorter and simpler. Good!

2. There was a 16 lines offset. I guess you used the released Lazarus version instead of trunk. Sometimes it can cause a conflict and a patch cannot be applied, thus trunk is recommended for all development.

$ cd components/codetools/
$ patch -p0 < ~/patch/fix-nested-generics-specialization.patch
patching file pascalparsertool.pas
Hunk # 1 succeeded at 6068 (offset 16 lines).
Hunk # 2 succeeded at 6085 (offset 16 lines).

soerensen3

2020-03-05 14:23

reporter   ~0121398

Thanks for applying. I worked a bit with Codetools but I wouldn't call me an expert.

I was working with trunk but I guess I did something wrong. This was my first patch.

Denis Golovan

2020-03-05 20:14

reporter   ~0121400

Tested and works as expected.
Thanks a lot.

Mattias Gaertner

2020-05-12 19:04

manager   ~0122736

This patch broke some things.
Juha, there is a test suite for codetools. When applying a patch you can check it easily:
https://wiki.freepascal.org/Codetools#Test_suite

Mattias Gaertner

2020-05-12 19:10

manager   ~0122737

I reverted the patch in 63137.

Juha Manninen

2020-05-12 20:10

developer   ~0122738

Last edited: 2020-05-12 20:11

View 2 revisions

Ok, I will run the test suite in future.
Detaching myself ...

Pascal Riekenberg

2020-05-13 15:31

developer   ~0122764

I think the patch is okay but other parts of codetools still rely on only getting identifiers as generic parameters.
I'll have a look and try to update those parts to get them working again.

Pascal Riekenberg

2020-05-13 17:42

developer   ~0122766

Fixed. See attached patch.
pascalparsertool.pas.patch (4,706 bytes)   
Index: components/codetools/pascalparsertool.pas
===================================================================
--- components/codetools/pascalparsertool.pas	(revision 63142)
+++ components/codetools/pascalparsertool.pas	(working copy)
@@ -176,7 +176,8 @@
     procedure ReadGenericParamList(Must, AllowConstraints: boolean);
     procedure ReadAttribute;
     procedure FixLastAttributes;
-    procedure ReadTypeReference(CreateNodes: boolean);
+    procedure ReadTypeReference(CreateNodes: boolean; Extract: boolean = false;
+      Copying: boolean = false; const Attr: TProcHeadAttributes = []);
     procedure ReadClassInterfaceContent;
     function KeyWordFuncTypeClass: boolean;
     function KeyWordFuncTypeClassInterface(IntfDesc: TCodeTreeNodeDesc): boolean;
@@ -249,7 +250,8 @@
       Copying: boolean = false; const Attr: TProcHeadAttributes = []);
     procedure ReadSpecializeParams(CreateChildNodes: boolean; Extract: boolean = false;
       Copying: boolean = false; const Attr: TProcHeadAttributes = []);
-    procedure ReadAnsiStringParams;
+    procedure ReadAnsiStringParams(Extract: boolean = false;
+      Copying: boolean = false; const Attr: TProcHeadAttributes = []);
     function ReadClosure(ExceptionOnError, CreateNodes: boolean): boolean;
     function SkipTypeReference(ExceptionOnError: boolean): boolean;
     function SkipSpecializeParams(ExceptionOnError: boolean): boolean;
@@ -4388,7 +4390,8 @@
   until Attr=nil;
 end;
 
-procedure TPascalParserTool.ReadTypeReference(CreateNodes: boolean);
+procedure TPascalParserTool.ReadTypeReference(CreateNodes: boolean; Extract: boolean;
+  Copying: boolean; const Attr: TProcHeadAttributes);
 { After reading CurPos is on atom behind the identifier
 
   Examples:
@@ -4399,11 +4402,20 @@
     specialize TGenericClass<TypeRef,TypeRef>
     atype<char>.subtype
 }
+
+  procedure Next; inline;
+  begin
+    if not Extract then
+      ReadNextAtom
+    else
+      ExtractNextAtom(Copying,Attr);
+  end;
+
 var
   Cnt: Integer;
 begin
   if (Scanner.CompilerMode=cmOBJFPC) and UpAtomIs('SPECIALIZE') then begin
-    ReadSpecialize(CreateNodes);
+    ReadSpecialize(CreateNodes,Extract,Copying,Attr);
     exit;
   end;
   if CreateNodes then begin
@@ -4410,12 +4422,12 @@
     CreateChildNode;
     CurNode.Desc:=ctnIdentifier;
   end;
-  ReadNextAtom;
+  Next;
   Cnt:=1;
   while CurPos.Flag=cafPoint do begin
-    ReadNextAtom;
+    Next;
     AtomIsIdentifierSaveE(20180411194207);
-    ReadNextAtom;
+    Next;
     inc(Cnt,2);
   end;
   if AtomIsChar('<') then begin
@@ -4423,8 +4435,8 @@
     or ((Cnt=3) and LastUpAtomIs(3,'SYSTEM') and LastUpAtomIs(1,'STRING'))
     then begin
       // e.g. string<codepage>
-      ReadAnsiStringParams;
-      ReadNextAtom;
+      ReadAnsiStringParams(Extract,Copying,Attr);
+      Next;
     end
     else if (Scanner.CompilerMode in [cmDELPHI,cmDELPHIUNICODE]) then begin
       // e.g. atype<params>
@@ -4436,13 +4448,13 @@
         CurNode.EndPos:=CurPos.StartPos;
         EndChildNode;
       end;
-      ReadSpecializeParams(CreateNodes);
-      ReadNextAtom;
+      ReadSpecializeParams(CreateNodes,Extract,Copying,Attr);
+      Next;
       while CurPos.Flag=cafPoint do begin
         // e.g. atype<params>.subtype
-        ReadNextAtom;
+        Next;
         AtomIsIdentifierSaveE(20180411194209);
-        ReadNextAtom;
+        Next;
       end;
     end;
   end;
@@ -6068,23 +6080,7 @@
   repeat
     // read identifier (a parameter of the generic type)
     Next;
-    AtomIsIdentifierSaveE(20180411194303);
-    if CreateChildNodes then begin
-      CreateChildNode;
-      CurNode.Desc:=ctnSpecializeParam;
-      CurNode.EndPos:=CurPos.EndPos;
-    end;
-    Next;
-    while Curpos.Flag=cafPoint do begin
-      // first identifier was unitname, now read the type
-      Next;
-      AtomIsIdentifierSaveE(20180411194305);
-      if CreateChildNodes then
-        CurNode.EndPos:=CurPos.EndPos;
-      Next;
-    end;
-    if CreateChildNodes then
-      EndChildNode; // close ctnSpecializeParam
+    ReadTypeReference(CreateChildNodes,Extract,Copying,Attr);
     if AtomIsChar('>') then
       break
     else if CurPos.Flag=cafComma then begin
@@ -6099,11 +6095,14 @@
   end;
 end;
 
-procedure TPascalParserTool.ReadAnsiStringParams;
+procedure TPascalParserTool.ReadAnsiStringParams(Extract: boolean; Copying: boolean; const Attr: TProcHeadAttributes);
 begin
   // string<codepage>
   repeat
-    ReadNextAtom;
+    if not Extract then
+      ReadNextAtom
+    else
+      ExtractNextAtom(Copying,Attr);
     if AtomIsChar('>') then break;
     case CurPos.Flag of
     cafRoundBracketOpen,cafEdgedBracketOpen: ReadTilBracketClose(true);
pascalparsertool.pas.patch (4,706 bytes)   

Pascal Riekenberg

2020-05-13 21:43

developer   ~0122771

Extended tests for nested specializations.
testcodecompletion.pas.patch (1,595 bytes)   
Index: components/codetools/tests/testcodecompletion.pas
===================================================================
--- components/codetools/tests/testcodecompletion.pas	(revision 63142)
+++ components/codetools/tests/testcodecompletion.pas	(working copy)
@@ -491,6 +491,7 @@
     'type',
     '  TBird = class',
     '    procedure DoIt(i: specialize TGenList<longint>);',
+    '    procedure DoIt2(i: specialize TGenList<specialize TGenObject<integer>>);',
     '  end;',
     'implementation',
     'end.'],
@@ -504,6 +505,7 @@
     '',
     '  TBird = class',
     '    procedure DoIt(i: specialize TGenList<longint>);',
+    '    procedure DoIt2(i: specialize TGenList<specialize TGenObject<integer>>);',
     '  end;',
     'implementation',
     '',
@@ -511,6 +513,10 @@
     'begin',
     'end;',
     '',
+    'procedure TBird.DoIt2(i: specialize TGenList<specialize TGenObject<integer>>);',
+    'begin',
+    'end;',
+    '',
     'end.']);
 end;
 
@@ -523,6 +529,7 @@
     'type',
     '  TBird = class',
     '    procedure DoIt(i: TGenList<longint>);',
+    '    procedure DoIt2(i: TGenList<TGenObject<longint>>);',
     '  end;',
     'implementation',
     'end.'],
@@ -536,6 +543,7 @@
     '',
     '  TBird = class',
     '    procedure DoIt(i: TGenList<longint>);',
+    '    procedure DoIt2(i: TGenList<TGenObject<longint>>);',
     '  end;',
     'implementation',
     '',
@@ -543,6 +551,10 @@
     'begin',
     'end;',
     '',
+    'procedure TBird.DoIt2(i: TGenList<TGenObject<longint>>);',
+    'begin',
+    'end;',
+    '',
     'end.']);
 end;
 
testcodecompletion.pas.patch (1,595 bytes)   

Mattias Gaertner

2020-05-14 10:18

manager   ~0122785

Thank You!

Issue History

Date Modified Username Field Change
2019-01-06 14:08 Denis Golovan New Issue
2019-06-07 20:52 Denis Golovan Note Added: 0116615
2019-12-30 12:38 Juha Manninen Relationship added related to 0036494
2019-12-30 12:45 Juha Manninen Relationship added related to 0023903
2019-12-30 13:27 Juha Manninen Relationship added duplicate of 0033788
2019-12-30 14:24 Juha Manninen Assigned To => Juha Manninen
2019-12-30 14:24 Juha Manninen Status new => feedback
2019-12-30 14:24 Juha Manninen LazTarget => -
2019-12-30 14:24 Juha Manninen Note Added: 0120151
2019-12-30 14:37 Denis Golovan File Added: Generic-code-completion.tar.gz
2019-12-30 14:37 Denis Golovan Note Added: 0120153
2019-12-30 14:37 Denis Golovan Status feedback => assigned
2019-12-30 14:49 Denis Golovan Note Edited: 0120153 View Revisions
2019-12-30 15:05 Juha Manninen Note Edited: 0120151 View Revisions
2019-12-30 15:24 Juha Manninen Note Added: 0120155
2019-12-30 15:25 Juha Manninen Assigned To Juha Manninen =>
2019-12-30 15:25 Juha Manninen Status assigned => new
2019-12-30 15:26 Juha Manninen Note Edited: 0120155 View Revisions
2019-12-31 12:08 Juha Manninen Note Edited: 0120155 View Revisions
2020-03-02 22:31 soerensen3 Note Added: 0121324
2020-03-02 22:34 soerensen3 Note Edited: 0121324 View Revisions
2020-03-03 00:01 soerensen3 File Added: fix-nested-generics-specialization.patch
2020-03-03 00:01 soerensen3 Note Added: 0121329
2020-03-05 11:09 Juha Manninen Assigned To => Juha Manninen
2020-03-05 11:09 Juha Manninen Status new => assigned
2020-03-05 11:55 Juha Manninen Status assigned => resolved
2020-03-05 11:55 Juha Manninen Resolution open => duplicate
2020-03-05 11:55 Juha Manninen Fixed in Revision => r62693
2020-03-05 11:55 Juha Manninen Widgetset GTK 2 => GTK 2
2020-03-05 11:55 Juha Manninen Note Added: 0121393
2020-03-05 12:02 Juha Manninen Note Edited: 0121393 View Revisions
2020-03-05 12:03 Juha Manninen Note Edited: 0121393 View Revisions
2020-03-05 14:23 soerensen3 Note Added: 0121398
2020-03-05 20:14 Denis Golovan Status resolved => closed
2020-03-05 20:14 Denis Golovan Note Added: 0121400
2020-05-12 19:04 Mattias Gaertner Status closed => assigned
2020-05-12 19:04 Mattias Gaertner Resolution duplicate => reopened
2020-05-12 19:04 Mattias Gaertner Note Added: 0122736
2020-05-12 19:10 Mattias Gaertner Note Added: 0122737
2020-05-12 20:10 Juha Manninen Fixed in Revision r62693 => r62693, r63137
2020-05-12 20:10 Juha Manninen Widgetset GTK 2 => GTK 2
2020-05-12 20:10 Juha Manninen Note Added: 0122738
2020-05-12 20:11 Juha Manninen Assigned To Juha Manninen =>
2020-05-12 20:11 Juha Manninen Note Edited: 0122738 View Revisions
2020-05-13 15:31 Pascal Riekenberg Note Added: 0122764
2020-05-13 17:42 Pascal Riekenberg Note Added: 0122766
2020-05-13 17:42 Pascal Riekenberg File Added: pascalparsertool.pas.patch
2020-05-13 21:43 Pascal Riekenberg Note Added: 0122771
2020-05-13 21:43 Pascal Riekenberg File Added: testcodecompletion.pas.patch
2020-05-14 10:18 Mattias Gaertner Assigned To => Mattias Gaertner
2020-05-14 10:18 Mattias Gaertner Status assigned => resolved
2020-05-14 10:18 Mattias Gaertner Resolution reopened => duplicate
2020-05-14 10:18 Mattias Gaertner Fixed in Revision r62693, r63137 => r62693, r63137, 63148, 63149.
2020-05-14 10:18 Mattias Gaertner Widgetset GTK 2 => GTK 2
2020-05-14 10:18 Mattias Gaertner Note Added: 0122785