View Issue Details

IDProjectCategoryView StatusLast Update
0032273FPCUtilitiespublic2017-09-22 16:10
ReporterSeth GroverAssigned ToMichael Van Canneyt 
PrioritynormalSeverityminorReproducibilityN/A
Status closedResolutionfixed 
Product VersionProduct Build 
Target Version3.2.0Fixed in Version3.1.1 
Summary0032273: large grammar causes "integer set overflow" in pyacc
DescriptionI use pyacc (http://wiki.freepascal.org/Plex_and_Pyacc) to generate a parser with quite a large grammar, which causes it to fail.

I've included a patch file to increase the sizes of the necessary arrays to support larger grammars. I haven't noticed any problems caused by the changes or extreme memory usage or anything like that.

I'll also include a sample .y file (sanitized) to illustrate the issue.
Steps To ReproduceWith the attached SQL.y example and the original pyacc:

$ pyacc SQL.y
TP Yacc Version 4.1a [April 2000], Copyright (c) 1990-2000 Albert Graef
parse ... sort ... closures ... first sets ...
FATAL: integer set overflow

With my patch:

$ pyacc SQL.y
TP Yacc Version 4.1a [April 2000], Copyright (c) 1990-2000 Albert Graef
parse ... sort ... closures ... first sets ... LR0 set ... lookaheads ...
code generation ... DONE
1262 lines, 618/900 rules, 1460/3000 s, 21915/40000 i, 20765/40000 t, 640/9600 r.
74 reduce/reduce conflicts.
TagsNo tags attached.
Fixed in Revision36922
FPCOldBugId
FPCTarget
Attached Files
  • fpc_trunk_36916_yacc_arrays.patch (2,215 bytes)
    diff -Naur fpc_trunk_36916/utils/tply/yaccbase.pas fpc_trunk_36916_patched/utils/tply/yaccbase.pas
    --- fpc_trunk_36916/utils/tply/yaccbase.pas	2015-01-18 06:11:38.157963000 -0700
    +++ fpc_trunk_36916_patched/utils/tply/yaccbase.pas	2017-08-15 10:03:58.462265388 -0600
    @@ -83,9 +83,9 @@
     const
     
     {$IFDEF MsDos}
    -max_elems  = 50;  (* maximum size of integer sets *)
    +max_elems  =  50; (* maximum size of integer sets *)
     {$ELSE}
    -max_elems  = 75; (* maximum size of integer sets *)
    +max_elems  = 150; (* maximum size of integer sets *)
     {$ENDIF}
     
     type
    diff -Naur fpc_trunk_36916/utils/tply/yacctabl.pas fpc_trunk_36916_patched/utils/tply/yacctabl.pas
    --- fpc_trunk_36916/utils/tply/yacctabl.pas	2005-05-18 14:24:09.513140000 -0600
    +++ fpc_trunk_36916_patched/utils/tply/yacctabl.pas	2017-08-15 10:03:58.502266469 -0600
    @@ -70,13 +70,13 @@
     max_types          =  100;  (* number of type tags                         *)
     max_prec           =   50;  (* maximum precedence level                    *)
     {$IFDEF Windows}
    -max_states         =  800;  (* number of LR(0) states                      *)
    +max_states         = 3000;  (* number of LR(0) states                      *)
     {$ELSE}
    -max_states         = 1200;  (* number of LR(0) states                      *)
    +max_states         = 3000;  (* number of LR(0) states                      *)
     {$ENDIF}
    -max_items          = 9600;  (* number of items                             *)
    -max_trans          = 9600;  (* number of transitions                       *)
    -max_redns          = 1200;  (* number of reductions                        *)
    +max_items          = 40000;  (* number of items                             *)
    +max_trans          = 40000;  (* number of transitions                       *)
    +max_redns          =  9600;  (* number of reductions                        *)
     {$ENDIF}
     
     {$IFDEF MsDos}
    @@ -84,7 +84,7 @@
     max_set_items      =   64;  (* maximum number of items in an item set      *)
     {$ELSE}
     max_rule_len       =   64;  (* maximum length of rules                     *)
    -max_set_items      =   96;  (* maximum number of items in an item set      *)
    +max_set_items      = 9600;  (* maximum number of items in an item set      *)
     {$ENDIF}
     
     var
    
  • SQL.y (31,968 bytes)

Activities

Seth Grover

2017-08-15 18:32

reporter  

fpc_trunk_36916_yacc_arrays.patch (2,215 bytes)
diff -Naur fpc_trunk_36916/utils/tply/yaccbase.pas fpc_trunk_36916_patched/utils/tply/yaccbase.pas
--- fpc_trunk_36916/utils/tply/yaccbase.pas	2015-01-18 06:11:38.157963000 -0700
+++ fpc_trunk_36916_patched/utils/tply/yaccbase.pas	2017-08-15 10:03:58.462265388 -0600
@@ -83,9 +83,9 @@
 const
 
 {$IFDEF MsDos}
-max_elems  = 50;  (* maximum size of integer sets *)
+max_elems  =  50; (* maximum size of integer sets *)
 {$ELSE}
-max_elems  = 75; (* maximum size of integer sets *)
+max_elems  = 150; (* maximum size of integer sets *)
 {$ENDIF}
 
 type
diff -Naur fpc_trunk_36916/utils/tply/yacctabl.pas fpc_trunk_36916_patched/utils/tply/yacctabl.pas
--- fpc_trunk_36916/utils/tply/yacctabl.pas	2005-05-18 14:24:09.513140000 -0600
+++ fpc_trunk_36916_patched/utils/tply/yacctabl.pas	2017-08-15 10:03:58.502266469 -0600
@@ -70,13 +70,13 @@
 max_types          =  100;  (* number of type tags                         *)
 max_prec           =   50;  (* maximum precedence level                    *)
 {$IFDEF Windows}
-max_states         =  800;  (* number of LR(0) states                      *)
+max_states         = 3000;  (* number of LR(0) states                      *)
 {$ELSE}
-max_states         = 1200;  (* number of LR(0) states                      *)
+max_states         = 3000;  (* number of LR(0) states                      *)
 {$ENDIF}
-max_items          = 9600;  (* number of items                             *)
-max_trans          = 9600;  (* number of transitions                       *)
-max_redns          = 1200;  (* number of reductions                        *)
+max_items          = 40000;  (* number of items                             *)
+max_trans          = 40000;  (* number of transitions                       *)
+max_redns          =  9600;  (* number of reductions                        *)
 {$ENDIF}
 
 {$IFDEF MsDos}
@@ -84,7 +84,7 @@
 max_set_items      =   64;  (* maximum number of items in an item set      *)
 {$ELSE}
 max_rule_len       =   64;  (* maximum length of rules                     *)
-max_set_items      =   96;  (* maximum number of items in an item set      *)
+max_set_items      = 9600;  (* maximum number of items in an item set      *)
 {$ENDIF}
 
 var

Seth Grover

2017-08-15 18:32

reporter  

SQL.y (31,968 bytes)

Michael Van Canneyt

2017-08-16 09:09

administrator   ~0102198

Last edited: 2017-08-16 09:09

View 2 revisions

The units stem from the TP days. No wonder the limits are very small.

I applied the patch, tested, and all seems well. Thanks for the patch !

Seth Grover

2017-09-22 16:10

reporter   ~0102978

Thank you!

Issue History

Date Modified Username Field Change
2017-08-15 18:32 Seth Grover New Issue
2017-08-15 18:32 Seth Grover File Added: fpc_trunk_36916_yacc_arrays.patch
2017-08-15 18:32 Seth Grover File Added: SQL.y
2017-08-16 09:09 Michael Van Canneyt Fixed in Revision => 36922
2017-08-16 09:09 Michael Van Canneyt Note Added: 0102198
2017-08-16 09:09 Michael Van Canneyt Status new => resolved
2017-08-16 09:09 Michael Van Canneyt Fixed in Version => 3.1.1
2017-08-16 09:09 Michael Van Canneyt Resolution open => fixed
2017-08-16 09:09 Michael Van Canneyt Assigned To => Michael Van Canneyt
2017-08-16 09:09 Michael Van Canneyt Target Version => 3.2.0
2017-08-16 09:09 Michael Van Canneyt Note Edited: 0102198 View Revisions
2017-09-22 16:10 Seth Grover Note Added: 0102978
2017-09-22 16:10 Seth Grover Status resolved => closed