View Issue Details

IDProjectCategoryView StatusLast Update
0036273FPCCompilerpublic2020-04-16 19:57
ReporterRick Hoover Assigned ToJonas Maebe  
PrioritynormalSeverityminorReproducibilityalways
Status closedResolutionfixed 
Product Version2.0.4 
Fixed in Version3.2.0 
Summary0036273: Command line overflow for macOS with ld
DescriptionWhen including a large number of packages, the link.res file becomes too large for macOS to handle the large command line generated for the linker (ld). macOS limits the command line to 256K and link.res can push near and beyond that limit when linking the IDE with a large number of installed packages.
Steps To ReproduceInclude a large number of packages from the on-line package manager should cause the command link limit to be hit.
Additional InformationA possible fix would be to split the list of object files out of the link.res file into a separate file. The macOS linker allows the use of the -filelist option to provide the list of object files to link. This fix should remove the list of object files from the command line, which is the bulk of what is put on the linker command line.
TagsNo tags attached.
Fixed in Revision
FPCOldBugId
FPCTarget-
Attached Files

Activities

Alfred

2019-11-14 06:30

reporter   ~0119285

Confirmed. Also on Windows.

When building for Haiku x64 on Windows, the ppas.bat becomes too long (for ld) with all the libraries and object files.
And errors out. If a few, not used, library directories are removed from this file, all runs well.

Alfred

2019-11-14 06:31

reporter   ~0119286

This bug / item should be moved towards fpc (the compiler) !!

Sven Barth

2019-11-15 10:53

manager   ~0119311

Moved to FPC project

Alfred

2019-11-15 11:29

reporter   ~0119312

The filelist option is already added into the compiler t_bsd.pas.
It is only enabled on Windows however, for the benefit of cross-compiling.
(as the [osxcross] linker on Windows understands this option)
It could be enabled on more/all systems, if the linker in use supports filelists.

Jonas Maebe

2019-11-15 21:32

manager   ~0119328

I have fixed several bugs in the filelist support and enabled in all cases when targeting Darwin.

Rick Hoover

2019-12-13 19:46

reporter   ~0119822

Thank you!

Rick Hoover

2020-04-16 18:28

reporter   ~0122176

Re-opened to attach patch file and macOS package that can replace the existing 3.0.4a installer for those who want a 64-bit version of the compiler that back-ports the fix.

Rick Hoover

2020-04-16 18:55

reporter   ~0122178

The patch to back-port the fix for 3.0.4 is attached. The patch modifies the "compiler/systems/t_bsd.pas" file. To patch the t_bsd.pas file, cd to compiler/systems for the 3.0.4 release. Then run "patch <t_bsd.pas.patch". Once patched, you can rebuild fpc.

A package file for macOS can be downloaded from "https://allterrainsw.com/wp-content/fpc-3.0.4a-ldfix.intel-macosx.pkg". Installing the package will add ppcx64-ldfix to /usr/local/bin while keeping the existing 32-bit and 64-bit compilers. To use the fixed version in Lazarus, go to Tools->Options from the main menu and select Environment->Files in the tree view. From there, change the "Compiler executable" to /usr/local/bin/ppcx64-ldfix for the fixed 64-bit version of the compiler. This has worked well for me and should work for releases through 2.0.8 since they all use the 3.0.4 compiler.
t_bsd.pas.patch (7,725 bytes)   
--- t_bsd.pas	2020-04-14 19:13:41.000000000 -0600
+++ t_bsd.pas.new	2020-04-14 15:21:04.000000000 -0600
@@ -146,8 +146,8 @@
        begin
          if not(target_info.system in systems_darwin) then
            begin
-             ExeCmd[1]:='ld $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -L. -o $EXE $CATRES';
-             DllCmd[1]:='ld $TARGET $EMUL $OPT -shared -L. -o $EXE $CATRES'
+             ExeCmd[1]:='ld $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -L. -o $EXE $CATRES $FILELIST';
+             DllCmd[1]:='ld $TARGET $EMUL $OPT -shared -L. -o $EXE $CATRES $FILELIST'
            end
          else
            begin
@@ -166,21 +166,21 @@
                programs with problems that require Valgrind will have more
                than 60KB of data (first 4KB of address space is always invalid)
              }
-               ExeCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -multiply_defined suppress -L. -o $EXE $CATRES';
+             ExeCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -multiply_defined suppress -L. -o $EXE $CATRES $FILELIST';
              if not(cs_gdb_valgrind in current_settings.globalswitches) then
                ExeCmd[1]:=ExeCmd[1]+' -pagezero_size 0x10000';
 {$else ndef cpu64bitaddr}
-             ExeCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -multiply_defined suppress -L. -o $EXE $CATRES';
+             ExeCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -multiply_defined suppress -L. -o $EXE $CATRES $FILELIST';
 {$endif ndef cpu64bitaddr}
              if (apptype<>app_bundle) then
-               DllCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $GCSECTIONS -dynamic -dylib -multiply_defined suppress -L. -o $EXE $CATRES'
+               DllCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $GCSECTIONS -dynamic -dylib -multiply_defined suppress -L. -o $EXE $CATRES $FILELIST'
              else
-               DllCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $GCSECTIONS -dynamic -bundle -multiply_defined suppress -L. -o $EXE $CATRES'
+               DllCmd[1]:='ld $PRTOBJ $TARGET $EMUL $OPT $GCSECTIONS -dynamic -bundle -multiply_defined suppress -L. -o $EXE $CATRES $FILELIST'
            end
        end
      else
        begin
-         ExeCmd[1]:='ld $TARGET $EMUL $OPT $DYNLINK $STATIC  $GCSECTIONS $STRIP -L. -o $EXE $RES';
+         ExeCmd[1]:='ld $TARGET $EMUL $OPT $DYNLINK $STATIC $GCSECTIONS $STRIP -L. -o $EXE $RES';
          DllCmd[1]:='ld $TARGET $EMUL $OPT $INIT $FINI $SONAME -shared -L. -o $EXE $RES';
        end;
      if not(target_info.system in systems_darwin) then
@@ -321,6 +321,7 @@
 Function TLinkerBSD.WriteResponseFile(isdll:boolean) : Boolean;
 Var
   linkres      : TLinkRes;
+  FilesList    : TLinkRes;
   i            : longint;
   cprtobj,
   gprtobj,
@@ -470,28 +471,29 @@
          HPath:=TCmdStrListItem(HPath.Next);
        end;
     end;
-      { force local symbol resolution (i.e., inside the shared }
-      { library itself) for all non-exorted symbols, otherwise }
-      { several RTL symbols of FPC-compiled shared libraries   }
-      { will be bound to those of a single shared library or   }
-      { to the main program                                    }
-      if (isdll) and (target_info.system in systems_bsd) then
-        begin
-          LinkRes.add('VERSION');
-          LinkRes.add('{');
-          LinkRes.add('  {');
-          if not texportlibunix(exportlib).exportedsymnames.empty then
-            begin
-              LinkRes.add('    global:');
-              repeat
-                LinkRes.add('      '+texportlibunix(exportlib).exportedsymnames.getfirst+';');
-              until texportlibunix(exportlib).exportedsymnames.empty;
-            end;
-          LinkRes.add('    local:');
-          LinkRes.add('      *;');
-          LinkRes.add('  };');
-          LinkRes.add('}');
+
+  { force local symbol resolution (i.e., inside the shared }
+  { library itself) for all non-exorted symbols, otherwise }
+  { several RTL symbols of FPC-compiled shared libraries   }
+  { will be bound to those of a single shared library or   }
+  { to the main program                                    }
+  if (isdll) and (target_info.system in systems_bsd) then
+    begin
+      LinkRes.add('VERSION');
+      LinkRes.add('{');
+      LinkRes.add('  {');
+      if not texportlibunix(exportlib).exportedsymnames.empty then
+        begin
+          LinkRes.add('    global:');
+          repeat
+            LinkRes.add('      '+texportlibunix(exportlib).exportedsymnames.getfirst+';');
+          until texportlibunix(exportlib).exportedsymnames.empty;
         end;
+      LinkRes.add('    local:');
+      LinkRes.add('      *;');
+      LinkRes.add('  };');
+      LinkRes.add('}');
+    end;
 
   if not LdSupportsNoResponseFile then
     LinkRes.Add('INPUT(');
@@ -516,16 +518,37 @@
          else if librarysearchpath.FindFile('crtbegin.o',false,s) then
              LinkRes.AddFileName(s);
    end;
+
   { main objectfiles }
-  while not ObjectFiles.Empty do
+
+  if (target_info.system in systems_darwin) then
    begin
-     s:=ObjectFiles.GetFirst;
-     if s<>'' then
-      if LdSupportsNoResponseFile then
-        LinkRes.AddFileName(s)
-      else
-        LinkRes.AddFileName(maybequoted(s));
+     FilesList:=TLinkRes.Create(outputexedir+'linkfiles.res',false);
+     while not ObjectFiles.Empty do
+      begin
+        s:=ObjectFiles.GetFirst;
+        if s<>'' then
+         begin
+           s:=TargetFixFileName(s);
+           FilesList.Add(s);
+         end;
+      end;
+     FilesList.writetodisk;
+     FilesList.Free;
+   end
+  else
+   begin
+     while not ObjectFiles.Empty do
+      begin
+        s:=ObjectFiles.GetFirst;
+        if s<>'' then
+          if LdSupportsNoResponseFile then
+            LinkRes.AddFileName(s)
+          else
+            LinkRes.AddFileName(maybequoted(s));
+      end;
    end;
+
   if not LdSupportsNoResponseFile then
    LinkRes.Add(')');
 
@@ -712,6 +735,11 @@
   Replace(cmdstr,'$EMUL',EmulStr);
   Replace(cmdstr,'$CATRES',CatFileContent(outputexedir+Info.ResName));
   Replace(cmdstr,'$RES',maybequoted(outputexedir+Info.ResName));
+
+  if (target_info.system in systems_darwin) then
+    Replace(cmdstr,'$FILELIST','-filelist '+maybequoted(outputexedir+'linkfiles.res'))
+  else
+    Replace(cmdstr,'$FILELIST','');
   Replace(cmdstr,'$STATIC',StaticStr);
   Replace(cmdstr,'$STRIP',StripStr);
   Replace(cmdstr,'$GCSECTIONS',GCSectionsStr);
@@ -762,6 +790,8 @@
          DeleteFile(linkscript.fn);
          linkscript.free
        end;
+     if target_info.system in systems_darwin then
+       DeleteFile(outputexedir+'linkfiles.res');
    end;
 
   MakeExecutable:=success;   { otherwise a recursive call to link method }
@@ -830,6 +860,10 @@
   Replace(cmdstr,'$TARGET',targetstr);
   Replace(cmdstr,'$EMUL',EmulStr);
   Replace(cmdstr,'$CATRES',CatFileContent(outputexedir+Info.ResName));
+  if (target_info.system in systems_darwin) then
+    Replace(cmdstr,'$FILELIST','-filelist '+maybequoted(outputexedir+'linkfiles.res'))
+  else
+    Replace(cmdstr,'$FILELIST','');
   Replace(cmdstr,'$RES',maybequoted(outputexedir+Info.ResName));
   Replace(cmdstr,'$INIT',InitStr);
   Replace(cmdstr,'$FINI',FiniStr);
@@ -904,7 +938,10 @@
           linkscript.free
         end;
       if (target_info.system in systems_darwin) then
-        DeleteFile(outputexedir+'linksyms.fpc');
+        begin
+          DeleteFile(outputexedir+'linksyms.fpc');
+          DeleteFile(outputexedir+'linkfiles.res');
+        end;
     end;
 
   MakeSharedLibrary:=success;   { otherwise a recursive call to link method }
t_bsd.pas.patch (7,725 bytes)   

Jonas Maebe

2020-04-16 19:09

manager   ~0122179

3.0.4(a) will not be updated again. The issue will be fixed in 3.2.0rc2 and later.

Rick Hoover

2020-04-16 19:57

reporter   ~0122180

Yes it is fixed. But for those who want to stay on the existing release 3.0.4 release, they can use the attached patch or package file. I greatly appreciate your work on the fix that will be in 3.2.0.

Issue History

Date Modified Username Field Change
2019-11-06 23:43 Rick Hoover New Issue
2019-11-14 06:30 Alfred Note Added: 0119285
2019-11-14 06:31 Alfred Note Added: 0119286
2019-11-15 10:52 Sven Barth Project Lazarus => FPC
2019-11-15 10:53 Sven Barth Note Added: 0119311
2019-11-15 10:54 Sven Barth Category Other => Compiler
2019-11-15 10:54 Sven Barth FPCTarget => -
2019-11-15 11:29 Alfred Note Added: 0119312
2019-11-15 21:32 Jonas Maebe Assigned To => Jonas Maebe
2019-11-15 21:32 Jonas Maebe Status new => resolved
2019-11-15 21:32 Jonas Maebe Resolution open => fixed
2019-11-15 21:32 Jonas Maebe Fixed in Version => 3.3.1
2019-11-15 21:32 Jonas Maebe Note Added: 0119328
2019-12-13 19:46 Rick Hoover Status resolved => closed
2019-12-13 19:46 Rick Hoover Note Added: 0119822
2020-04-16 18:28 Rick Hoover Status closed => feedback
2020-04-16 18:28 Rick Hoover Resolution fixed => reopened
2020-04-16 18:28 Rick Hoover Note Added: 0122176
2020-04-16 18:55 Rick Hoover File Added: t_bsd.pas.patch
2020-04-16 18:55 Rick Hoover Note Added: 0122178
2020-04-16 18:55 Rick Hoover Status feedback => assigned
2020-04-16 19:09 Jonas Maebe Status assigned => resolved
2020-04-16 19:09 Jonas Maebe Resolution reopened => fixed
2020-04-16 19:09 Jonas Maebe Fixed in Version 3.3.1 => 3.2.0
2020-04-16 19:09 Jonas Maebe Note Added: 0122179
2020-04-16 19:57 Rick Hoover Status resolved => closed
2020-04-16 19:57 Rick Hoover Note Added: 0122180