View Issue Details

IDProjectCategoryView StatusLast Update
0031066FPCCompilerpublic2020-04-17 14:38
ReporterDenis Golovan Assigned To 
PrioritynormalSeverityfeatureReproducibilityalways
Status newResolutionopen 
Product Version3.1.1 
Summary0031066: FPC projects can't be linked using gold linker
DescriptionHi

Currently FPC produces linker scripts which are not compatible with gold linker.
Thus gold linker cannot be used for Linux development.
See https://sourceware.org/bugzilla/show_bug.cgi?id=20870 for details.

AFAIU, there is a possibility for generating linker scripts which would be compatible with both "old" and gold linkers.
TagsNo tags attached.
Fixed in Revision
FPCOldBugId
FPCTarget
Attached Files

Activities

Florian

2016-12-03 21:14

administrator   ~0096486

While I see this as a feature request only, what would be the advantage of supporting gold?

Denis Golovan

2016-12-03 23:10

reporter   ~0096488

They say gold is much quicker at linking (5x-10x times).
This would help to shorten compile cycle for large projects under Linux.

Thaddy de Koning

2016-12-04 10:09

reporter   ~0096494

ld.gold is rather restrictive to ELF only.
Furthermore the speed gain is also more or less restricted to GNU C++.
You can test that for yourself by compiling and linking e.g. a Linux kernel (which is plain C) with both linkers: there is a speed gain but not nearly 5-10 times, rather 20% to 100% i.e. 1.2 to 2 times. I suspect that is just due to better caching.
Note that with C++ the linker needs to perform a lot more work than for C or FPC.

Denis Golovan

2016-12-04 10:50

reporter   ~0096496

@Thaddy de Koning

It would be nice to elaborate your numbers in C/FPC vs. C++.
Is it because of high template usage? Some kind of function deduplication?
Maybe gold has not been built with --thread and --thread-count options?

My own project is being linked in around 15 sec now by standard ld.
It's around 10 times slower than small changes compilation by fpc itself (1.2-1.5 sec). And yes I use templated/macroexpanded (via m4) code a lot.

Thaddy de Koning

2016-12-04 13:37

reporter   ~0096500

It is simply an incremental linker approach with some de-dup and patch space.
E.g.: like our own internal linker ;)

See this document for internals.
https://events.linuxfoundation.org/images/stories/pdf/lfcs2012_ccoutant.pdf

Note specifically the limitations section: it is - still - not for production code!

Denis Golovan

2016-12-04 14:18

reporter   ~0096502

Are you trying to tell me that it's not possible to gain 5x+ improvement without incremental linking?
And it's not supported by fpc for now?

BTW, is internal linker supported for Linux targets?

Thaddy de Koning

2016-12-04 16:00

reporter   ~0096503

Well, the link is to a presentation of the makers of ld.gold.

So yes, initial compile is the same. Initial linking is only 50% faster and only later recompiles and linking has any significant gains. Read for yourself.

Denis Golovan

2016-12-05 09:22

reporter   ~0096525

Yes. It seems that incremental linking gives the largest speed-ups.
And incremental debug builds from IDE will be built much faster.
So I still consider it's a good trade-off - fixing couple lines in linker script + passing extra parameters to linker for reducing linking time.

@Florian
Is it true, that internal linker for Linux target is still work-in-progress?

Thaddy de Koning

2016-12-05 12:04

reporter   ~0096526

"And incremental debug builds from IDE will be built much faster."
That's debatable, since FPC already has the PPU format.

Sven Barth

2016-12-06 22:31

manager   ~0096548

@Thaddy:
1) The PDF you linked to is from 2012, so it's debateable whether gold is still considered not ready for production use.
2) It could indeed be that incremental builds could be faster as the linker wouldn't need to recreate the whole binary as it does currently; after all linking *is* rather time consuming and if the linker knows that 98% of the object files didn't change than it merely needs to add/update those 2% that did...
3) That gold is restricted to ELF is not really a problem. It would simply be an additional option for ELF targets then...

Jonas Maebe

2016-12-08 11:26

manager   ~0096593

Last edited: 2016-12-08 11:28

View 2 revisions

> Is it true, that internal linker for Linux target is still work-in-progress?

Yes, which I think is a good thing, because we have plenty of work with maintaining a compiler already. The person who originally implemented the internal linker is no longer active either.

Additionally, linking on Linux is much more complicated than on Windows and more work would probably be needed to be able to handle every kind of relocation and feature used by non-FPC-compiled code (or a new feature in FPC may then require new internal linker features). Although even on Windows we are now getting such issues (0030614 ).

As to topic of this original bug report:
* I think we can remove the .fpcdata part altogether, it does not seem to be used anywhere (except for a reference we insert in the linker script)
* maybe we can get rid of the .threadvar stuff too. If we actually would implement threadvars based on tls for ELF, we should probably put them in the standard .tbss and .tdata sections instead

Denis Golovan

2016-12-09 22:26

reporter   ~0096630

@Jonas

Thanks for info.
I guess in this case gold linker provide a nice backup until internal linker is ready.

Jonas Maebe

2016-12-09 22:40

manager   ~0096631

The issue is that the gold linker apparently does not support all features either (as demonstrated by this bug report), so it may also cause more issues in the future and more support work (especially for people wanting to add new stuff to the compiler, which then may break gold again even if this issue would be fixed).

Sergei Gorelkin

2016-12-10 02:44

developer   ~0096634

Please do not be mistaken: the internal linker is not incremental and it never was. Its huge speed advantage on Windows/COFF targets is caused by poor quality of BFD COFF backend. ELF targets are much better maintained on BFD side, therefore internal linker provides approximately same speed as GNU ld.

As for its readiness, in the mean time I've reached the state where it can run the test suite without failures, but there are endless issues related to debugging, profiling and such. If anyone wants to see for himself, it is sufficient to replace "link: ld_none" with "link: ld_int_linux" line for appropriate target in systems/i_linux.pas.

Jonas Maebe

2016-12-10 12:21

manager   ~0096637

> Its huge speed advantage on Windows/COFF targets is caused by poor quality of BFD COFF backend.

Maybe also because we still use library-based smart linking on Windows? The speed might be better if we would use section-based smart linking on Windows like on other platforms (if it works there).

Sergei Gorelkin

2016-12-10 14:37

developer   ~0096642

Section-based smart linking was not supported by BFD COFF backend when I debugged that around 3 years ago.
--gc-sections option is listed by 'ld --help' and is not rejected by ld, and it even triggers execution of some generic code, but there is no backend-specific routine.

Jonas Maebe

2016-12-10 14:44

manager   ~0096643

It seems there has been some progress, but there are still some things that don't work with it yet: https://sourceware.org/bugzilla/show_bug.cgi?id=11539

rusty_robot

2018-10-23 22:12

reporter   ~0111530

https://lld.llvm.org linker might decrease link time too (if compatible with fpc).

Denis Golovan

2019-06-16 10:48

reporter   ~0116746

The bug is still in-effect in svn rev.42189

Thaddy de Koning

2019-06-16 11:52

reporter   ~0116749

Last edited: 2019-06-16 12:00

View 3 revisions

It's not a bug. It is a feature request.
And lld might be the better option as it stands: it accepts ld linker scripts afaict.
https://lld.llvm.org/

Denis Golovan

2019-06-16 17:54

reporter   ~0116752

I've just tested lld.
It fails with:
Error: llvm-ld: error: unknown -format value: elf64-x86-64 (supported formats: elf, default, binary)

Benito van der Zander

2019-06-16 19:34

reporter   ~0116753

Btw, this has cost my a few hours of debugging recently.

In the Android NDK gold is the default linker.

Thaddy de Koning

2019-06-17 11:27

reporter   ~0116758

Last edited: 2019-06-17 11:29

View 3 revisions

Denis, if you have the x86_64 binaries installed, ld.lld also supports 64 bit elf. I tested that. I made some modifications to the script, but I don't think these were necessary.
I also use a release candidate 8 https://prereleases.llvm.org/8.0.0/

The changes were replacing some things with default. That is elf x86_64

Denis Golovan

2019-06-17 14:07

reporter   ~0116759

denis@desktop64 /tmp $ llvm-ld --version
LLD 8.0.1 (compatible with GNU linkers)
denis@desktop64 /tmp $ fpc -XPllvm- project1.lpr
Free Pascal Compiler version 3.3.1 [2019/06/08] for x86_64
Copyright (c) 1993-2018 by Florian Klaempfl and others
llvm-ld: error: unknown -format value: elf64-x86-64 (supported formats: elf, default, binary)
project1.lpr(6,1) Error: Error while linking
project1.lpr(6,1) Fatal: There were 1 errors compiling module, stopping

Thaddy de Koning

2019-06-18 09:52

reporter   ~0116767

Denis, are you sure you have a 64 bit installation and not a 32 bit for LLVM? So not Freepascal, but the toolchain?

Denis Golovan

2019-06-18 11:15

reporter   ~0116769

Yes. I am sure.
Both Gentoo x64 and Devuan x64 tested.
Same error.

Unfortunately, there seems to be no way to ask lld of supported architectures.
It just emits:
llvm-ld: supported targets: elf

Florian

2019-11-23 17:54

administrator   ~0119466

Last edited: 2019-11-23 17:54

View 2 revisions

I tried with the linker script FPC uses when cross compiling:

/usr/local/bin/gold: internal error in read_script_file, at script.cc:1638

Edit: gold from binutils 2.33.1

Thaddy de Koning

2019-11-23 22:21

reporter   ~0119468

Last edited: 2019-11-23 22:24

View 2 revisions

I will test again. I made a copy of ld.gold to ld on arm (Raspberry Pi) and that works for me.
Will report back.
[edit]
version is 2.31.1

BBaz

2020-04-17 12:29

reporter   ~0122189

Following the status of this issue. Just experienced this using ld.lld

Denis Golovan

2020-04-17 14:38

reporter   ~0122197

Yeap.
LLD has the same issue.

Issue History

Date Modified Username Field Change
2016-12-03 12:58 Denis Golovan New Issue
2016-12-03 21:14 Florian Note Added: 0096486
2016-12-03 23:10 Denis Golovan Note Added: 0096488
2016-12-04 10:09 Thaddy de Koning Note Added: 0096494
2016-12-04 10:50 Denis Golovan Note Added: 0096496
2016-12-04 13:37 Thaddy de Koning Note Added: 0096500
2016-12-04 14:18 Denis Golovan Note Added: 0096502
2016-12-04 16:00 Thaddy de Koning Note Added: 0096503
2016-12-05 09:22 Denis Golovan Note Added: 0096525
2016-12-05 12:04 Thaddy de Koning Note Added: 0096526
2016-12-06 22:31 Sven Barth Note Added: 0096548
2016-12-08 11:26 Jonas Maebe Note Added: 0096593
2016-12-08 11:28 Jonas Maebe Note Edited: 0096593 View Revisions
2016-12-09 22:26 Denis Golovan Note Added: 0096630
2016-12-09 22:40 Jonas Maebe Note Added: 0096631
2016-12-10 02:44 Sergei Gorelkin Note Added: 0096634
2016-12-10 12:21 Jonas Maebe Note Added: 0096637
2016-12-10 14:37 Sergei Gorelkin Note Added: 0096642
2016-12-10 14:44 Jonas Maebe Note Added: 0096643
2018-07-07 20:04 Marco van de Voort Severity minor => feature
2018-10-23 22:12 rusty_robot Note Added: 0111530
2019-06-16 10:48 Denis Golovan Note Added: 0116746
2019-06-16 11:52 Thaddy de Koning Note Added: 0116749
2019-06-16 11:59 Thaddy de Koning Note Edited: 0116749 View Revisions
2019-06-16 12:00 Thaddy de Koning Note Edited: 0116749 View Revisions
2019-06-16 17:54 Denis Golovan Note Added: 0116752
2019-06-16 19:34 Benito van der Zander Note Added: 0116753
2019-06-17 11:27 Thaddy de Koning Note Added: 0116758
2019-06-17 11:28 Thaddy de Koning Note Edited: 0116758 View Revisions
2019-06-17 11:29 Thaddy de Koning Note Edited: 0116758 View Revisions
2019-06-17 14:07 Denis Golovan Note Added: 0116759
2019-06-18 09:52 Thaddy de Koning Note Added: 0116767
2019-06-18 11:15 Denis Golovan Note Added: 0116769
2019-11-23 17:54 Florian Note Added: 0119466
2019-11-23 17:54 Florian Note Edited: 0119466 View Revisions
2019-11-23 22:21 Thaddy de Koning Note Added: 0119468
2019-11-23 22:24 Thaddy de Koning Note Edited: 0119468 View Revisions
2020-04-17 12:29 BBaz Note Added: 0122189
2020-04-17 14:38 Denis Golovan Note Added: 0122197