View Issue Details

IDProjectCategoryView StatusLast Update
0028448LazarusCompilerpublic2020-10-15 14:09
ReporterGraham Inggs Assigned ToJuha Manninen  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformARMOSDebian 
Product Version1.4 
Summary0028448: lazbuild appears to stall when building on armel and armhf
DescriptionSince lazarus 1.4.0 appeared in the Debian unstable distribution, builds on armel and armhf have been failing, appearing to stall.
Full build logs can be viewed [1][2][3] by clicking on the Build-Attempted links for a particular architecture.

[1] https://buildd.debian.org/status/package.php?p=doublecmd&suite=unstable
[2] https://buildd.debian.org/status/package.php?p=easymp3gain&suite=unstable
[3] https://buildd.debian.org/status/package.php?p=cqrlog&suite=unstable
Steps To ReproduceI chose a simpler project [1] and attempted to build it on real ARM hardware (NVIDIA Jetson TK1). Instead of building the Debian package, I simply ran:

lazbuild source/ddrescueview.lpi --bm="GNU/Linux Release"

The build was successful with lazarus 1.2.4 and 1.2.6, but appeared to stall with lazarus 1.4.0 and 1.4.2. In all cases, fpc 2.6.4 as it is in Debian was used. I attach build logs of the 1.2.6, 1.4.0 and 1.4.2 builds.

I noticed that the failed build logs were 4096 bytes in size and, as can be seen in the logs from the Debian buildds, the last line of the log is truncated, e.g.

TExternalTool.DoExecute Title="Compile Project, Mode: GNU/Linux Release, Target: ddrescueview" Process.CurrentDirectory="/srv/ddrescueview/source/" Executable="/usr/bin/fpc" Params:
-B
-Tlinux
-MObjFPC
-Schi
-CX
-O3
-Xs
-XX
-k-z
-krelro
-k--as-needed
-l
-vewnhibq
-Fl/opt/gnome/lib
-Fu/usr/lib/lazarus/1.4.2/lcl/units/arm-linux/gtk2
-Fu/usr/lib/lazar

After hitting ctrl-c, I noticed that the build had actually completed successfully and there was a working 'ddrescueview' executable.

[1] http://ddrescueview.sourceforge.net/
TagsNo tags attached.
Fixed in Revisionr64005
LazTarget-
Widgetset
Attached Files

Relationships

related to 0036318 resolvedJuha Manninen lazbuild crashes when compiling packages 

Activities

Graham Inggs

2015-07-31 12:40

reporter  

Graham Inggs

2015-07-31 17:15

reporter   ~0085118

Manually running the fpc command line that lazbuild runs (see below), seems fine, and the output appears on stdout as expected.

/usr/bin/fpc -B -Tlinux -MObjFPC -Schi -CX -O3 -Xs -XX -k-z -krelro -k--as-needed -l -vewnhibq -Fl/opt/gnome/lib -Fu/usr/lib/lazarus/1.4.2/lcl/units/arm-linux/gtk2 -Fu/usr/lib/lazarus/1.4.2/lcl/units/arm-linux -Fu/usr/lib/lazarus/1.4.2/components/lazutils/lib/arm-linux -Fu/usr/lib/lazarus/1.4.2/packager/units/arm-linux -Fu/srv/ddrescueview/source/ -dLCL -dLCLgtk2 ddrescueview.lpr

Graham Inggs

2015-08-04 11:38

reporter   ~0085193

On ARM, the following call from around line 510 in ide/exttools.pas never returns:

Thread.Synchronize(Thread,@NotifyHandlerNewOutput);

Any ideas where else I can look?

Anton Kavalenka

2015-08-04 13:59

reporter   ~0085197

Last edited: 2015-08-04 14:00

View 2 revisions

--max-process-count=0 bug in lazbuild?

Automatic detection of CPU count sometimes fails.

try to run with
lazbuild --max-process-count=1

Graham Inggs

2015-08-04 21:49

reporter   ~0085207

Thanks, but that option doesn't seem to be available in lazarus 1.4.2:

Invalid option at position 1: "max-process-count"

Graham Inggs

2015-08-06 00:51

reporter   ~0085221

I tried merging r48422 [1] (which added --max-process-count) and r48447 [2] (which I believe fixed the issue Anton was referring to) into lazarus 1.4.2 and still experienced the problem, even when setting --max-process-count=1.

I also tried the lazarus 1.5-49593-20150805 snapshot [3] and the problem is still present.

[1] http://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=48422
[2] http://svn.freepascal.org/cgi-bin/viewvc.cgi?view=revision&root=lazarus&revision=48447
[3] ftp://ftp.freepascal.org/pub/lazarus/snapshots/lazarus-1.5-49593-20150805-src.zip

Jonas Maebe

2015-08-06 14:31

manager   ~0085231

Does the linker on that system suffer from https://sourceware.org/bugzilla/show_bug.cgi?id=18344 ? You should be able to verify that by compiling any program with fpc and seeing whether there are a bunch of stray empty lines.

If so and if that output gets written to stderr rather than standard out, what possibly happens is that stderr gets full and stalls the child process when it tries to write more data to it.

FPC (not Lazarus) revision 29303 contains a work around for child processes launched via the tprocess class getting stuck if they write a lot to stderr and little or nothing to stdout. Even if the linker does not suffer from the problem mentioned at the start of this comment, it could still be useful to try.

Graham Inggs

2015-08-06 17:13

reporter   ~0085237

> Does the linker on that system suffer from
> https://sourceware.org/bugzilla/show_bug.cgi?id=18344 [^] ?
> You should be able to verify that by compiling any program with fpc and
> seeing whether there are a bunch of stray empty lines.

Yes, it does suffer from that problem (I didn't even know it was a problem!). You can even see it in, for example, the i386 build of doublecmd from one of the links in my description, although it doesn't stall on that architecture.

> FPC (not Lazarus) revision 29303 contains a work around for child processes
> launched via the tprocess class getting stuck if they write a lot to stderr
> and little or nothing to stdout.

I'm going to try anyway, but do you think it would be sufficient just to merge in that commit? So far, I have been unable to build the fixes branch or the snapshot of fpc on this system. I can, however, rebuild the Debian fpc package with patches.

Jonas Maebe

2015-08-06 17:57

manager   ~0085238

> I'm going to try anyway, but do you think it would be sufficient just to merge in that commit?

Yes, it should be independent of everything else.

> So far, I have been unable to build the fixes branch or the snapshot of fpc on this system.

What is the problem?

Graham Inggs

2015-08-07 14:10

reporter   ~0085251

> Yes, it should be independent of everything else.

I ended up merging r25363, r28972 and r29303, rebuilding fpc and then rebuilding lazarus. Unfortunately lazbuild still appears to hang, and needs Ctrl-C.

>> So far, I have been unable to build the fixes branch or the snapshot of fpc on this system.
> What is the problem?

There are many patches against the Debian fpc package [1], and I haven't been able to go through them all to determine which ones are still needed and get them to apply cleanly, so I couldn't easily drop in an updated fpc source tarball and build a new Debian package, like I could do with lazarus.

I also tried 'NOGDB=1 make all' in the daily source snapshot of the fixes tree (fpcbuild.zip), but got the following error:

flt_pack.inc(119,5) Error: Syntax error while parsing a conditional compiling expression
system.inc(1821,8) Warning: Implicit string type conversion from "RawByteString" to "UnicodeString"
system.pp(375) Fatal: There were 1 errors compiling module, stopping

Unfortunately, I will not be able to do any further testing on this for the next three weeks or so.


[1] https://anonscm.debian.org/cgit/pkg-pascal/fpc.git/tree/debian/patches

Graham Inggs

2015-09-03 13:08

reporter   ~0085686

The problem seems to be related to the '-z relro' linker option and ARM.

If we build Lazarus with the Debian default hardening flags, which include '-z relro', then lazbuild appears to stall.

If we rebuild Lazarus and filter out '-z relro' from the linker flags, lazbuild works as expected.

Jonas Maebe

2015-09-13 00:21

manager   ~0085868

Could you try with trunk r31451 merged?

Graham Inggs

2015-09-15 13:37

reporter   ~0085939

> Could you try with trunk r31451 merged?

I was able to apply r31451 to fpc 2.6.4 and that worked, thanks!

I have tested on what Debian call 'armhf', but do you think this should work for 'armel' as well? Unfortunately, I don't have any suitable hardware to test on.

BTW, I still get the flt_pack.inc and system.inc errors mentioned above when tryng to build fpc 3.0.0 rc1. Should I open a separate bug for that?

Jonas Maebe

2015-09-15 13:59

manager   ~0085940

> I was able to apply r31451 to fpc 2.6.4 and that worked, thanks!

Thanks, I'll ask to merge it into 3.0 as well then.

> I have tested on what Debian call 'armhf', but do you think this should work for 'armel' as well?

Yes.

> BTW, I still get the flt_pack.inc and system.inc errors mentioned above when tryng to build fpc 3.0.0 rc1.

The system.inc message is a warning, not an error, and is benign. I can't imagine where the flt_pack.inc error comes from, other than a bug in the Debian 2.6.4 ARMHF compiler binary (which uses patches that have been backported by Debian developers, and which are not supported by the FPC developers with FPC 2.6.4). The code in flt_pack.inc is used on all platforms and builds fine with FPC 2.6.4 everywhere else, as far as I know.

Joost van der Sluis

2015-09-30 21:43

manager   ~0086193

I've merged r31451 to fixes-3_0. Can this bug be closed now? Or is it something unrelated?

Jonas Maebe

2015-10-01 07:53

manager   ~0086194

That's a good question. The title of the bug report mentions ARMEL and ARMHF, but r31451 only affects ARMHF.

Paul Gevers

2016-02-27 21:52

reporter   ~0090417

I just rebuild winff with fpc 3.0.0+dfsg-2 and lazarus 1.6~rc2+dfsg-1 and the issue doesn't manifest itself. So I would say this bug can be closed.

Juha Manninen

2020-10-15 14:09

developer   ~0126330

Apparently fixed. Please test.

Issue History

Date Modified Username Field Change
2015-07-31 12:40 Graham Inggs New Issue
2015-07-31 12:40 Graham Inggs File Added: ddrescueview-build-logs.tar.gz
2015-07-31 17:15 Graham Inggs Note Added: 0085118
2015-08-04 11:38 Graham Inggs Note Added: 0085193
2015-08-04 13:59 Anton Kavalenka Note Added: 0085197
2015-08-04 14:00 Anton Kavalenka Note Edited: 0085197 View Revisions
2015-08-04 21:49 Graham Inggs Note Added: 0085207
2015-08-06 00:51 Graham Inggs Note Added: 0085221
2015-08-06 14:31 Jonas Maebe Note Added: 0085231
2015-08-06 17:13 Graham Inggs Note Added: 0085237
2015-08-06 17:57 Jonas Maebe Note Added: 0085238
2015-08-07 14:10 Graham Inggs Note Added: 0085251
2015-09-03 13:08 Graham Inggs Note Added: 0085686
2015-09-13 00:21 Jonas Maebe Note Added: 0085868
2015-09-15 13:37 Graham Inggs Note Added: 0085939
2015-09-15 13:59 Jonas Maebe Note Added: 0085940
2015-09-30 21:43 Joost van der Sluis Note Added: 0086193
2015-10-01 07:53 Jonas Maebe Note Added: 0086194
2016-02-27 21:52 Paul Gevers Note Added: 0090417
2017-08-27 00:28 Juha Manninen Relationship added related to 0031989
2017-08-28 20:06 Juha Manninen Relationship deleted related to 0031989
2019-11-16 07:42 Juha Manninen Relationship added related to 0036318
2020-10-15 14:09 Juha Manninen Assigned To => Juha Manninen
2020-10-15 14:09 Juha Manninen Status new => resolved
2020-10-15 14:09 Juha Manninen Resolution open => fixed
2020-10-15 14:09 Juha Manninen Fixed in Revision => r64005
2020-10-15 14:09 Juha Manninen LazTarget => -
2020-10-15 14:09 Juha Manninen Note Added: 0126330