0037179: [lazbuild] [patch] Lazbuild randomly aborts project building process.

All Projects FPC fpcprojects fpprofiler fpGUI Lazarus Packages Patches Lazarus CCR Mantis pas2js

ID	Project	Category	View Status	Date Submitted	Last Update

0037179	Lazarus	IDE	public	2020-06-06 04:26	2020-10-15 14:11

Reporter	Cyrax	Assigned To	Juha Manninen
Priority	normal	Severity	minor	Reproducibility	random
Status	resolved	Resolution	reopened
Platform	Linux x86_64	OS	Arch
Product Version	2.1 (SVN)

Summary	0037179: [lazbuild] [patch] Lazbuild randomly aborts project building process.
Description	procedure TExternalToolThread.SetTool(AValue: TExternalTool); begin if FTool=AValue then Exit; if FTool<>nil then FTool.Thread:=nil; FTool:=AValue; if FTool<>nil then FTool.Thread:=Self; end; destructor TExternalToolThread.Destroy; begin Tool:=nil; inherited Destroy; end; Setting FTool field to NIL (aka trying to prepare it and thread associated to it to be destroyed as well) via property setter causes already running thread to product an access violation and thus breaking the build process along side of it. Attached patch will fix this by accessing FTool field directly. Valgrind didn't show any leaks.
Tags	No tags attached.

Fixed in Revision	r63321, r64005
LazTarget	-
Widgetset	GTK 2

Attached Files

Cyrax 2020-06-06 04:26 reporter	exttools.pas.diff (267 bytes) diff --git a/ide/exttools.pas b/ide/exttools.pas index f81f0f27e2..598948ce2a 100644 --- a/ide/exttools.pas +++ b/ide/exttools.pas @@ -1678,7 +1678,7 @@ end; destructor TExternalToolThread.Destroy; begin - Tool:=nil; + FTool:=nil; inherited Destroy; end; exttools.pas.diff (267 bytes)

Alfred 2020-06-06 08:53 reporter ~0123253	After applying this patch, no crashes anymore when building Lazarus. The crash also happened very often when building lazpaint. This patch also solved the lazpaint crash. So, it seems to work. And. If valgrind shows no leaks, this could mean that the issues can be considered as solved. https://bugs.freepascal.org/view.php?id=36318 https://bugs.freepascal.org/view.php?id=36959 @juha: please review !

Mattias Gaertner 2020-06-06 10:04 manager ~0123254	The patch omits calling SetThread. This might work in lazbuild, but not in the IDE. CallAutoFree in SetThread was not initialized. I fixed that.

Alfred 2020-06-06 10:29 reporter ~0123256	@Mattias Your fix works on Windows. Even with lazbuild max-process-count set to a value > 1. Going to test on Catalina now. Will report re-occurrence of issue. You might close related issues I guess.

Alfred 2020-06-06 10:33 reporter ~0123257	Celebrating too early I guess ... :-( After 12 x building lazpaint ... it happened again unfortunately. lazbuild01.JPG (84,991 bytes) lazbuild01.JPG (84,991 bytes)

Cyrax 2020-06-07 07:37 reporter ~0123289	Stack trace $081A02A0 file: components/buildintf/ideexterntoolintf.pas : 1237; $08757A34 file: ide/exttools.pas : 475; $0875C958 file: ide/exttools.pas : 1383; $0875DF6F file: ide/exttools.pas : 1689; $0806D702 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; $080A4062 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; $0808C465 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; $F7EE8687 $00000000

Alfred 2020-06-07 10:46 reporter ~0123293	@Mattias. Latest fix looks good. 25x rebuild/install of lazpaint : all ok. Will run for max 100x.

Alfred 2020-06-07 11:46 reporter ~0123298	After 62 times, the (repeated) failures start again. (3104) Compiling lcvectormultishape.pas (1008) 15619 lines compiled, 3.6 sec (1021) 31 warning(s) issued (1022) 226 hint(s) issued (1023) 63 note(s) issued An unhandled exception occurred at $778B3D63: EAccessViolation: Access violation $778B3D63 $77888911 $778860F9 $004172B9 $00416BA6 $00490998 TABSTRACTEXTERNALTOOL__ENTERCRITICALSECTION, line 1231 of ideexterntoolintf.pas $0072B44F TEXTERNALTOOL__SETTHREAD, line 469 of exttools.pas $0072E788 TEXTERNALTOOLTHREAD__SETTOOL, line 1389 of exttools.pas $0072F5ED TEXTERNALTOOLTHREAD__DESTROY, line 1695 of exttools.pas (1002) Target OS: Win32 for i386 (3104) Compiling lazpaint.lpr $004132F2 $00409D6F $768F6359 $778B7C24 $778B7BF4 I will start my own investigation again. Sorry about this.

Cyrax 2020-06-08 06:41 reporter ~0123324	SynUniHighlighter.pas(1206) Hint: (8028) Change of bind type of symbol INIT_$SYNUNIHIGHLIGHTER_$$_TSYNUNISYN from external to global after use SynUniHighlighter.pas(1206) Hint: (8028) Change of bind type of symbol RTTI_$SYNUNIHIGHLIGHTER_$$_TSYNUNISYN from external to global after use /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,3) Hint: (5023) Unit "SynUniClasses" not used in SynUni TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlAn unhandled exception occurred at $08072419: EStackOverflow: Stack overflow $08072419 $080F05DA $080F15F9 $08072B31 $0916FF88 $087320C8 $08733287 $0806B7E2 $080A2142 $0808A545 $F7EFC687 ighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni (1008) 4023 lines compiled, 5.9 sec (1021) 3 warning(s) issued (1022) 77 hint(s) issued (1023) 9682 note(s) issued TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! An unhandled exception occurred at $080612EB: EAccessViolation: Access violation $080612EB $081DF627 $081EAE04 $081EACFF $0823A467 $0804C55F $0804BD50 $0804A47C $0804F301 $08051A67

Cyrax 2020-06-08 08:23 reporter ~0123327	Lazarus trunk r63325. FPC trunk r45588. Still getting random crashes at lazbuild. ----- Stack trace $0819E380 file: components/buildintf/ideexterntoolintf.pas : 1237; $0872E430 file: ide/exttools.pas : 475; $087320C8 file: ide/exttools.pas : 1389; $08733287 file: ide/exttools.pas : 1695; $0806B7E2 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; $080A2142 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; $0808A545 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; $F7ED7687

Juha Manninen 2020-06-08 11:04 developer ~0123331	> EStackOverflow: Stack overflow That is a new one, isn't it? Interesting. I have seen only Access violation so far in debugger backtraces of Cyrax and Alfred.

Cyrax 2020-07-13 17:58 reporter ~0123987 Last edited: 2020-07-13 18:05 View 2 revisions	The patch in https://bugs.freepascal.org/view.php?id=37345 may lower the possibility of triggering this bug, but I have to come conclusion that it is a thread race condition on freeing FWorkerMessages in TAbstractExternalTool.Destroy (components/buildintf/ideexterntoolintf.pas, line 1226) https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/buildintf/ideexterntoolintf.pas?view=markup&revision=63144&root=lazarus#l1226 EDIT : Also moving clause FTool:=nil at https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/ide/exttools.pas?view=markup&revision=63456&root=lazarus#l1389 to line 1391 may lessen crashes

Cyrax 2020-07-13 18:08 reporter ~0123988	Attached patch will move clause FTool:=nil to line 1391 in ide/exttools.pas exttools-2.pas.diff (347 bytes) diff --git a/ide/exttools.pas b/ide/exttools.pas index 59eab75172..c19cbeb569 100644 --- a/ide/exttools.pas +++ b/ide/exttools.pas @@ -1386,9 +1386,9 @@ var begin if FTool=AValue then Exit; OldTool:=FTool; - FTool:=nil; if OldTool<>nil then OldTool.Thread:=nil; + FTool:=nil; if AValue<>nil then begin FTool:=AValue; exttools-2.pas.diff (347 bytes)

Cyrax 2020-07-13 18:17 reporter ~0123989	Attached patch is an attempt to fix the race condition by creating subclass of TObjectList, which will free instance of FWorkerMessages class later time by utilising a timer functionality. ideexterntoolintf-2.pas.diff (2,299 bytes) diff --git a/components/buildintf/ideexterntoolintf.pas b/components/buildintf/ideexterntoolintf.pas index 2da4a39ad7..b1d69d1e2f 100644 --- a/components/buildintf/ideexterntoolintf.pas +++ b/components/buildintf/ideexterntoolintf.pas @@ -727,6 +727,26 @@ function dbgs(s: TExternalToolStage): string; overload; implementation +uses + contnrs, fptimer; + +type + + { TDelayedObjectFreeList } + + TDelayedObjectFreeList = class(TObjectList) + private + FFPTimer : TFPTimer; + procedure DoOnTime(Sender: TObject); + public + constructor Create; + destructor Destroy;override; + procedure Tick; + end; + +Var + ADelayedObjectFreeList : TDelayedObjectFreeList = NIL; + function CompareMsgLinesSrcPos(MsgLine1, MsgLine2: Pointer): integer; var Line1: TMessageLine absolute MsgLine1; @@ -787,6 +807,34 @@ begin WriteStr(Result,s); end; +{ TDelayedObjectFreeList } + +procedure TDelayedObjectFreeList.DoOnTime(Sender: TObject); +begin + FFPTimer.Enabled:=False; + Self.Delete(0); +end; + +constructor TDelayedObjectFreeList.Create; +begin + inherited Create(True); + FFPTimer := TFPTimer.Create(NIL); + FFPTimer.Interval := 1000; + FFPTimer.Enabled:=False; + FFPTimer.OnTimer := @DoOnTime; +end; + +destructor TDelayedObjectFreeList.Destroy; +begin + FreeAndNil(FFPTimer); + inherited Destroy; +end; + +procedure TDelayedObjectFreeList.Tick; +begin + FFPTimer.Enabled:=True; +end; + { TIDEExternalToolData } constructor TIDEExternalToolData.Create(aKind, aModuleName, aFilename: string); @@ -1197,6 +1245,7 @@ begin FEstimatedLoad:=1; FEnvironmentOverrides:=TStringList.Create; FReferences:=TStringList.Create; + ADelayedObjectFreeList.Add(FWorkerMessages); end; destructor TAbstractExternalTool.Destroy; @@ -1223,7 +1272,7 @@ begin finally LeaveCriticalsection; end; - FreeAndNil(FWorkerMessages); + ADelayedObjectFreeList.Tick; end; procedure TAbstractExternalTool.EnterCriticalSection; @@ -2523,6 +2572,10 @@ initialization // on single cores there is delay due to file reads // => use 2 processes in parallel by default DefaultMaxProcessCount:=Max(2,GetSystemThreadCount); + ADelayedObjectFreeList:=TDelayedObjectFreeList.Create; + +finalization + FreeAndNil(ADelayedObjectFreeList); end. ideexterntoolintf-2.pas.diff (2,299 bytes)

Cyrax 2020-08-16 15:27 reporter ~0124923	FPC trunk r46453. Lazarus trunk r63750. --- TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni (1008) 4023 lines compiled, 3.9 sec (1021) 4 warning(s) issued (1022) 77 hint(s) issued (1023) 9685 An unhandled exception occurred at $08072609: EStackOverflow: Stack overflow $08072609 $080F0F0A $080F1F29 $08072D1F $09E785E8 $0873ACDE $0873BEA7 $0806B942 $080A2462 $0808A885 $F7EC8687 note(s) issued TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! TProject.DoLoadStateFile Statefile not found: /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/units/i386-linux-gtk2-30301/.compiled Hint: (lazarus) Project needs building: No state file for Project An unhandled exception occurred at $0814D095: EAccessViolation: Access violation $0814D095 $0814D1C6 $08152D81 $08241B68 $082421DC $0804C8A6 $0804BD68 $0804A48C $0804F361 $08051AD7 Stack trace $08072609 file: ../../../../../../fpc/source/git_source/rtl/inc/system.inc : 1037; $080F0F0A file: ../../../../../../fpc/source/git_source/rtl/objpas/sysutils/sysutils.inc : 194; $080F1F29 file: ../../../../../../fpc/source/git_source/rtl/objpas/sysutils/sysutils.inc : 407; $08072D1F file: ../../../../../../fpc/source/git_source/rtl/inc/system.inc : 1325; $09E785E8 $0873ACDE file: ide/exttools.pas : 1390; $0873BEA7 file: ide/exttools.pas : 1697; $0806B942 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; $080A2462 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; $0808A885 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; $F7EC8687 Stack trace $0814D095 file: components/codetools/codecache.pas : 578; $0814D1C6 file: components/codetools/codecache.pas : 607; $08152D81 file: components/codetools/codetoolmanager.pas : 1264; $08241B68 file: ide/buildmanager.pas : 1902; $082421DC file: ide/buildmanager.pas : 1987; $0804C8A6 file: ide/lazbuild.lpr : 857; $0804BD68 file: ide/lazbuild.lpr : 938; $0804A48C file: ide/lazbuild.lpr : 426; $0804F361 file: ide/lazbuild.lpr : 1476; $08051AD7 file: ide/lazbuild.lpr : 1879; $00000000

Cyrax 2020-08-16 21:19 reporter ~0124930	/mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni (1008) 4023 lines compiled, 4.0 sec (1021) 4 warning(s) issued (1022) 77 hint(s) issued An unhandled exception occurred at $0877189C: EObjectCheck: Object reference is Nil $0877189C $087767EE $08777E0F $0806D882 $080A43A2 $0808C7C5 $F7F6F687 (1023) 9685 note(s) issued TExternalToolsConsole.HandleMesages: Calling CheckSynchronize! codetoolmanager.pas - finalization [TCodeToolManager.Destroy] A [TCodeToolManager.Destroy] B [TCodeToolManager.Destroy] C [TCodeToolManager.Destroy] D [TCodeToolManager.Destroy] E TProject.DoLoadStateFile Statefile not found: /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/units/i386-linux-gtk2-30301/.compiled Hint: (lazarus) Project needs building: No state file for Project >>>>>> [TCodeToolManager.LoadFile] /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/src/fmain.pas Update=True Revert=False An unhandled exception occurred at $0814EFEB: EAccessViolation: Access violation $0814EFEB $0814F1E6 $08154FD9 $08269C6B $0826A41C $0804D524 $0804C60F $0804A89F $08050AAE $08053A05 Stack trace $0877189C file: ide/exttools.pas : 475; $087767EE file: ide/exttools.pas : 1390; $08777E0F file: ide/exttools.pas : 1697; $0806D882 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; $080A43A2 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; $0808C7C5 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; $F7F6F687 Stack trace $0814EFEB file: components/codetools/codecache.pas : 578; $0814F1E6 file: components/codetools/codecache.pas : 607; $08154FD9 file: components/codetools/codetoolmanager.pas : 1264; $08269C6B file: ide/buildmanager.pas : 1902; $0826A41C file: ide/buildmanager.pas : 1987; $0804D524 file: ide/lazbuild.lpr : 857; $0804C60F file: ide/lazbuild.lpr : 938; $0804A89F file: ide/lazbuild.lpr : 426; $08050AAE file: ide/lazbuild.lpr : 1476; $08053A05 file: ide/lazbuild.lpr : 1879;

Juha Manninen 2020-10-15 14:11 developer ~0126332	Apparently fixed. Please test.

Date Modified	Username	Field	Change
2020-06-06 04:26	Cyrax	New Issue
2020-06-06 04:26	Cyrax	File Added: exttools.pas.diff
2020-06-06 08:53	Alfred	Note Added: 0123253
2020-06-06 10:04	Mattias Gaertner	Assigned To	=> Mattias Gaertner
2020-06-06 10:04	Mattias Gaertner	Status	new => resolved
2020-06-06 10:04	Mattias Gaertner	Resolution	open => fixed
2020-06-06 10:04	Mattias Gaertner	LazTarget	=> -
2020-06-06 10:04	Mattias Gaertner	Widgetset	GTK 2 => GTK 2
2020-06-06 10:04	Mattias Gaertner	Note Added: 0123254
2020-06-06 10:29	Alfred	Note Added: 0123256
2020-06-06 10:33	Alfred	Note Added: 0123257
2020-06-06 10:33	Alfred	File Added: lazbuild01.JPG
2020-06-06 11:00	Juha Manninen	Relationship added	related to 0036318
2020-06-06 11:01	Juha Manninen	Relationship added	related to 0036959
2020-06-07 07:37	Cyrax	Status	resolved => assigned
2020-06-07 07:37	Cyrax	Resolution	fixed => reopened
2020-06-07 07:37	Cyrax	Note Added: 0123289
2020-06-07 09:21	Mattias Gaertner	Status	assigned => resolved
2020-06-07 09:21	Mattias Gaertner	Fixed in Revision	=> 63321.
2020-06-07 09:21	Mattias Gaertner	Widgetset	GTK 2 => GTK 2
2020-06-07 10:46	Alfred	Note Added: 0123293
2020-06-07 11:46	Alfred	Note Added: 0123298
2020-06-08 06:41	Cyrax	Status	resolved => assigned
2020-06-08 06:41	Cyrax	Note Added: 0123324
2020-06-08 08:23	Cyrax	Note Added: 0123327
2020-06-08 11:04	Juha Manninen	Note Added: 0123331
2020-07-13 17:58	Cyrax	Note Added: 0123987
2020-07-13 18:05	Cyrax	Note Edited: 0123987	View Revisions
2020-07-13 18:08	Cyrax	Note Added: 0123988
2020-07-13 18:08	Cyrax	File Added: exttools-2.pas.diff
2020-07-13 18:17	Cyrax	Note Added: 0123989
2020-07-13 18:17	Cyrax	File Added: ideexterntoolintf-2.pas.diff
2020-08-16 15:27	Cyrax	Note Added: 0124923
2020-08-16 21:19	Cyrax	Note Added: 0124930
2020-10-15 14:11	Juha Manninen	Assigned To	Mattias Gaertner => Juha Manninen
2020-10-15 14:11	Juha Manninen	Status	assigned => resolved
2020-10-15 14:11	Juha Manninen	Fixed in Revision	63321. => r63321, r64005
2020-10-15 14:11	Juha Manninen	Widgetset	GTK 2 => GTK 2
2020-10-15 14:11	Juha Manninen	Note Added: 0126332

View Issue Details

Relationships

Activities

Issue History

related to	0036318	resolved	Juha Manninen	lazbuild crashes when compiling packages
related to	0036959	resolved	Juha Manninen	LazBuild crashes with --recursive builds