View Issue Details

IDProjectCategoryView StatusLast Update
0037179LazarusIDEpublic2020-10-15 14:11
ReporterCyrax Assigned ToJuha Manninen  
PrioritynormalSeverityminorReproducibilityrandom
Status resolvedResolutionreopened 
PlatformLinux x86_64OSArch 
Product Version2.1 (SVN) 
Summary0037179: [lazbuild] [patch] Lazbuild randomly aborts project building process.
Description
procedure TExternalToolThread.SetTool(AValue: TExternalTool);
begin
  if FTool=AValue then Exit;
  if FTool<>nil then
    FTool.Thread:=nil;
  FTool:=AValue;
  if FTool<>nil then
    FTool.Thread:=Self;
end;


destructor TExternalToolThread.Destroy;
begin
  Tool:=nil;
  inherited Destroy;
end;


Setting FTool field to NIL (aka trying to prepare it and thread associated to it to be destroyed as well) via property setter causes already running thread to product an access violation and thus breaking the build process along side of it.

Attached patch will fix this by accessing FTool field directly.
Valgrind didn't show any leaks.
TagsNo tags attached.
Fixed in Revisionr63321, r64005
LazTarget-
WidgetsetGTK 2
Attached Files

Relationships

related to 0036318 resolvedJuha Manninen lazbuild crashes when compiling packages 
related to 0036959 resolvedJuha Manninen LazBuild crashes with --recursive builds 

Activities

Cyrax

2020-06-06 04:26

reporter  

exttools.pas.diff (267 bytes)   
diff --git a/ide/exttools.pas b/ide/exttools.pas
index f81f0f27e2..598948ce2a 100644
--- a/ide/exttools.pas
+++ b/ide/exttools.pas
@@ -1678,7 +1678,7 @@ end;
 
 destructor TExternalToolThread.Destroy;
 begin
-  Tool:=nil;
+  FTool:=nil;
   inherited Destroy;
 end;
 
exttools.pas.diff (267 bytes)   

Alfred

2020-06-06 08:53

reporter   ~0123253

After applying this patch, no crashes anymore when building Lazarus.
The crash also happened very often when building lazpaint. This patch also solved the lazpaint crash.
So, it seems to work.

And.

If valgrind shows no leaks, this could mean that the issues can be considered as solved.

https://bugs.freepascal.org/view.php?id=36318
https://bugs.freepascal.org/view.php?id=36959

@juha: please review !

Mattias Gaertner

2020-06-06 10:04

manager   ~0123254

The patch omits calling SetThread. This might work in lazbuild, but not in the IDE.

CallAutoFree in SetThread was not initialized. I fixed that.

Alfred

2020-06-06 10:29

reporter   ~0123256

@Mattias
Your fix works on Windows. Even with lazbuild max-process-count set to a value > 1.
Going to test on Catalina now. Will report re-occurrence of issue.
You might close related issues I guess.

Alfred

2020-06-06 10:33

reporter   ~0123257

Celebrating too early I guess ... :-(
After 12 x building lazpaint ... it happened again unfortunately.
lazbuild01.JPG (84,991 bytes)   
lazbuild01.JPG (84,991 bytes)   

Cyrax

2020-06-07 07:37

reporter   ~0123289

Stack trace
    $081A02A0 file: components/buildintf/ideexterntoolintf.pas : 1237; 
    $08757A34 file: ide/exttools.pas : 475; 
    $0875C958 file: ide/exttools.pas : 1383; 
    $0875DF6F file: ide/exttools.pas : 1689; 
    $0806D702 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; 
    $080A4062 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; 
    $0808C465 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; 
    $F7EE8687
    $00000000

Alfred

2020-06-07 10:46

reporter   ~0123293

@Mattias.
Latest fix looks good.
25x rebuild/install of lazpaint : all ok.
Will run for max 100x.

Alfred

2020-06-07 11:46

reporter   ~0123298

After 62 times, the (repeated) failures start again.

(3104) Compiling lcvectormultishape.pas
(1008) 15619 lines compiled, 3.6 sec
(1021) 31 warning(s) issued
(1022) 226 hint(s) issued
(1023) 63 note(s) issued
An unhandled exception occurred at $778B3D63:
EAccessViolation: Access violation
  $778B3D63
  $77888911
  $778860F9
  $004172B9
  $00416BA6
  $00490998 TABSTRACTEXTERNALTOOL__ENTERCRITICALSECTION, line 1231 of ideexterntoolintf.pas
  $0072B44F TEXTERNALTOOL__SETTHREAD, line 469 of exttools.pas
  $0072E788 TEXTERNALTOOLTHREAD__SETTOOL, line 1389 of exttools.pas
  $0072F5ED TEXTERNALTOOLTHREAD__DESTROY, line 1695 of exttools.pas
(1002) Target OS: Win32 for i386
(3104) Compiling lazpaint.lpr
  $004132F2
  $00409D6F
  $768F6359
  $778B7C24
  $778B7BF4

I will start my own investigation again. Sorry about this.

Cyrax

2020-06-08 06:41

reporter   ~0123324

SynUniHighlighter.pas(1206) Hint: (8028) Change of bind type of symbol INIT_$SYNUNIHIGHLIGHTER_$$_TSYNUNISYN from external to global after use
SynUniHighlighter.pas(1206) Hint: (8028) Change of bind type of symbol RTTI_$SYNUNIHIGHLIGHTER_$$_TSYNUNISYN from external to global after use
/mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,3) Hint: (5023) Unit "SynUniClasses" not used in SynUni
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
/mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlAn unhandled exception occurred at $08072419:
EStackOverflow: Stack overflow
  $08072419
  $080F05DA
  $080F15F9
  $08072B31
  $0916FF88
  $087320C8
  $08733287
  $0806B7E2
  $080A2142
  $0808A545
  $F7EFC687

ighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni
(1008) 4023 lines compiled, 5.9 sec
(1021) 3 warning(s) issued
(1022) 77 hint(s) issued
(1023) 9682 note(s) issued
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
An unhandled exception occurred at $080612EB:
EAccessViolation: Access violation
  $080612EB
  $081DF627
  $081EAE04
  $081EACFF
  $0823A467
  $0804C55F
  $0804BD50
  $0804A47C
  $0804F301
  $08051A67

Cyrax

2020-06-08 08:23

reporter   ~0123327

Lazarus trunk r63325.
FPC trunk r45588.

Still getting random crashes at lazbuild.


-----
Stack trace
    $0819E380 file: components/buildintf/ideexterntoolintf.pas : 1237; 
    $0872E430 file: ide/exttools.pas : 475; 
    $087320C8 file: ide/exttools.pas : 1389; 
    $08733287 file: ide/exttools.pas : 1695; 
    $0806B7E2 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; 
    $080A2142 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; 
    $0808A545 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; 
    $F7ED7687

Juha Manninen

2020-06-08 11:04

developer   ~0123331

> EStackOverflow: Stack overflow

That is a new one, isn't it? Interesting.
I have seen only Access violation so far in debugger backtraces of Cyrax and Alfred.

Cyrax

2020-07-13 17:58

reporter   ~0123987

Last edited: 2020-07-13 18:05

View 2 revisions

The patch in https://bugs.freepascal.org/view.php?id=37345 may lower the possibility of triggering this bug, but I have to come conclusion that it is a thread race condition on freeing FWorkerMessages in TAbstractExternalTool.Destroy (components/buildintf/ideexterntoolintf.pas, line 1226) https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/buildintf/ideexterntoolintf.pas?view=markup&revision=63144&root=lazarus#l1226

EDIT : Also moving clause FTool:=nil at https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/ide/exttools.pas?view=markup&revision=63456&root=lazarus#l1389 to line 1391 may lessen crashes

Cyrax

2020-07-13 18:08

reporter   ~0123988

Attached patch will move clause FTool:=nil to line 1391 in ide/exttools.pas
exttools-2.pas.diff (347 bytes)   
diff --git a/ide/exttools.pas b/ide/exttools.pas
index 59eab75172..c19cbeb569 100644
--- a/ide/exttools.pas
+++ b/ide/exttools.pas
@@ -1386,9 +1386,9 @@ var
 begin
   if FTool=AValue then Exit;
   OldTool:=FTool;
-  FTool:=nil;
   if OldTool<>nil then
     OldTool.Thread:=nil;
+  FTool:=nil;
   if AValue<>nil then
     begin
     FTool:=AValue;
exttools-2.pas.diff (347 bytes)   

Cyrax

2020-07-13 18:17

reporter   ~0123989

Attached patch is an attempt to fix the race condition by creating subclass of TObjectList, which will free instance of FWorkerMessages class later time by utilising a timer functionality.
ideexterntoolintf-2.pas.diff (2,299 bytes)   
diff --git a/components/buildintf/ideexterntoolintf.pas b/components/buildintf/ideexterntoolintf.pas
index 2da4a39ad7..b1d69d1e2f 100644
--- a/components/buildintf/ideexterntoolintf.pas
+++ b/components/buildintf/ideexterntoolintf.pas
@@ -727,6 +727,26 @@ function dbgs(s: TExternalToolStage): string; overload;
 
 implementation
 
+uses
+  contnrs, fptimer;
+
+type
+
+  { TDelayedObjectFreeList }
+
+  TDelayedObjectFreeList = class(TObjectList)
+    private
+      FFPTimer : TFPTimer;
+      procedure DoOnTime(Sender: TObject);
+    public
+      constructor Create;
+      destructor Destroy;override;
+      procedure Tick;
+  end;
+
+Var
+  ADelayedObjectFreeList : TDelayedObjectFreeList = NIL;
+
 function CompareMsgLinesSrcPos(MsgLine1, MsgLine2: Pointer): integer;
 var
   Line1: TMessageLine absolute MsgLine1;
@@ -787,6 +807,34 @@ begin
   WriteStr(Result,s);
 end;
 
+{ TDelayedObjectFreeList }
+
+procedure TDelayedObjectFreeList.DoOnTime(Sender: TObject);
+begin
+  FFPTimer.Enabled:=False;
+  Self.Delete(0);
+end;
+
+constructor TDelayedObjectFreeList.Create;
+begin
+  inherited Create(True);
+  FFPTimer := TFPTimer.Create(NIL);
+  FFPTimer.Interval := 1000;
+  FFPTimer.Enabled:=False;
+  FFPTimer.OnTimer := @DoOnTime;
+end;
+
+destructor TDelayedObjectFreeList.Destroy;
+begin
+  FreeAndNil(FFPTimer);
+  inherited Destroy;
+end;
+
+procedure TDelayedObjectFreeList.Tick;
+begin
+  FFPTimer.Enabled:=True;
+end;
+
 { TIDEExternalToolData }
 
 constructor TIDEExternalToolData.Create(aKind, aModuleName, aFilename: string);
@@ -1197,6 +1245,7 @@ begin
   FEstimatedLoad:=1;
   FEnvironmentOverrides:=TStringList.Create;
   FReferences:=TStringList.Create;
+  ADelayedObjectFreeList.Add(FWorkerMessages);
 end;
 
 destructor TAbstractExternalTool.Destroy;
@@ -1223,7 +1272,7 @@ begin
   finally
     LeaveCriticalsection;
   end;
-  FreeAndNil(FWorkerMessages);
+  ADelayedObjectFreeList.Tick;
 end;
 
 procedure TAbstractExternalTool.EnterCriticalSection;
@@ -2523,6 +2572,10 @@ initialization
   // on single cores there is delay due to file reads
   // => use 2 processes in parallel by default
   DefaultMaxProcessCount:=Max(2,GetSystemThreadCount);
+  ADelayedObjectFreeList:=TDelayedObjectFreeList.Create;
+
+finalization
+  FreeAndNil(ADelayedObjectFreeList);
 
 end.
 
ideexterntoolintf-2.pas.diff (2,299 bytes)   

Cyrax

2020-08-16 15:27

reporter   ~0124923

FPC trunk r46453.
Lazarus trunk r63750.

---

TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
/mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni
(1008) 4023 lines compiled, 3.9 sec
(1021) 4 warning(s) issued
(1022) 77 hint(s) issued
(1023) 9685 An unhandled exception occurred at $08072609:
EStackOverflow: Stack overflow
  $08072609
  $080F0F0A
  $080F1F29
  $08072D1F
  $09E785E8
  $0873ACDE
  $0873BEA7
  $0806B942
  $080A2462
  $0808A885
  $F7EC8687

note(s) issued
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
TProject.DoLoadStateFile Statefile not found: /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/units/i386-linux-gtk2-30301/.compiled
Hint: (lazarus) Project needs building: No state file for Project
An unhandled exception occurred at $0814D095:
EAccessViolation: Access violation
  $0814D095
  $0814D1C6
  $08152D81
  $08241B68
  $082421DC
  $0804C8A6
  $0804BD68
  $0804A48C
  $0804F361
  $08051AD7


Stack trace
    $08072609 file: ../../../../../../fpc/source/git_source/rtl/inc/system.inc : 1037; 
    $080F0F0A file: ../../../../../../fpc/source/git_source/rtl/objpas/sysutils/sysutils.inc : 194; 
    $080F1F29 file: ../../../../../../fpc/source/git_source/rtl/objpas/sysutils/sysutils.inc : 407; 
    $08072D1F file: ../../../../../../fpc/source/git_source/rtl/inc/system.inc : 1325; 
    $09E785E8
    $0873ACDE file: ide/exttools.pas : 1390; 
    $0873BEA7 file: ide/exttools.pas : 1697; 
    $0806B942 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; 
    $080A2462 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; 
    $0808A885 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; 
    $F7EC8687

Stack trace
    $0814D095 file: components/codetools/codecache.pas : 578; 
    $0814D1C6 file: components/codetools/codecache.pas : 607; 
    $08152D81 file: components/codetools/codetoolmanager.pas : 1264; 
    $08241B68 file: ide/buildmanager.pas : 1902; 
    $082421DC file: ide/buildmanager.pas : 1987; 
    $0804C8A6 file: ide/lazbuild.lpr : 857; 
    $0804BD68 file: ide/lazbuild.lpr : 938; 
    $0804A48C file: ide/lazbuild.lpr : 426; 
    $0804F361 file: ide/lazbuild.lpr : 1476; 
    $08051AD7 file: ide/lazbuild.lpr : 1879; 
    $00000000

Cyrax

2020-08-16 21:19

reporter   ~0124930

/mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/components/synunihighlighter/synuni.pas(10,37) Hint: (5023) Unit "SynUniRules" not used in SynUni
(1008) 4023 lines compiled, 4.0 sec
(1021) 4 warning(s) issued
(1022) 77 hint(s) issued
An unhandled exception occurred at $0877189C:
EObjectCheck: Object reference is Nil
  $0877189C
  $087767EE
  $08777E0F
  $0806D882
  $080A43A2
  $0808C7C5
  $F7F6F687

(1023) 9685 note(s) issued
TExternalToolsConsole.HandleMesages: Calling CheckSynchronize!
codetoolmanager.pas - finalization
[TCodeToolManager.Destroy] A
[TCodeToolManager.Destroy] B
[TCodeToolManager.Destroy] C
[TCodeToolManager.Destroy] D
[TCodeToolManager.Destroy] E
TProject.DoLoadStateFile Statefile not found: /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/units/i386-linux-gtk2-30301/.compiled
Hint: (lazarus) Project needs building: No state file for Project
>>>>>> [TCodeToolManager.LoadFile] /mnt/shares/ohjelmointi/32-bit/kielet/pascal/free_pascal/lazarus/projects_from_others/double_commander/git_double_commander/source/src/fmain.pas Update=True Revert=False
An unhandled exception occurred at $0814EFEB:
EAccessViolation: Access violation
  $0814EFEB
  $0814F1E6
  $08154FD9
  $08269C6B
  $0826A41C
  $0804D524
  $0804C60F
  $0804A89F
  $08050AAE
  $08053A05


Stack trace
    $0877189C file: ide/exttools.pas : 475; 
    $087767EE file: ide/exttools.pas : 1390; 
    $08777E0F file: ide/exttools.pas : 1697; 
    $0806D882 file: ../../../../../../fpc/source/git_source/rtl/inc/objpas.inc : 336; 
    $080A43A2 file: ../../../../../../fpc/source/git_source/rtl/unix/tthread.inc : 137; 
    $0808C7C5 file: ../../../../../../fpc/source/git_source/rtl/unix/cthreads.pp : 317; 
    $F7F6F687


Stack trace
    $0814EFEB file: components/codetools/codecache.pas : 578; 
    $0814F1E6 file: components/codetools/codecache.pas : 607; 
    $08154FD9 file: components/codetools/codetoolmanager.pas : 1264; 
    $08269C6B file: ide/buildmanager.pas : 1902; 
    $0826A41C file: ide/buildmanager.pas : 1987; 
    $0804D524 file: ide/lazbuild.lpr : 857; 
    $0804C60F file: ide/lazbuild.lpr : 938; 
    $0804A89F file: ide/lazbuild.lpr : 426; 
    $08050AAE file: ide/lazbuild.lpr : 1476; 
    $08053A05 file: ide/lazbuild.lpr : 1879; 

Juha Manninen

2020-10-15 14:11

developer   ~0126332

Apparently fixed. Please test.

Issue History

Date Modified Username Field Change
2020-06-06 04:26 Cyrax New Issue
2020-06-06 04:26 Cyrax File Added: exttools.pas.diff
2020-06-06 08:53 Alfred Note Added: 0123253
2020-06-06 10:04 Mattias Gaertner Assigned To => Mattias Gaertner
2020-06-06 10:04 Mattias Gaertner Status new => resolved
2020-06-06 10:04 Mattias Gaertner Resolution open => fixed
2020-06-06 10:04 Mattias Gaertner LazTarget => -
2020-06-06 10:04 Mattias Gaertner Widgetset GTK 2 => GTK 2
2020-06-06 10:04 Mattias Gaertner Note Added: 0123254
2020-06-06 10:29 Alfred Note Added: 0123256
2020-06-06 10:33 Alfred Note Added: 0123257
2020-06-06 10:33 Alfred File Added: lazbuild01.JPG
2020-06-06 11:00 Juha Manninen Relationship added related to 0036318
2020-06-06 11:01 Juha Manninen Relationship added related to 0036959
2020-06-07 07:37 Cyrax Status resolved => assigned
2020-06-07 07:37 Cyrax Resolution fixed => reopened
2020-06-07 07:37 Cyrax Note Added: 0123289
2020-06-07 09:21 Mattias Gaertner Status assigned => resolved
2020-06-07 09:21 Mattias Gaertner Fixed in Revision => 63321.
2020-06-07 09:21 Mattias Gaertner Widgetset GTK 2 => GTK 2
2020-06-07 10:46 Alfred Note Added: 0123293
2020-06-07 11:46 Alfred Note Added: 0123298
2020-06-08 06:41 Cyrax Status resolved => assigned
2020-06-08 06:41 Cyrax Note Added: 0123324
2020-06-08 08:23 Cyrax Note Added: 0123327
2020-06-08 11:04 Juha Manninen Note Added: 0123331
2020-07-13 17:58 Cyrax Note Added: 0123987
2020-07-13 18:05 Cyrax Note Edited: 0123987 View Revisions
2020-07-13 18:08 Cyrax Note Added: 0123988
2020-07-13 18:08 Cyrax File Added: exttools-2.pas.diff
2020-07-13 18:17 Cyrax Note Added: 0123989
2020-07-13 18:17 Cyrax File Added: ideexterntoolintf-2.pas.diff
2020-08-16 15:27 Cyrax Note Added: 0124923
2020-08-16 21:19 Cyrax Note Added: 0124930
2020-10-15 14:11 Juha Manninen Assigned To Mattias Gaertner => Juha Manninen
2020-10-15 14:11 Juha Manninen Status assigned => resolved
2020-10-15 14:11 Juha Manninen Fixed in Revision 63321. => r63321, r64005
2020-10-15 14:11 Juha Manninen Widgetset GTK 2 => GTK 2
2020-10-15 14:11 Juha Manninen Note Added: 0126332