View Issue Details

IDProjectCategoryView StatusLast Update
0021868FPCUtilitiespublic2018-12-13 15:12
ReporterMarco van de VoortAssigned ToJoost van der Sluis 
PrioritynormalSeverityblockReproducibilityhave not tried
Status resolvedResolutionfixed 
Platformwindows 7/32OSOS Version
Product Version2.7.1Product Buildtrunk of last weeks. 
Target VersionFixed in Version3.2.0 
Summary0021868: fpmake terminates while clean repo
DescriptionOn Windows: make clean terminates with

.\fpmake.exe distclean --localunitdir=.. --os=win32 --cpu=i386 -o -Ur -o -Xs -o -O2 -o -n -o -Fud:/repo/fpc/rtl -o -O- -o -gl -o -dUse_MINGW_GDB -o -di386 -o -d
RELEASE --compiler=d:/fpc/2.6.0/bin/i386-win32/ppc386.exe -bu
Clean of package a52 completed
Clean of package amunits completed
Clean of package aspell completed
Clean of package bfd completed
The installer encountered the following error:
Failed to remove directory "units"

* no make -j used
* restarting the build a few time allows to run to completion. So whatever it is, it is transient. It also seems to be limited to cleaning in packages/
* I had this from time to time, but since I put "Windows Defender"'s realtime protection back on, this seems to happen more often then not. The link with "Windows Defender" is a strong suspicion, but not a hard fact.

Possible solution pauze and retry a few times instead before terminating, to allow transient locks to settle.
Tagsfpmake
Fixed in Revision40520
FPCOldBugId0
FPCTarget
Attached Files

Activities

Sergei Gorelkin

2012-04-26 17:17

developer   ~0059022

I experience (very rarely) this issue on WinXP, which does not have Windows Defender.

Marco van de Voort

2012-04-26 18:01

manager   ~0059026

tortoisesvn's svncache is also a possible culprit. Anything that keeps watches on activity in folders.

Marco van de Voort

2012-05-16 23:00

manager   ~0059656

I played a bit with proc(es)exp(lorer) during make clean, and the windows defender process started eating CPU....

Marco van de Voort

2012-11-01 16:40

manager   ~0063611

In the last month the problem seems to have disappeared. Don't know why.

Joost van der Sluis

2012-11-11 16:45

manager   ~0063803

I think it has to do with the invalid check on the age of a file, which is removed. Maybe that invalid check leads to a lock on the file...

If you experience this problem once again, let me know.

Marco van de Voort

2012-12-28 23:33

manager   ~0064518

Haven't seen it since. Even on a new windows install with windows defender on again.

Marco van de Voort

2013-03-10 13:56

manager   ~0066168

I had simlar trouble on FreeBSD for a while, and now finally caught it in logs:

Clean of package cairo completed
Clean of package cdrom completed
Clean of package chm completed
Clean of package cocoaint completed
Clean of package dblib completed
Clean of package dbus completed
Clean of package dts completed
Clean of package fastcgi completed
Clean of package fcl-async completed
The installer encountered the following error:
Failed to remove directory "units/x86_64-freebsd"
Something wrong with fpmake exectable. Remove the executable and call make recursively to recover.

Like in the Windows case, a second run of exactly the same fpmake works fine.

The relevant dirs are on NFS though.

Marco van de Voort

2013-03-27 14:14

manager   ~0066586

When I wrote above report I added some writelns in fpmkunit's intremovedir to see which caused the problem.

It's the removedir at the end of intremovedir, and the reason is errno=66 (not empty).

I haven't been able to figure out yet what file remains in that directory, the last few times it was always fcl-async btw.

But probably an error like this shouldn't abort the recursive cleaning. Packages after the error location are not cleaned this way. Add an ignore errors boolean to intremovedir or so, and use it in case of clean?

Marco van de Voort

2013-04-22 17:43

manager   ~0067087

This error also happens a lot with Jenkins.

E.g. https://fpcbuildserver.firmos.at/job/buildfpc_branch/./platform=i386-win32/643/console

Probably in its build scripts, the clean should be restarted once if failed.

Marco van de Voort

2013-10-17 21:20

manager   ~0070865

Still visible in todays build

Marco van de Voort

2013-10-17 21:21

manager   ~0070866

Status changed to block, since it is a blocking issue to make a release based on triunk

Joost van der Sluis

2014-12-06 11:41

manager   ~0079672

I've added a 5 seconds delay and a retry when RemoveDir failes in r29206. I can not reproduce the problem, so it would be nice if some people could test, and report back when the problem is still there...

Marco van de Voort

2014-12-06 17:02

manager   ~0079683

I haven't seen it recently, but I don't use the relevant system much atm.

Jonas Maebe

2014-12-06 18:31

manager   ~0079684

Maybe we could use SHFileOperation (http://ibeblog.com/2011/05/29/delphi-remove-directory-recursively/ ) on Windows to avoid such problems.

Thaddy de Koning

2014-12-07 09:23

reporter   ~0079689

Last edited: 2014-12-07 09:26

View 2 revisions

@Jonas: Just as a precaution:
I think that is a good option, but note that your link points to a function with a bug in it. The buffer for the structure should be of size MAX_PATH+2, so 0.. MAX_PATH+1 instead of 0..255. See http://msdn.microsoft.com/en-us/library/windows/desktop/bb759795(v=vs.85).aspx

Afaik the double 0 terminator also applies to removal.

Joost van der Sluis

2014-12-14 11:54

manager   ~0079799

I've implemented this in r29284, although I do not like to add platform-specific code to fpmkunit.

Btw: Microsoft's documentation explicitly demands the double 0 terminator. But I don't see this one in the code, the fillchar takes care of that? So what did you mean?

Thaddy de Koning

2014-12-14 14:10

reporter   ~0079803

Last edited: 2014-12-14 16:56

View 2 revisions

If it is about the buffer length bug in the example code see:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath
A buffer of 256 bytes is simply too small.
Even if taken into account 256 - length('<driveletter>:\') the minimum to provide as buffer is the canonical name and max_path (like also '\\?\') including two zero's equals 262.

Marco van de Voort

2015-01-19 20:39

manager   ~0080518

Last edited: 2015-01-19 20:40

View 2 revisions

Still happening on FreeBSD sometimes, though no longer catastrophic. A manual clean worked, so it is still due to something transient. (there is nfs involved on this system as you can see from the .mnt extension)

Log excerpt at http://www.stack.nl/~marcov/fpmakefailoutput.txt

Marco van de Voort

2015-03-26 16:02

manager   ~0082340

Got an abortive build today due to the failed removal of a buildunit.

Marco van de Voort

2015-04-10 16:49

manager   ~0082788

Last edited: 2015-04-10 17:58

View 3 revisions

In recent days (bisecting because of the textmode IDE ini problem) I saw this more often. Maybe removing (autogenerated?) buildunits do not have the delete guards that cleaning out units/ dir have?

e.g.:

Failed to delete file "h2pas\BuildUnit_utils_h2pas.pp"

Rerunning simply ran to completion. Probably we really must add the delete retrying to each and every deletion.

Note that the SVN activity before each build (tortoisesvn icon cache) might be the trigger of the issue. It was not rare, I guess 1 in 4 builds. No antivirus or other disturbing agents, which points to svn

Thaddy de Koning

2015-04-10 17:08

reporter   ~0082789

I suspect there is just a single zero instead of a double zero. That may or may not work. A double zero terminator works.

Marco van de Voort

2018-12-10 23:47

manager   ~0112481

Since I could reproduce the buildunits delete problem (on win32) quite well the last few days, I tried a fix (for sysdeletefiles, similar to the one in removedirs, which seemed to have solved that problem). Initially it seems to work, but will keep an eye on it for a few days.

The NFS using stack systems are no more, so if the buildunit problem is indeed gone, I hope we can finally close this.

Marco van de Voort

2018-12-13 15:12

manager   ~0112533

After the change no more problems for two days on multiple machines. Assume fixed for now.

Issue History

Date Modified Username Field Change
2012-04-26 15:50 Marco van de Voort New Issue
2012-04-26 15:50 Marco van de Voort FPCOldBugId => 0
2012-04-26 15:50 Marco van de Voort Status new => assigned
2012-04-26 15:50 Marco van de Voort Assigned To => Joost van der Sluis
2012-04-26 15:51 Marco van de Voort Description Updated
2012-04-26 17:17 Sergei Gorelkin Note Added: 0059022
2012-04-26 18:01 Marco van de Voort Note Added: 0059026
2012-05-16 23:00 Marco van de Voort Note Added: 0059656
2012-06-04 21:30 Florian Tag Attached: fpmake
2012-11-01 16:40 Marco van de Voort Note Added: 0063611
2012-11-11 16:45 Joost van der Sluis Note Added: 0063803
2012-12-28 23:33 Marco van de Voort Note Added: 0064518
2013-03-10 13:56 Marco van de Voort Note Added: 0066168
2013-03-27 14:14 Marco van de Voort Note Added: 0066586
2013-04-22 17:43 Marco van de Voort Note Added: 0067087
2013-10-17 21:20 Marco van de Voort Note Added: 0070865
2013-10-17 21:21 Marco van de Voort Note Added: 0070866
2013-10-17 21:21 Marco van de Voort Severity minor => block
2014-12-06 11:41 Joost van der Sluis Note Added: 0079672
2014-12-06 17:02 Marco van de Voort Note Added: 0079683
2014-12-06 18:31 Jonas Maebe Note Added: 0079684
2014-12-07 09:23 Thaddy de Koning Note Added: 0079689
2014-12-07 09:26 Thaddy de Koning Note Edited: 0079689 View Revisions
2014-12-14 11:54 Joost van der Sluis Note Added: 0079799
2014-12-14 14:10 Thaddy de Koning Note Added: 0079803
2014-12-14 16:56 Thaddy de Koning Note Edited: 0079803 View Revisions
2015-01-19 20:39 Marco van de Voort Note Added: 0080518
2015-01-19 20:40 Marco van de Voort Note Edited: 0080518 View Revisions
2015-03-26 16:02 Marco van de Voort Note Added: 0082340
2015-04-10 16:49 Marco van de Voort Note Added: 0082788
2015-04-10 16:59 Marco van de Voort Note Edited: 0082788 View Revisions
2015-04-10 17:08 Thaddy de Koning Note Added: 0082789
2015-04-10 17:58 Marco van de Voort Note Edited: 0082788 View Revisions
2018-12-10 23:47 Marco van de Voort Note Added: 0112481
2018-12-13 15:12 Marco van de Voort Fixed in Revision => 40520
2018-12-13 15:12 Marco van de Voort Note Added: 0112533
2018-12-13 15:12 Marco van de Voort Status assigned => resolved
2018-12-13 15:12 Marco van de Voort Fixed in Version => 3.2.0
2018-12-13 15:12 Marco van de Voort Resolution open => fixed