View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0021868||FPC||Utilities||public||2012-04-26 15:50||2018-12-13 15:12|
|Reporter||Marco van de Voort||Assigned To||Joost van der Sluis|
|Priority||normal||Severity||block||Reproducibility||have not tried|
|Platform||windows 7/32||OS||OS Version|
|Product Version||2.7.1||Product Build||trunk of last weeks.|
|Target Version||Fixed in Version||3.2.0|
|Summary||0021868: fpmake terminates while clean repo|
|Description||On Windows: make clean terminates with|
.\fpmake.exe distclean --localunitdir=.. --os=win32 --cpu=i386 -o -Ur -o -Xs -o -O2 -o -n -o -Fud:/repo/fpc/rtl -o -O- -o -gl -o -dUse_MINGW_GDB -o -di386 -o -d
RELEASE --compiler=d:/fpc/2.6.0/bin/i386-win32/ppc386.exe -bu
Clean of package a52 completed
Clean of package amunits completed
Clean of package aspell completed
Clean of package bfd completed
The installer encountered the following error:
Failed to remove directory "units"
* no make -j used
* restarting the build a few time allows to run to completion. So whatever it is, it is transient. It also seems to be limited to cleaning in packages/
* I had this from time to time, but since I put "Windows Defender"'s realtime protection back on, this seems to happen more often then not. The link with "Windows Defender" is a strong suspicion, but not a hard fact.
Possible solution pauze and retry a few times instead before terminating, to allow transient locks to settle.
|Fixed in Revision||40520|
||I experience (very rarely) this issue on WinXP, which does not have Windows Defender.|
||tortoisesvn's svncache is also a possible culprit. Anything that keeps watches on activity in folders.|
||I played a bit with proc(es)exp(lorer) during make clean, and the windows defender process started eating CPU....|
||In the last month the problem seems to have disappeared. Don't know why.|
I think it has to do with the invalid check on the age of a file, which is removed. Maybe that invalid check leads to a lock on the file...
If you experience this problem once again, let me know.
||Haven't seen it since. Even on a new windows install with windows defender on again.|
I had simlar trouble on FreeBSD for a while, and now finally caught it in logs:
Clean of package cairo completed
Clean of package cdrom completed
Clean of package chm completed
Clean of package cocoaint completed
Clean of package dblib completed
Clean of package dbus completed
Clean of package dts completed
Clean of package fastcgi completed
Clean of package fcl-async completed
The installer encountered the following error:
Failed to remove directory "units/x86_64-freebsd"
Something wrong with fpmake exectable. Remove the executable and call make recursively to recover.
Like in the Windows case, a second run of exactly the same fpmake works fine.
The relevant dirs are on NFS though.
When I wrote above report I added some writelns in fpmkunit's intremovedir to see which caused the problem.
It's the removedir at the end of intremovedir, and the reason is errno=66 (not empty).
I haven't been able to figure out yet what file remains in that directory, the last few times it was always fcl-async btw.
But probably an error like this shouldn't abort the recursive cleaning. Packages after the error location are not cleaned this way. Add an ignore errors boolean to intremovedir or so, and use it in case of clean?
This error also happens a lot with Jenkins.
Probably in its build scripts, the clean should be restarted once if failed.
||Still visible in todays build|
||Status changed to block, since it is a blocking issue to make a release based on triunk|
||I've added a 5 seconds delay and a retry when RemoveDir failes in r29206. I can not reproduce the problem, so it would be nice if some people could test, and report back when the problem is still there...|
||I haven't seen it recently, but I don't use the relevant system much atm.|
||Maybe we could use SHFileOperation (http://ibeblog.com/2011/05/29/delphi-remove-directory-recursively/ ) on Windows to avoid such problems.|
@Jonas: Just as a precaution:
I think that is a good option, but note that your link points to a function with a bug in it. The buffer for the structure should be of size MAX_PATH+2, so 0.. MAX_PATH+1 instead of 0..255. See http://msdn.microsoft.com/en-us/library/windows/desktop/bb759795(v=vs.85).aspx
Afaik the double 0 terminator also applies to removal.
I've implemented this in r29284, although I do not like to add platform-specific code to fpmkunit.
Btw: Microsoft's documentation explicitly demands the double 0 terminator. But I don't see this one in the code, the fillchar takes care of that? So what did you mean?
If it is about the buffer length bug in the example code see:
A buffer of 256 bytes is simply too small.
Even if taken into account 256 - length('<driveletter>:\') the minimum to provide as buffer is the canonical name and max_path (like also '\\?\') including two zero's equals 262.
Still happening on FreeBSD sometimes, though no longer catastrophic. A manual clean worked, so it is still due to something transient. (there is nfs involved on this system as you can see from the .mnt extension)
Log excerpt at http://www.stack.nl/~marcov/fpmakefailoutput.txt
||Got an abortive build today due to the failed removal of a buildunit.|
In recent days (bisecting because of the textmode IDE ini problem) I saw this more often. Maybe removing (autogenerated?) buildunits do not have the delete guards that cleaning out units/ dir have?
Failed to delete file "h2pas\BuildUnit_utils_h2pas.pp"
Rerunning simply ran to completion. Probably we really must add the delete retrying to each and every deletion.
Note that the SVN activity before each build (tortoisesvn icon cache) might be the trigger of the issue. It was not rare, I guess 1 in 4 builds. No antivirus or other disturbing agents, which points to svn
||I suspect there is just a single zero instead of a double zero. That may or may not work. A double zero terminator works.|
Since I could reproduce the buildunits delete problem (on win32) quite well the last few days, I tried a fix (for sysdeletefiles, similar to the one in removedirs, which seemed to have solved that problem). Initially it seems to work, but will keep an eye on it for a few days.
The NFS using stack systems are no more, so if the buildunit problem is indeed gone, I hope we can finally close this.
||After the change no more problems for two days on multiple machines. Assume fixed for now.|
|2012-04-26 15:50||Marco van de Voort||New Issue|
|2012-04-26 15:50||Marco van de Voort||FPCOldBugId||=> 0|
|2012-04-26 15:50||Marco van de Voort||Status||new => assigned|
|2012-04-26 15:50||Marco van de Voort||Assigned To||=> Joost van der Sluis|
|2012-04-26 15:51||Marco van de Voort||Description Updated|
|2012-04-26 17:17||Sergei Gorelkin||Note Added: 0059022|
|2012-04-26 18:01||Marco van de Voort||Note Added: 0059026|
|2012-05-16 23:00||Marco van de Voort||Note Added: 0059656|
|2012-06-04 21:30||Florian||Tag Attached: fpmake|
|2012-11-01 16:40||Marco van de Voort||Note Added: 0063611|
|2012-11-11 16:45||Joost van der Sluis||Note Added: 0063803|
|2012-12-28 23:33||Marco van de Voort||Note Added: 0064518|
|2013-03-10 13:56||Marco van de Voort||Note Added: 0066168|
|2013-03-27 14:14||Marco van de Voort||Note Added: 0066586|
|2013-04-22 17:43||Marco van de Voort||Note Added: 0067087|
|2013-10-17 21:20||Marco van de Voort||Note Added: 0070865|
|2013-10-17 21:21||Marco van de Voort||Note Added: 0070866|
|2013-10-17 21:21||Marco van de Voort||Severity||minor => block|
|2014-12-06 11:41||Joost van der Sluis||Note Added: 0079672|
|2014-12-06 17:02||Marco van de Voort||Note Added: 0079683|
|2014-12-06 18:31||Jonas Maebe||Note Added: 0079684|
|2014-12-07 09:23||Thaddy de Koning||Note Added: 0079689|
|2014-12-07 09:26||Thaddy de Koning||Note Edited: 0079689||View Revisions|
|2014-12-14 11:54||Joost van der Sluis||Note Added: 0079799|
|2014-12-14 14:10||Thaddy de Koning||Note Added: 0079803|
|2014-12-14 16:56||Thaddy de Koning||Note Edited: 0079803||View Revisions|
|2015-01-19 20:39||Marco van de Voort||Note Added: 0080518|
|2015-01-19 20:40||Marco van de Voort||Note Edited: 0080518||View Revisions|
|2015-03-26 16:02||Marco van de Voort||Note Added: 0082340|
|2015-04-10 16:49||Marco van de Voort||Note Added: 0082788|
|2015-04-10 16:59||Marco van de Voort||Note Edited: 0082788||View Revisions|
|2015-04-10 17:08||Thaddy de Koning||Note Added: 0082789|
|2015-04-10 17:58||Marco van de Voort||Note Edited: 0082788||View Revisions|
|2018-12-10 23:47||Marco van de Voort||Note Added: 0112481|
|2018-12-13 15:12||Marco van de Voort||Fixed in Revision||=> 40520|
|2018-12-13 15:12||Marco van de Voort||Note Added: 0112533|
|2018-12-13 15:12||Marco van de Voort||Status||assigned => resolved|
|2018-12-13 15:12||Marco van de Voort||Fixed in Version||=> 3.2.0|
|2018-12-13 15:12||Marco van de Voort||Resolution||open => fixed|