possible problem with dynamic arrays are a lot of use
Original Reporter info from Mantis: DrJohn
-
Reporter name: John
Original Reporter info from Mantis: DrJohn
- Reporter name: John
Description:
I think that there is a problem with dynamic arrays, but I’m not 100% sure. I’m getting exceptions/crashes when I call setlength to set the initial size of a dynamic array, or perhaps to change its size. Here are some details.
I’m writing an image processing application. It takes scanned images and, in part, removes dirt/noise/crud from those images prior to OCR. The images are loaded from files as contiguous bitmaps (dynamic arrays of bytes) and converted to a series of lines. Hence, a page comprises a dynamic array of lines, each of which comprises a dynamic array of pixels (which, in my case, are bytes). I chose to organize the data in this way because the nature of the image processing that I was doing is somewhat non-linear and it is much easier to write code that addresses pixels by their X, Y indices, and it seems like it runs faster too.
The algorithms seem to work well the vast majority of the time, but occasionally, when I call setlength, the application generates an exception. This can be after many pages and therefore very many calls to setlength, e.g., 500,000 (a number suspiciously close to 2 ^ 19). However, in an early incarnation of the software, I successfully loaded all of the pages (400+) into RAM prior to doing any image processing (an approach that I abandoned after I determined that it was slower than doing the image processing on a page by page basis, reading and writing the files before and after respectively), so I don’t believe that the problem is directly related to number of calls to setlength per se.
For awhile, I wondered about whether there is enough “time” for the memory management algorithms to complete between calls to setlength, but after having introduced Application.ProcessMessages after calls to setlength without effect, I’m pretty sure that this is not the problem.
Steps to reproduce:
run the software on many scanned pages
Additional information:
Below, find some call stack traces that I’ve recorded.
#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
#1 ?? at :0
#2 ?? at :0
#3 .Ld241 at :0
#4 ?? at :0
#5 ?? at :0
#6 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
#7 ?? at :0
#8 ?? at :0
#9 ?? at :0
#10 SYSTEM_$$_GETMEM$POINTER$QWORD at :0
#11 ?? at :0
#12 fpc_dynarray_setlength at :0
#13 WRITE_TIFF_FILE({FILE_NAME = 0x7ffff7fca9d8 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups/cleaned/page-238.tif', FILE_SIZE = 3018112, WAS_ERROR = false, LAST_ERROR = 0x0, IMAGE_COUNT = 1, IMAGES = 0x7ffff7fd0530}) at sf_tiff_io.pas:819
#14 WRITE_PAGE({FILE_NAME = 0x7ffff7feedd8 'page-238.tif', SAMPLES_PER_PIXEL = 1, BITS_PER_SAMPLE = 1, PAGE = 0x7fffd1793960}) at do_cleanup_scans.pas:260
#15 CLEAN_BOOK at do_cleanup_scans.pas:379
#16 DIRECTORYEDIT1CHANGE(0x7ffff7faaad0, 0x7ffff7fa15d0) at do_cleanup_scans.pas:456
#17 EDITCHANGE(0x7ffff7fa15d0) at editbtn.pas:1715
#18 INTERNALONEDITCHANGE(0x7ffff7fa15d0, 0x7ffff7fa1d30) at editbtn.pas:1128
#19 CHANGE(0x7ffff7fa1d30) at include/customedit.inc:582
#20 CHANGE(0x7ffff7fa1d30) at maskedit.pp:1592
#21 TEXTCHANGED(0x7ffff7fa1d30) at include/customedit.inc:574
#22 TEXTCHANGED(0x7ffff7fa1d30) at maskedit.pp:1577
#23 CMTEXTCHANGED(0x7ffff7fa1d30, {MSG = 45074, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 0, RESULT = 0}) at include/control.inc:1109
#24 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0
#25 .Ld240 at :0
#26 .Ld239 at :0
#27 ?? at :0
#28 ?? at :0
#29 ?? at :0
#30 ?? at :0
#31 WSREGISTERCLASS(0x7fffffffd1c0) at include/speedbutton.inc:852
#32 WNDPROC(0x7ffff7fa1d30, {MSG = 45074, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 0, RESULT = 0}) at include/wincontrol.inc:5383
#33 DELIVERMESSAGE(0x7ffff7fa1d30, 0) at lclmessageglue.pas:112
#34 SETTEXT(0x7ffff7fa3370, 0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at gtk2wsstdctrls.pp:1113
#35 WSSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/wincontrol.inc:5388
#36 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/wincontrol.inc:8231
#37 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/customedit.inc:523
#38 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at maskedit.pp:1076
#39 SETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/control.inc:4782
#40 SETTEXT(0x7ffff7fa15d0, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at editbtn.pas:1677
#41 SETDIRECTORY(0x7ffff7fa15d0, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at editbtn.pas:2424
#42 RUNDIALOG(0x7ffff7fa15d0) at editbtn.pas:2491
#43 BUTTONCLICK(0x7ffff7fa15d0) at editbtn.pas:2453
#44 INTERNALONBUTTONCLICK(0x7ffff7fa15d0, 0x7ffff7fa0010) at editbtn.pas:1122
#45 CLICK(0x7ffff7fa0010) at include/control.inc:2736
#46 CLICK(0x7ffff7fa0010) at include/speedbutton.inc:111
#47 WMLBUTTONUP(0x7ffff7fa0010, {MSG = 514, KEYS = 0, XPOS = 18, YPOS = 13, POS = {X = 18, Y = 13}, DUMMY = 851986, RESULT = 0}) at include/speedbutton.inc:907
#48 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0
#49 .Ld21 at :0
#50 .Ld20 at :0
#51 ?? at :0
#52 ?? at :0
#53 ?? at :0
#54 ?? at :0
#55 ?? at :0
#56 ?? at :0
#57 ?? at :0
#58 ?? at :0
#59 ?? at :0
#60 WNDPROC(0x7ffff7fa0010, {MSG = 514, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 851986, RESULT = 0}) at include/control.inc:2124
#61 PERFORM(0x7ffff7fa0010, 514, 0, 851986) at include/control.inc:1453
#62 ISCONTROLMOUSEMSG(0x7ffff7fa15d0, 0) at include/wincontrol.inc:4713
#63 WNDPROC(0x7ffff7fa15d0, {MSG = 514, UNUSEDMSG = 32767, WPARAM = 0, LPARAM = 852667, RESULT = 0}) at include/wincontrol.inc:5348
#64 DELIVERMESSAGE(0x7ffff7fa15d0, 0) at lclmessageglue.pas:112
#65 DELIVERMESSAGE(0x7ffff7fa15d0, 0) at gtk2proc.inc:3628
#66 DELIVERMOUSEUPMESSAGE(0xc47a20, 0xd500f0, 0x7ffff7fa15d0) at gtk2callback.inc:2361
#67 GTKMOUSEBTNRELEASE(0xc47a20, 0xd500f0, 0x7ffff7fa15d0) at gtk2callback.inc:2436
#68 ?? at :0
#69 g_closure_invoke at :0
#70 ?? at :0
#71 g_signal_emit_valist at :0
#72 g_signal_emit at :0
#73 ?? at :0
#74 gtk_propagate_event at :0
#75 gtk_main_do_event at :0
#76 ?? at :0
#77 g_main_context_dispatch at :0
#78 ?? at :0
#79 g_main_context_iteration at :0
#80 APPWAITMESSAGE(0x7ffff7f6ca50) at gtk2widgetset.inc:2404
#81 IDLE(0x7ffff7f6c430, true) at include/application.inc:405
#82 HANDLEMESSAGE(0x7ffff7f6c430) at include/application.inc:1261
#83 RUNLOOP(0x7ffff7f6c430) at include/application.inc:1395
#84 APPRUN(0x7ffff7f6ca50, {Proc = {procedure (POINTER)} 0x7fffffffe400, Self = 0x7ffff7f6c430}) at include/interfacebase.inc:54
#85 RUN(0x7ffff7f6c430) at include/application.inc:1383
#86 main at cleanup_scans.lpr:19
#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
#1 ?? at :0
#2 ?? at :0
#3 .Ld241 at :0
#4 ?? at :0
#5 ?? at :0
#6 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
#7 ?? at :0
#0 SYSTEM_$$_TRY_CONCAT_FREE_CHUNK$PMEMCHUNK_VAR$$PMEMCHUNK_VAR at :0
#1 ?? at :0
#2 ?? at :0
#3 U_$SYSTEM_$$_FREELISTS at :0
#4 SYSTEM_$$_SYSFREEMEM_VAR$PFREELISTS$PMEMCHUNK_VAR$$QWORD at :0
#5 .Ld241 at :0
#6 ?? at :0
#7 U_$SYSTEM_$$_FREELISTS at :0
#8 ?? at :0
#9 ?? at :0
#10 SYSTEM_$$_SYSFREEMEM$POINTER$$QWORD at :0
#11 ?? at :0
#12 ?? at :0
#13 ?? at :0
#14 RTTI_$DO_CLEANUP_SCANS_$$_LINE at :0
#15 ?? at :0
#16 SYSTEM_$$_FREEMEM$POINTER$$QWORD at :0
#17 ?? at :0
#18 fpc_dynarray_clear at :0
#19 fpc_finalize at :0
#20 .Ld241 at :0
#21 ?? at :0
#22 ?? at :0
#23 fpc_finalize_array at :0
#24 ?? at :0
#25 ?? at :0
#26 ?? at :0
#27 RTTI_$DO_CLEANUP_SCANS_$$_PAGE_BM at :0
#28 ?? at :0
#29 fpc_dynarray_clear at :0
#30 fpc_dynarray_assign at :0
#31 ?? at :0
#32 ?? at :0
#33 ?? at :0
#34 fpc_copy at :0
#35 ?? at :0
#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
#1 ?? at :0
#2 ?? at :0
#3 .Ld241 at :0
#4 ?? at :0
#5 ?? at :0
#6 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
#7 ?? at :0
They seem to point to the problem being related to something happening in a FreePascal library or its interface to an OS library.
I have this problem on two different (Intel based) computers, and, although the crash point varied between them to begin with, having removed calls to setlength wherever possible, they now seem to converge to the same page. One computer has 8GB of RAM; the other 16GB. In both cases when running the software, RAM use hovers around 1GB.
It is clear that the page image being processed is NOT the problem. If I start the software on a different page, it will process the offending page without issues.
I can provide source code and image files to help you demonstrate/find the problem.
Mantis conversion info:
- Mantis ID: 33424
- OS: Lubuntu
- OS Build: 16.0.4.3
- Build: x64, dfsg-2
- Platform: Intel
- Version: 3.0.0