View Issue Details

IDProjectCategoryView StatusLast Update
0033424FPCCompilerpublic2020-10-20 16:53
ReporterJohn Assigned ToJonas Maebe  
PrioritynormalSeverityminorReproducibilitysometimes
Status resolvedResolutionunable to reproduce 
PlatformIntelOSLubuntu 
Product Version3.0.0 
Summary0033424: possible problem with dynamic arrays are a lot of use
DescriptionI think that there is a problem with dynamic arrays, but I’m not 100% sure. I’m getting exceptions/crashes when I call setlength to set the initial size of a dynamic array, or perhaps to change its size. Here are some details.

I’m writing an image processing application. It takes scanned images and, in part, removes dirt/noise/crud from those images prior to OCR. The images are loaded from files as contiguous bitmaps (dynamic arrays of bytes) and converted to a series of lines. Hence, a page comprises a dynamic array of lines, each of which comprises a dynamic array of pixels (which, in my case, are bytes). I chose to organize the data in this way because the nature of the image processing that I was doing is somewhat non-linear and it is much easier to write code that addresses pixels by their X, Y indices, and it seems like it runs faster too.

The algorithms seem to work well the vast majority of the time, but occasionally, when I call setlength, the application generates an exception. This can be after many pages and therefore very many calls to setlength, e.g., 500,000 (a number suspiciously close to 2 ^ 19). However, in an early incarnation of the software, I successfully loaded all of the pages (400+) into RAM prior to doing any image processing (an approach that I abandoned after I determined that it was slower than doing the image processing on a page by page basis, reading and writing the files before and after respectively), so I don’t believe that the problem is directly related to number of calls to setlength per se.

For awhile, I wondered about whether there is enough “time” for the memory management algorithms to complete between calls to setlength, but after having introduced Application.ProcessMessages after calls to setlength without effect, I’m pretty sure that this is not the problem.
Steps To Reproducerun the software on many scanned pages
Additional InformationBelow, find some call stack traces that I’ve recorded.

#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
0000001 ?? at :0
0000002 ?? at :0
0000003 .Ld241 at :0
0000004 ?? at :0
0000005 ?? at :0
0000006 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
0000007 ?? at :0
0000008 ?? at :0
0000009 ?? at :0
0000010 SYSTEM_$$_GETMEM$POINTER$QWORD at :0
0000011 ?? at :0
0000012 fpc_dynarray_setlength at :0
0000013 WRITE_TIFF_FILE({FILE_NAME = 0x7ffff7fca9d8 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups/cleaned/page-238.tif', FILE_SIZE = 3018112, WAS_ERROR = false, LAST_ERROR = 0x0, IMAGE_COUNT = 1, IMAGES = 0x7ffff7fd0530}) at sf_tiff_io.pas:819
0000014 WRITE_PAGE({FILE_NAME = 0x7ffff7feedd8 'page-238.tif', SAMPLES_PER_PIXEL = 1, BITS_PER_SAMPLE = 1, PAGE = 0x7fffd1793960}) at do_cleanup_scans.pas:260
0000015 CLEAN_BOOK at do_cleanup_scans.pas:379
0000016 DIRECTORYEDIT1CHANGE(0x7ffff7faaad0, 0x7ffff7fa15d0) at do_cleanup_scans.pas:456
0000017 EDITCHANGE(0x7ffff7fa15d0) at editbtn.pas:1715
0000018 INTERNALONEDITCHANGE(0x7ffff7fa15d0, 0x7ffff7fa1d30) at editbtn.pas:1128
0000019 CHANGE(0x7ffff7fa1d30) at include/customedit.inc:582
0000020 CHANGE(0x7ffff7fa1d30) at maskedit.pp:1592
0000021 TEXTCHANGED(0x7ffff7fa1d30) at include/customedit.inc:574
0000022 TEXTCHANGED(0x7ffff7fa1d30) at maskedit.pp:1577
0000023 CMTEXTCHANGED(0x7ffff7fa1d30, {MSG = 45074, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 0, RESULT = 0}) at include/control.inc:1109
0000024 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0
0000025 .Ld240 at :0
0000026 .Ld239 at :0
0000027 ?? at :0
0000028 ?? at :0
0000029 ?? at :0
0000030 ?? at :0
0000031 WSREGISTERCLASS(0x7fffffffd1c0) at include/speedbutton.inc:852
0000032 WNDPROC(0x7ffff7fa1d30, {MSG = 45074, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 0, RESULT = 0}) at include/wincontrol.inc:5383
0000033 DELIVERMESSAGE(0x7ffff7fa1d30, 0) at lclmessageglue.pas:112
0000034 SETTEXT(0x7ffff7fa3370, 0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at gtk2wsstdctrls.pp:1113
0000035 WSSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/wincontrol.inc:5388
0000036 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/wincontrol.inc:8231
0000037 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/customedit.inc:523
0000038 REALSETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at maskedit.pp:1076
0000039 SETTEXT(0x7ffff7fa1d30, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at include/control.inc:4782
0000040 SETTEXT(0x7ffff7fa15d0, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at editbtn.pas:1677
0000041 SETDIRECTORY(0x7ffff7fa15d0, 0x7ffff7fd0e38 '/home/jhb/Images/Scans/Motorola Zener Diode Manual 1980/backups') at editbtn.pas:2424
0000042 RUNDIALOG(0x7ffff7fa15d0) at editbtn.pas:2491
0000043 BUTTONCLICK(0x7ffff7fa15d0) at editbtn.pas:2453
0000044 INTERNALONBUTTONCLICK(0x7ffff7fa15d0, 0x7ffff7fa0010) at editbtn.pas:1122
0000045 CLICK(0x7ffff7fa0010) at include/control.inc:2736
0000046 CLICK(0x7ffff7fa0010) at include/speedbutton.inc:111
0000047 WMLBUTTONUP(0x7ffff7fa0010, {MSG = 514, KEYS = 0, XPOS = 18, YPOS = 13, POS = {X = 18, Y = 13}, DUMMY = 851986, RESULT = 0}) at include/speedbutton.inc:907
0000048 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0
0000049 .Ld21 at :0
0000050 .Ld20 at :0
0000051 ?? at :0
0000052 ?? at :0
#53 ?? at :0
#54 ?? at :0
0000055 ?? at :0
0000056 ?? at :0
0000057 ?? at :0
0000058 ?? at :0
0000059 ?? at :0
0000060 WNDPROC(0x7ffff7fa0010, {MSG = 514, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 851986, RESULT = 0}) at include/control.inc:2124
0000061 PERFORM(0x7ffff7fa0010, 514, 0, 851986) at include/control.inc:1453
0000062 ISCONTROLMOUSEMSG(0x7ffff7fa15d0, 0) at include/wincontrol.inc:4713
0000063 WNDPROC(0x7ffff7fa15d0, {MSG = 514, UNUSEDMSG = 32767, WPARAM = 0, LPARAM = 852667, RESULT = 0}) at include/wincontrol.inc:5348
0000064 DELIVERMESSAGE(0x7ffff7fa15d0, 0) at lclmessageglue.pas:112
0000065 DELIVERMESSAGE(0x7ffff7fa15d0, 0) at gtk2proc.inc:3628
0000066 DELIVERMOUSEUPMESSAGE(0xc47a20, 0xd500f0, 0x7ffff7fa15d0) at gtk2callback.inc:2361
0000067 GTKMOUSEBTNRELEASE(0xc47a20, 0xd500f0, 0x7ffff7fa15d0) at gtk2callback.inc:2436
0000068 ?? at :0
0000069 g_closure_invoke at :0
0000070 ?? at :0
0000071 g_signal_emit_valist at :0
0000072 g_signal_emit at :0
0000073 ?? at :0
0000074 gtk_propagate_event at :0
0000075 gtk_main_do_event at :0
0000076 ?? at :0
0000077 g_main_context_dispatch at :0
0000078 ?? at :0
0000079 g_main_context_iteration at :0
0000080 APPWAITMESSAGE(0x7ffff7f6ca50) at gtk2widgetset.inc:2404
0000081 IDLE(0x7ffff7f6c430, true) at include/application.inc:405
0000082 HANDLEMESSAGE(0x7ffff7f6c430) at include/application.inc:1261
0000083 RUNLOOP(0x7ffff7f6c430) at include/application.inc:1395
0000084 APPRUN(0x7ffff7f6ca50, {Proc = {procedure (POINTER)} 0x7fffffffe400, Self = 0x7ffff7f6c430}) at include/interfacebase.inc:54
0000085 RUN(0x7ffff7f6c430) at include/application.inc:1383
0000086 main at cleanup_scans.lpr:19


#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
0000001 ?? at :0
0000002 ?? at :0
0000003 .Ld241 at :0
0000004 ?? at :0
0000005 ?? at :0
0000006 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
0000007 ?? at :0


#0 SYSTEM_$$_TRY_CONCAT_FREE_CHUNK$PMEMCHUNK_VAR$$PMEMCHUNK_VAR at :0
0000001 ?? at :0
0000002 ?? at :0
0000003 U_$SYSTEM_$$_FREELISTS at :0
0000004 SYSTEM_$$_SYSFREEMEM_VAR$PFREELISTS$PMEMCHUNK_VAR$$QWORD at :0
0000005 .Ld241 at :0
0000006 ?? at :0
0000007 U_$SYSTEM_$$_FREELISTS at :0
0000008 ?? at :0
0000009 ?? at :0
0000010 SYSTEM_$$_SYSFREEMEM$POINTER$$QWORD at :0
0000011 ?? at :0
0000012 ?? at :0
0000013 ?? at :0
0000014 RTTI_$DO_CLEANUP_SCANS_$$_LINE at :0
0000015 ?? at :0
0000016 SYSTEM_$$_FREEMEM$POINTER$$QWORD at :0
0000017 ?? at :0
0000018 fpc_dynarray_clear at :0
0000019 fpc_finalize at :0
0000020 .Ld241 at :0
0000021 ?? at :0
0000022 ?? at :0
0000023 fpc_finalize_array at :0
0000024 ?? at :0
0000025 ?? at :0
0000026 ?? at :0
0000027 RTTI_$DO_CLEANUP_SCANS_$$_PAGE_BM at :0
0000028 ?? at :0
0000029 fpc_dynarray_clear at :0
0000030 fpc_dynarray_assign at :0
0000031 ?? at :0
0000032 ?? at :0
0000033 ?? at :0
0000034 fpc_copy at :0
0000035 ?? at :0


#0 SYSTEM_$$_SYSGETMEM_VAR$QWORD$$POINTER at :0
0000001 ?? at :0
0000002 ?? at :0
0000003 .Ld241 at :0
0000004 ?? at :0
0000005 ?? at :0
0000006 SYSTEM_$$_SYSGETMEM$QWORD$$POINTER at :0
0000007 ?? at :0


They seem to point to the problem being related to something happening in a FreePascal library or its interface to an OS library.

I have this problem on two different (Intel based) computers, and, although the crash point varied between them to begin with, having removed calls to setlength wherever possible, they now seem to converge to the same page. One computer has 8GB of RAM; the other 16GB. In both cases when running the software, RAM use hovers around 1GB.

It is clear that the page image being processed is NOT the problem. If I start the software on a different page, it will process the offending page without issues.

I can provide source code and image files to help you demonstrate/find the problem.

TagsNo tags attached.
Fixed in Revision
FPCOldBugId
FPCTarget
Attached Files

Activities

Florian

2018-03-13 19:52

administrator   ~0107089

There is probably not much we can do about this: heavy setlength calls might cause memory fragmentation at a certain point. To avoid it:
- reduce number of setlength calls, especially reallocating is a problem
- use a different memory manager like cmem (this does not mean that cmem handles fragmentation better, it depends on the specific allocation pattern were it might be better or worse)

Marco van de Voort

2018-03-14 11:09

manager   ~0107103

Without a simplified program that demonstrates the problems, it will be hard to work on this.

John

2018-03-15 18:18

reporter   ~0107139

Last edited: 2018-03-15 18:29

View 2 revisions

For the sake of getting some more data, I compiled and ran the identical software with Lazarus on a 32 bit version of Linux. Although it ran slowly on my relatively old Atom processor, the software completed without errors. In other words, the problem seems to be related to the 64-bit version.

Below find a code snippet that is directly from the software that I'm writing. The failure tends to occur when running this function.

  line = array of byte; // corresponds to one scan line
  page_bm = array of line; // the scan lines that comprise the image of the page


function expanded_binary_bit_map(input: TIFF_image): page_bm;
// returns a binary bit map expanded to an 8 bit bitmap
var
  sample, X_mod: byte;
  X, Y: word;
  index: longword;

begin
  // allocate memory for the image data
  setlength(result, input.height);
  // convert from 1 bit per pixel bit map to 8 bits per pixel page
  index := 0;
  sample := input.image_data[index];
  for Y := 0 to input.height - 1 do
    begin
      setlength(result[Y], input.width);
      for X := 0 to input.width - 1 do
        begin
          X_mod := X and 7;
          if bytebool(sample and ($80 >> X_mod)) then
            result[Y][X] := white
          else
            result[Y][X] := black;
          if (X_mod = 7) then
            begin
              inc(index);
              sample := input.image_data[index];
            end;
        end;
      if (X_mod <> 7) then inc(index);
    end;
end;

I can send the complete project, including the image files, as a .zip file if that is of interest. I just didn't want to burden you with that as a start point.

Please let me know how I might help.

John

2018-03-17 18:49

reporter   ~0107212

More feedback.

I tried Florian’s suggestion about using cmem instead of the native memory manager. This induces failures on both 32 bit and 64 bit Lubuntu at exactly the same point in the code, and rather quickly – before having completed the entire process of reading, correcting, and writing a single page. Below are the stack traces.

//
// on 32 bit Lubuntu with cmem
//
#0 __kernel_vsyscall at :0
0000001 __GI_raise(6) at ../sysdeps/unix/sysv/linux/raise.c:54
0000002 __GI_abort at abort.c:89
0000003 __libc_message(2, 0xb74ace54 '*** Error in `%s'': %s: 0x%s ***'0000010) at ../sysdeps/posix/libc_fatal.c:175
0000004 malloc_printerr(<optimized out>, 0xb74acec4 'double free or corruption (!prev)', <optimized out>, 0xb74ff780 <main_arena>) at malloc.c:5006
0000005 _int_free(0xb74ff780 <main_arena>, <optimized out>, 0) at malloc.c:3867
0000006 CMEM_$$_CFREEMEM$POINTER$$LONGWORD(0xbfffe9c8, 0x2e0d88 <error: Cannot access memory at address 0x2e0d88>) at :2823
0000007 SYSTEM_$$_FREEMEM$POINTER$$LONGWORD at :0
0000008 fpc_dynarray_clear at :0
0000009 fpc_finalize at :0
0000010 ?? at :0
0000011 CLEAN_BOOK at do_cleanup_scans.pas:377
0000012 TFORM1__DIRECTORYEDIT1CHANGE(0x834fc64, 0x834c494) at do_cleanup_scans.pas:454
0000013 TCUSTOMEDITBUTTON__EDITCHANGE(0x834fc64) at editbtn.pas:1715
0000014 TCUSTOMEDITBUTTON__INTERNALONEDITCHANGE(0x8350a04, 0x834fc64) at editbtn.pas:1128
0000015 TCUSTOMEDIT__CHANGE(0x8350a04) at ./include/customedit.inc:582
0000016 TCUSTOMMASKEDIT__CHANGE(0x8350a04) at maskedit.pp:1592
0000017 TCUSTOMEDIT__TEXTCHANGED(0x8350a04) at ./include/customedit.inc:574
0000018 TCUSTOMMASKEDIT__TEXTCHANGED(0x8350a04) at maskedit.pp:1577
0000019 TCONTROL__CMTEXTCHANGED({MSG = 45074, WPARAM = 0, LPARAM = 0, RESULT = 0, WPARAMLO = 0, WPARAMHI = 0, LPARAMLO = 0, LPARAMHI = 0, RESULTLO = 0, RESULTHI = 0}, 0x8350a04) at ./include/control.inc:1109
0000020 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0

//
// on 64 bit Lubuntu with cmem
//
#0 __GI_raise(6, 6) at ../sysdeps/unix/sysv/linux/raise.c:54
0000001 __GI_abort at abort.c:89
0000002 __libc_message(2, 2, 0x7ffff5832ed8 '*** Error in `%s'': %s: 0x%s ***'0000010, 0x7ffff5832ed8 '*** Error in `%s'': %s: 0x%s ***'0000010) at ../sysdeps/posix/libc_fatal.c:175
0000003 malloc_printerr(<optimized out>, 0x7ffff5833008 'double free or corruption (!prev)', 3) at malloc.c:5006
0000004 _int_free(<optimized out>, <optimized out>, 0) at malloc.c:3867
0000005 __GI___libc_free(<optimized out>) at malloc.c:2968
0000006 CMEM_$$_CFREEMEM$POINTER$$QWORD at :0
0000007 ?? at :0
0000008 SYSTEM_$$_FREEMEM$POINTER$$QWORD at :0
0000009 ?? at :0
0000010 fpc_dynarray_clear at :0
0000011 fpc_finalize at :0
0000012 ?? at :0
0000013 ?? at :0
0000014 ?? at :0
0000015 WRITE_PAGE({FILE_NAME = 0xf7fde0 'page-001.tif', SAMPLES_PER_PIXEL = 1, BITS_PER_SAMPLE = 1, PAGE = 0x3f0df98}) at do_cleanup_scans.pas:227
0000016 CLEAN_BOOK at do_cleanup_scans.pas:377
0000017 DIRECTORYEDIT1CHANGE(0xccf008, 0xcd4698) at do_cleanup_scans.pas:454
0000018 EDITCHANGE(0xcd4698) at editbtn.pas:1715
0000019 INTERNALONEDITCHANGE(0xcd4698, 0xd0cb68) at editbtn.pas:1128
0000020 CHANGE(0xd0cb68) at include/customedit.inc:582
0000021 CHANGE(0xd0cb68) at maskedit.pp:1592
0000022 TEXTCHANGED(0xd0cb68) at include/customedit.inc:574
0000023 TEXTCHANGED(0xd0cb68) at maskedit.pp:1577
0000024 CMTEXTCHANGED(0xd0cb68, {MSG = 45074, UNUSEDMSG = 0, WPARAM = 0, LPARAM = 0, RESULT = 0}) at include/control.inc:1109
0000025 SYSTEM$_$TOBJECT_$__$$_DISPATCH$formal at :0

Here is the code that was being executed.

function binary_bit_map(image: page_bm; bright: byte): byte_buffer;
// takes a binary black & white page and returns a binary bitmap
// if a pixel = bright, then the output is 1
var
  height, width, bytes_per_line, bm_size, X, Y, index: longword;

begin
  height := length(image);
  width := length(image[0]);
  bytes_per_line := width div 8;
  if bytes_per_line * 8 <> width then inc(bytes_per_line);
  bm_size := height * bytes_per_line;
  setlength(result, bm_size);
  index := 0;
  result[index] := 0;
  // write out the image line by line
  for Y := 0 to height - 1 do
    for X := 0 to width - 1 do
      begin
        if image[Y][X] = bright then
          result[index] := result[index] or ($80 >> (X and 7));
        if ((X and 7) = 7) or (X = width - 1) then
          begin
            inc(index);
            result[index] := 0;
          end;
      end;
end;


function write_page(input: page_type): TIFF_write_result;
// writes the input page to a TIFF output file
var
  file_record: TIFF_file_record;
  bitmap: byte_buffer;
  Y, line_length: longword;
  bits_per_sample: word_buffer;

begin
  Form1.StatusBar1.SimpleText := input.file_name + ': writing';
  Application.ProcessMessages; // give other system processes a chance to work
  // put everything into a TIFF_file_record format for the TIFF writer
  if input.bits_per_sample = 1 then
    bitmap := binary_bit_map(input.page, white)
  else
    // write the lines as a single bitmap
    begin
      line_length := length(input.page[0]);
      setlength(bitmap, length(input.page) * line_length);
      for Y := 0 to length(input.page) - 1 do
        move(input.page[Y][0],
             bitmap[Y * line_length],
             line_length);
    end;
  setlength(bits_per_sample, input.samples_per_pixel);
  for Y := 0 to input.samples_per_pixel - 1 do
    bits_per_sample[Y] := input.bits_per_sample;
  file_record.file_name := output_directory + short_file_name(input.file_name) + '.tif';
  file_record.was_error := false;
  file_record.last_error := '';
  file_record.image_count := 1;
  file_record.file_size := length(bitmap);
  setlength(file_record.images, 1);
  file_record.images[0].was_error := false;
  file_record.images[0].last_error := '';
  file_record.images[0].description := '';
  file_record.images[0].bits_per_sample := bits_per_sample;
  file_record.images[0].samples_per_pixel := input.samples_per_pixel;
  file_record.images[0].height := length(input.page);
  file_record.images[0].width := length(input.page[0]);
  file_record.images[0].image_data := bitmap;
  result := write_tiff_file(file_record);
end;

function cleaned_page(file_name: ansistring): page_type;
// cleans the page associated with file_name
var
  image_record: TIFF_file_record;
  page_num: word;

begin
  image_record := read_tiff_file(file_name);
  if (image_record.was_error) then
    begin
      Form1.StatusBar1.SimpleText := 'error reading ' + file_name + ': ' + image_record.last_error;
      setlength(result.page, 0);
    end
  else
    begin
      result := read_page(image_record);
      Application.ProcessMessages; // give other system processes a chance to work
      page_num := file_number(result.file_name) - 1;
      if result.bits_per_sample = 1 then
        begin
          mark_halftone_areas(result);
          Form1.StatusBar1.SimpleText := result.file_name + ': removing single speckle strings';
          Application.ProcessMessages; // give other system processes a chance to work
          result := remove_single_pixel_strings(result);
          Form1.StatusBar1.SimpleText := result.file_name + ': removing 2 x 2 blocks';
          Application.ProcessMessages; // give other system processes a chance to work
          result := remove_single_pixel_strings(remove_two_by_two_blocks(result));
          Form1.StatusBar1.SimpleText := result.file_name + ': aligning the page';
          Application.ProcessMessages; // give other system processes a chance to work
          result := align_page(result);
          crop_width := min(crop_width, book_data[page_num].right_margin - book_data[page_num].left_margin);
          if (book_data[page_num].bottom_margin <> length(result.page)) and (not outlier(book_data[page_num].bottom_margin, crop_height_stats)) then
            crop_height := min(crop_height, book_data[page_num].bottom_margin);
        end;;
      Application.ProcessMessages; // give other system processes a chance to work
    end;
end;


procedure clean_book;
// cleans all of the pages within the book and saves them to the cleaned subdirectory
var
  file_count: longword;

begin
  crop_width := $1000000;
  crop_height := $1000000;
  for file_count := 0 to length(files) - 1 do
    write_page(cleaned_page(files[file_count].directory + files[file_count].info.Name));
end;


It seems that, on both 32 bit and 64 bit Lubuntu, the failure occurs immediately after the function write_page, that is, during whatever cleanup operation is undertaken prior to returning to the calling procedure, clean_book. I’m not sure why this should be and why cmem is so much worse than the native memory manager, but, near as I can tell, it doesn’t seem to have anything to do with my code, but rather the implementation of dynamic arrays. I can confirm that the file writing process was complete and that the file was properly closed.

Dynamic arrays are an extremely valuable capability in FreePascal since they free the programmer from having to worry about memory management details. I use them extensively in other software that I’ve written, especially in a geometric algebra calculator that I’d like to release to the public domain. Since dynamic arrays are an intrinsic part of the FreePascal language definition, I would hope that one can count on being able to use them without concern that they will at some time fail. Yes, I know that I’m pushing hard on them (and the underlying memory manager) in this software, but as a programmer, it should be reasonable to believe that these functions will not fail, although, due to memory fragmentation, they might take longer over time.

Again, if I can be of help in tracking down and correcting this problem let me know. I’m happy to help make this feature of FreePascal bug free.

Martok

2018-03-17 20:07

reporter   ~0107216

Are you sure this is due to memory fragmentation? 500000 allocations is not a lot.

I see crashes near FreeMem usually after some form of heap corruption, most likely caused by out-of-bounds writes. The memory manager tends to be the first thing that breaks.

Florian

2018-03-17 22:41

administrator   ~0107220

Crash with cmem as well and the point Martok mentioned points for me also into the direction of memory corruptions. Did you try to turn on all checks fpc offers (e.g. range, overflow, heap)?

Thaddy de Koning

2018-03-18 07:32

reporter   ~0107228

Last edited: 2018-03-18 07:40

View 4 revisions

[code=pascal]
for Y := 0 to input.height - 1 do
 begin
   setlength(result[Y], input.width);
[/code]
Is highly inefficient code and a possible cause of fragmentation, because it keeps creating copies.
Can you bring the setlength outside of the loop? That results in width-1 less (re-)allocations. You can set it in one go, since the result is rectanglar.
See http://wiki.freepascal.org/Example:_Multidimensional_dynamic_array
Setlength(Result,input.Height,input.width);

Marco van de Voort

2018-03-18 13:46

manager   ~0107237

The cmem traceback has double free warnings?

Anyway first -CRriot, and otherwise valgrind.

Thaddy de Koning

2018-03-23 18:32

reporter   ~0107375

I re-tested and have no problems with well (somewhat) written code. As per my remark.

Jonas Maebe

2018-12-24 16:20

manager   ~0112862

Try compiling with -gv and running under valgrind to ensure you don't have buffer overflows. Otherwise please provide a compilable example that demonstrates the problem.

Peter Blackman

2020-10-20 16:53

reporter   ~0126426

Wondering if this problem is related to 35187?
https://bugs.freepascal.org/view.php?id=35187

PB

Issue History

Date Modified Username Field Change
2018-03-13 18:14 John New Issue
2018-03-13 19:52 Florian Note Added: 0107089
2018-03-14 11:09 Marco van de Voort Note Added: 0107103
2018-03-15 18:18 John Note Added: 0107139
2018-03-15 18:29 John Note Edited: 0107139 View Revisions
2018-03-17 18:49 John Note Added: 0107212
2018-03-17 20:07 Martok Note Added: 0107216
2018-03-17 22:41 Florian Note Added: 0107220
2018-03-18 07:32 Thaddy de Koning Note Added: 0107228
2018-03-18 07:33 Thaddy de Koning Note Edited: 0107228 View Revisions
2018-03-18 07:39 Thaddy de Koning Note Edited: 0107228 View Revisions
2018-03-18 07:41 Thaddy de Koning Note Edited: 0107228 View Revisions
2018-03-18 13:46 Marco van de Voort Note Added: 0107237
2018-03-23 18:32 Thaddy de Koning Note Added: 0107375
2018-12-24 16:20 Jonas Maebe Note Added: 0112862
2018-12-24 16:20 Jonas Maebe Status new => resolved
2018-12-24 16:20 Jonas Maebe Resolution open => unable to reproduce
2018-12-24 16:20 Jonas Maebe Assigned To => Jonas Maebe
2020-10-20 16:53 Peter Blackman Note Added: 0126426