View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0013105FPCRTLpublic2009-02-03 12:412009-03-21 13:14
ReporterErich Kuba 
Assigned ToJonas Maebe 
PrioritynormalSeveritycrashReproducibilityalways
StatusclosedResolutionno change required 
PlatformOSOS Version
Product Version2.2.2Product Build 
Target VersionFixed in Version 
Summary0013105: Max concurrent thread count limited to 383 on Linux
DescriptionHi there,

I've hit a limit of 383 concurrent threads in an application that we are developing. This is a consistent limit, and can be reproduced through a simple program.

The purpose of the sample program is to emmulate 500 concurrent threads. Ideally we would like to get to several thousand.

Please let me know if this is a bug or something that I can control.

The error message that I get when we exceed 382 threads is: EThread: Failed to create new thread

Here is a sample program based on another sample that I found on this site:

program test_thread;

{$mode objfpc}{$H+}
{$define usecthreads}

uses
{$IFDEF FPC}
  cthreads,
{$ENDIF}
  SysUtils,
  Classes;

type
  TTestThread = class(TThread)
  private
  protected
    procedure Execute; override;
  end;

const
  MAX_THREAD : integer = 1000;
var
  I : integer;


{ TTestThread }

procedure TTestThread.Execute;
begin
  Writeln('[' + IntToStr(Self.ThreadID) + '] Running...');
  Sleep(10000);
  Writeln('[' + IntToStr(Self.ThreadID) + '] Done...');
  Terminate;
end;

procedure CreateThread;
var
  TestThread : TTestThread;
begin
  TestThread := TTestThread.Create(true);
  TestThread.FreeOnTerminate := True;
  TestThread.Resume;
end;

begin
  Sleep(1000);

  try
    for I := 1 to MAX_THREAD do
      begin
        Writeln('Create ' + IntToStr(I) + ' ...');
        CreateThread;
        Sleep(20);
      end;
    Writeln('Finish');
    ReadLn;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Kind regards

Erich
TagsNo tags attached.
FPCOldBugId
Fixed in Revision
Attached Files

- Relationships
child of 0012942closedJonas Maebe Segmentation fault on stack overflow even with -Ct ! 

-  Notes
(0025052)
Jonas Maebe (manager)
2009-02-03 12:56
edited on: 2009-02-03 13:22

The limit is because the default stack size for a thread under Linux is the same as the stack size of the main program. This size varies between 8 MB and 10 MB, depending on your Linux distribution. 383 * 10MB = 3.83 GB, so you are simply running out of virtual memory space.

In FPC 2.2.x, it is however not possible to specify a custom stack size for threads on unix platforms. This has been fixed in FPC 2.3.1 (see 0012942 ).

The -Cs parameter only specifies the stack *checking* size for the main program, not the actual stack size (so you cannot use this either to reduce the stack size of the threads); the reason is that the compiler/linker simply cannot specify the stack size on unix platforms.

The only way to limit the main program's stack size (and thereby the stack size of the threads in FPC 2.2.x) is to use the "ulimit" command. For example:
$ ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16371
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) 1000000
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192 <----------------------------------------
cpu time (seconds, -t) unlimited
max user processes (-u) 16371
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ ulimit -s 1024
$ ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16371
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) 1000000
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 1024 <----------------------------------------
cpu time (seconds, -t) unlimited
max user processes (-u) 16371
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

The downside of this method is that you must ensure that your main program also does not use more stack size than this amount.

(0025085)
Erich Kuba (reporter)
2009-02-04 01:11

Hi Jonas,

Thank you for your response. I have confirmed that the application indeed uses over 3.5GB of ram just before crashing, so I agree with you on this.

I have to be able to support several (perhaps 8) thousand concurrent threads, so have upgarded to FPC 2.3.1 (I'd rather be on stable ... but this has to be fixed), but I am still having this problem. Please advise if there are any specific settings that I need to set.

Kind regards

Erich
(0025087)
Erich Kuba (reporter)
2009-02-04 04:10

Hi Jonas,

A further update. I've managed to get this sorted out.

I wouldn't mind asking you one question though ... how do I calcluate the required stack size for a thread. I'd obviously like to trim it down to the lowest safe value. Whats the process to find this out.

Cheers

Erich
PS. Thank you so much for your help.
(0025098)
Jonas Maebe (manager)
2009-02-04 10:42

> A further update. I've managed to get this sorted out.

For anyone else who may end up here when searching for a solution to this problem: you have to specify a stack size to tthread.create: http://www.freepascal.org/docs-html/rtl/classes/tthread.create.html [^]

> how do I calcluate the required stack size for a thread.
> I'd obviously like to trim it down to the lowest safe value.
> Whats the process to find this out.

There is no process that can guarantee you that the stack size you select is always correct. Calculating the maximum stack size is a very formal and complex procedure that is used in real time embedded programming to prove that programs will run using the allotted memory size, but it has several preconditions (no recursion; all control flow is predictable at compile-time, i.e. no procedure variables; etc).

To simply get an idea: test using different stack sizes until it crashes, and then add some safety margin to the value you found.

- Issue History
Date Modified Username Field Change
2009-02-03 12:41 Erich Kuba New Issue
2009-02-03 12:56 Jonas Maebe Status new => resolved
2009-02-03 12:56 Jonas Maebe Resolution open => no change required
2009-02-03 12:56 Jonas Maebe Assigned To => Jonas Maebe
2009-02-03 12:56 Jonas Maebe Note Added: 0025052
2009-02-03 12:56 Jonas Maebe Relationship added child of 0012942
2009-02-03 12:57 Jonas Maebe Note Edited: 0025052
2009-02-03 12:58 Jonas Maebe Note Edited: 0025052
2009-02-03 13:11 Vincent Snijders Note Added: 0025053
2009-02-03 13:12 Vincent Snijders Note Deleted: 0025053
2009-02-03 13:22 Jonas Maebe Note Edited: 0025052
2009-02-04 01:11 Erich Kuba Status resolved => feedback
2009-02-04 01:11 Erich Kuba Resolution no change required => reopened
2009-02-04 01:11 Erich Kuba Note Added: 0025085
2009-02-04 04:10 Erich Kuba Note Added: 0025087
2009-02-04 10:42 Jonas Maebe Status feedback => resolved
2009-02-04 10:42 Jonas Maebe Resolution reopened => no change required
2009-02-04 10:42 Jonas Maebe Note Added: 0025098
2009-03-21 13:14 Jonas Maebe Status resolved => closed



MantisBT 1.2.12[^]
Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker