0013105: Max concurrent thread count limited to 383 on Linux

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0013105

FPC

RTL

public

2009-02-03 12:41

2009-03-21 13:14

Reporter

Erich Kuba

Assigned To

Jonas Maebe

Priority

normal

Severity

crash

Reproducibility

always

Status

closed

Resolution

no change required

Platform

OS Version

Product Version

2.2.2

Product Build

Target Version

Fixed in Version

Summary

0013105: Max concurrent thread count limited to 383 on Linux

Description

Hi there,

I've hit a limit of 383 concurrent threads in an application that we are developing. This is a consistent limit, and can be reproduced through a simple program.

The purpose of the sample program is to emmulate 500 concurrent threads. Ideally we would like to get to several thousand.

Please let me know if this is a bug or something that I can control.

The error message that I get when we exceed 382 threads is: EThread: Failed to create new thread

Here is a sample program based on another sample that I found on this site:

program test_thread;

{$mode objfpc}{$H+}
{$define usecthreads}

uses
{$IFDEF FPC}
  cthreads,
{$ENDIF}
  SysUtils,
  Classes;

type
  TTestThread = class(TThread)
  private
  protected
    procedure Execute; override;
  end;

const
  MAX_THREAD : integer = 1000;
var
  I : integer;

{ TTestThread }

procedure TTestThread.Execute;
begin
  Writeln('[' + IntToStr(Self.ThreadID) + '] Running...');
  Sleep(10000);
  Writeln('[' + IntToStr(Self.ThreadID) + '] Done...');
  Terminate;
end;

procedure CreateThread;
var
  TestThread : TTestThread;
begin
  TestThread := TTestThread.Create(true);
  TestThread.FreeOnTerminate := True;
  TestThread.Resume;
end;

begin
  Sleep(1000);

  try
    for I := 1 to MAX_THREAD do
      begin
        Writeln('Create ' + IntToStr(I) + ' ...');
        CreateThread;
        Sleep(20);
      end;
    Writeln('Finish');
    ReadLn;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Kind regards

Erich

Tags

No tags attached.

FPCOldBugId

Fixed in Revision

Attached Files

Relationships

child of

0012942

closed

Jonas Maebe

Segmentation fault on stack overflow even with -Ct !

Notes
(0025052) Jonas Maebe (manager) 2009-02-03 12:56 edited on: 2009-02-03 13:22	The limit is because the default stack size for a thread under Linux is the same as the stack size of the main program. This size varies between 8 MB and 10 MB, depending on your Linux distribution. 383 * 10MB = 3.83 GB, so you are simply running out of virtual memory space. In FPC 2.2.x, it is however not possible to specify a custom stack size for threads on unix platforms. This has been fixed in FPC 2.3.1 (see 0012942 ). The -Cs parameter only specifies the stack checking size for the main program, not the actual stack size (so you cannot use this either to reduce the stack size of the threads); the reason is that the compiler/linker simply cannot specify the stack size on unix platforms. The only way to limit the main program's stack size (and thereby the stack size of the threads in FPC 2.2.x) is to use the "ulimit" command. For example: $ ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 16371 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) 1000000 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 <---------------------------------------- cpu time (seconds, -t) unlimited max user processes (-u) 16371 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited $ ulimit -s 1024 $ ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 16371 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) 1000000 open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 1024 <---------------------------------------- cpu time (seconds, -t) unlimited max user processes (-u) 16371 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited The downside of this method is that you must ensure that your main program also does not use more stack size than this amount.

(0025085) Erich Kuba (reporter) 2009-02-04 01:11	Hi Jonas, Thank you for your response. I have confirmed that the application indeed uses over 3.5GB of ram just before crashing, so I agree with you on this. I have to be able to support several (perhaps 8) thousand concurrent threads, so have upgarded to FPC 2.3.1 (I'd rather be on stable ... but this has to be fixed), but I am still having this problem. Please advise if there are any specific settings that I need to set. Kind regards Erich

(0025087) Erich Kuba (reporter) 2009-02-04 04:10	Hi Jonas, A further update. I've managed to get this sorted out. I wouldn't mind asking you one question though ... how do I calcluate the required stack size for a thread. I'd obviously like to trim it down to the lowest safe value. Whats the process to find this out. Cheers Erich PS. Thank you so much for your help.

(0025098) Jonas Maebe (manager) 2009-02-04 10:42	> A further update. I've managed to get this sorted out. For anyone else who may end up here when searching for a solution to this problem: you have to specify a stack size to tthread.create: http://www.freepascal.org/docs-html/rtl/classes/tthread.create.html [^] > how do I calcluate the required stack size for a thread. > I'd obviously like to trim it down to the lowest safe value. > Whats the process to find this out. There is no process that can guarantee you that the stack size you select is always correct. Calculating the maximum stack size is a very formal and complex procedure that is used in real time embedded programming to prove that programs will run using the allotted memory size, but it has several preconditions (no recursion; all control flow is predictable at compile-time, i.e. no procedure variables; etc). To simply get an idea: test using different stack sizes until it crashes, and then add some safety margin to the value you found.

Issue History
Date Modified	Username	Field	Change
2009-02-03 12:41	Erich Kuba	New Issue
2009-02-03 12:56	Jonas Maebe	Status	new => resolved
2009-02-03 12:56	Jonas Maebe	Resolution	open => no change required
2009-02-03 12:56	Jonas Maebe	Assigned To	=> Jonas Maebe
2009-02-03 12:56	Jonas Maebe	Note Added: 0025052
2009-02-03 12:56	Jonas Maebe	Relationship added	child of 0012942
2009-02-03 12:57	Jonas Maebe	Note Edited: 0025052
2009-02-03 12:58	Jonas Maebe	Note Edited: 0025052
2009-02-03 13:11	Vincent Snijders	Note Added: 0025053
2009-02-03 13:12	Vincent Snijders	Note Deleted: 0025053
2009-02-03 13:22	Jonas Maebe	Note Edited: 0025052
2009-02-04 01:11	Erich Kuba	Status	resolved => feedback
2009-02-04 01:11	Erich Kuba	Resolution	no change required => reopened
2009-02-04 01:11	Erich Kuba	Note Added: 0025085
2009-02-04 04:10	Erich Kuba	Note Added: 0025087
2009-02-04 10:42	Jonas Maebe	Status	feedback => resolved
2009-02-04 10:42	Jonas Maebe	Resolution	reopened => no change required
2009-02-04 10:42	Jonas Maebe	Note Added: 0025098
2009-03-21 13:14	Jonas Maebe	Status	resolved => closed

freepascal bugtracker

Open source compiler for Pascal and Object Pascal