View Issue Details

IDProjectCategoryView StatusLast Update
0028991LazarusIDEpublic2019-08-21 00:19
ReporterVasily I.VolchenkoAssigned ToMattias Gaertner 
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
Platformi386 (may be amd64)OSWindowsOS Version10,may be other
Product Version1.5 (SVN)Product Build50279 (and some less) 
Target VersionFixed in Version 
Summary0028991: IDE cannot compile projects with FPC if path has national symbols
DescriptionIf path to the project has national symbols, the strange error - panic, internal error "Failed to execute: 267" happens.
Tested with trunk FPC, but it is not an FPC issue: FPC can compile files being called from command prompt from that directory or from outer directory with the partial path.
This is true for national symbols having 2 bytes in UTF-8, like Russian. Seems that some UTF-8 <-> ANSI/OEM processing is broken in fpc process connecting code.
IDE itself (codetools etc) works fine with such a project.
Lazbuild failed to build the projects too.
Steps To Reproduce1. Create new project - may be application.
2. Save the project on some path with national symbol - for example,
C:\Temp\ы\
3. Try to build (compile, run) it on MS Windows system. The result will be the same.
Additional InformationMay be, it should be moved to "compiler" category, but this is not FPC issue, so let it be in IDE by now, but LazBuild is in touch too.
TagsIDE, lazbuild, localization
Fixed in Revision50586,50596-50605
LazTarget-
WidgetsetWin32/Win64
Attached Files
  • lazutils.patch (2,541 bytes)
    Index: components/lazutils/fpcadds.pas
    ===================================================================
    --- components/lazutils/fpcadds.pas	(revision 50331)
    +++ components/lazutils/fpcadds.pas	(working copy)
    @@ -32,6 +32,11 @@
     function AlignToPtr(const p: Pointer): Pointer;
     function AlignToInt(const p: Pointer): Pointer;
     
    +{$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
    +var
    +  RealSystemCodePage:TSystemCodePage;
    +{$ENDIF}
    +
     implementation
     
     function StrToWord(const s: string): word;
    @@ -66,6 +71,7 @@
     
     {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
     initialization
    +  RealSystemCodePage:=DefaultSystemCodePage;
       SetMultiByteConversionCodePage(CP_UTF8);
       // SetMultiByteFileSystemCodePage(CP_UTF8); not needed, this is the default under Windows
       SetMultiByteRTLFileSystemCodePage(CP_UTF8);
    Index: components/lazutils/lazutf8.pas
    ===================================================================
    --- components/lazutils/lazutf8.pas	(revision 50331)
    +++ components/lazutils/lazutf8.pas	(working copy)
    @@ -53,6 +53,7 @@
     function ConsoleToUTF8(const s: string): string;
     // converts UTF8 string to console encoding (used by Write, WriteLn)
     function UTF8ToConsole(const s: string): string;
    +
     {$IFDEF MSWindows}
     // for all Windows supporting 8bit codepages (e.g. not WinCE)
     // converts string in Windows code page to UTF8 (used with some Windows specific functions)
    @@ -253,6 +254,7 @@
     begin
       {$IFDEF ReallyUseUTF8RTL}
       Result:=s;
    +  SetCodePage(RawByteString(Result), RealSystemCodePage, True);
       {$ELSE}
       if NeedRTLAnsi and (not IsASCII(s)) then
         Result:=UTF8ToAnsi(s)
    Index: components/lazutils/utf8process.pp
    ===================================================================
    --- components/lazutils/utf8process.pp	(revision 50331)
    +++ components/lazutils/utf8process.pp	(working copy)
    @@ -22,7 +22,8 @@
     interface
     
     uses
    -  Classes, SysUtils, Process, FileUtil, LazFileUtils, LazUTF8, LazUtilsStrConsts;
    +  Classes, SysUtils, Process, FileUtil, LazFileUtils, LazUTF8, LazUtilsStrConsts,
    +  FPCAdds;
     
     type
       { TProcessUTF8 }
    @@ -244,7 +245,13 @@
     procedure TProcessUTF8.Execute;
     begin
       UpdateEnvironment;
    +  {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
    +  DefaultSystemCodePage := RealSystemCodePage;
    +  {$ENDIF}
       inherited Execute;
    +  {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
    +  DefaultSystemCodePage := CP_UTF8;
    +  {$ENDIF}
     end;
     
     function FindFilenameOfCmd(ProgramFilename: string): string;
    
    lazutils.patch (2,541 bytes)

Relationships

related to 0028962 new FPC Compilation failed if Unicode char is in project name or path 
has duplicate 0019483 closedJuha Manninen Lazarus Lazarus cannot build to a directory which has unicode characters in its name 
related to 0035991 new Lazarus Crash (sigsegv) or Exception caused by TProcessUtf8 
child of 0028857 closedBart Broersma Lazarus Implicit Codepage Conversion meta issue 

Activities

Vasily I.Volchenko

2015-11-11 14:18

reporter   ~0087229

Last edited: 2015-11-11 14:19

View 2 revisions

I don't agree this is minor severity. It is not fatal, but at leas major, or even critical (at least for new release) - this stops lazarus working with projects stored everywhere. It might be not nesessury for those who use low 127 letters, but for others it is very important. Lazarus still can be used, but moving all the projects from their right place (in directories which names are connected with one exact task, including documents, data, programs) to that place where they can be compiled.
But seems like Lazarus (and FPC) team has a tradition to maximally spoil the usage of these tools for non-Latin language-speaking users (UTF8 is not fogotten, it is still problemable and even doubtful way for us, and we can see it results in a lot of problems with "minor" severity).

Juha Manninen

2015-11-11 14:53

developer   ~0087230

It should work now when using FPC trunk or 3.0.
Strange.

Vasily I.Volchenko

2015-11-11 14:58

reporter   ~0087231

Last edited: 2015-11-11 14:58

View 2 revisions

I tested it after updating trunk both of FPC and Lazarus today in the morning. Lazarus 1.4.4 on FPC 2.6.4 has no such a bug.

Vasily I.Volchenko

2015-11-11 15:01

reporter   ~0087232

Last edited: 2015-11-11 15:02

View 2 revisions

These bugs (current and 0019483) are not identical, the bug 0019483 cannot be reproduced presently with trunk FPC+Lazarus. This bug is the problem of calling directly FPC (all other things seems to be done, but FPC is called that way when it see no project), that was on creating icon.

Martok

2015-11-11 15:14

reporter   ~0087233

Does the commandline in the error message contain replaced '?' characters, like in the other bug?

Also, what is your system codepage set to?

Vasily I.Volchenko

2015-11-11 15:24

reporter   ~0087234

Last edited: 2015-11-11 15:31

View 2 revisions

Very strange. It might be an issue of FPC too, or something else. Command line is, for example
e:\pp\bin\i386-win32\fpc.exe -MObjFPC -Scghi -O1 -g -gl -WG -l -vewnhibq -FiE:\1\ы\lib\i386-win32 -FuE:\fpc\lazarus\lcl\units\i386-win32\win32 -FuE:\fpc\lazarus\lcl\units\i386-win32 -FuE:\fpc\lazarus\components\lazutils\lib\i386-win32 -FuE:\fpc\lazarus\packager\units\i386-win32 -FuE:\1\ы\-FUE:\1\ы\lib\i386-win32\ -dLCL -dLCLwin32 project1.lpr
When it is called from somewhere except E:\1\ы\, we have this bug. When it is called from E:\1\ы\, compilation succeed. If placing the full path except of project1.lpr, compiling succeed too.

Martok

2015-11-11 16:21

reporter   ~0087236

Looks like a TProcess issue.

TProcess executes using CreateProcessA, with parameters converted via UTF8ToSys by TProcessUTF8 (or not, now that we have UTF8 RTL that is a NOP). Variables in TProcess are declared as String/PChar, so that might pass raw UTF8 data to the winapi? Certainly looks like it in my test (cyrillic characters in path, win10, german, call stack intercepted with API Monitor).

Vasily I.Volchenko

2015-11-11 16:46

reporter   ~0087237

May be not: command line (in lazarus compiled with -WG) is correct, but project1.lpr is alone. Alone project in -Fu path is not recognized by trunk FPC if it is not in the current directory. Bugfix may try to call fpc with the full project name (with path). May be not.

Martok

2015-11-11 17:06

reporter   ~0087238

Command line (and Working Directory, which is where 267 Invalid Directory comes from) are wrong for me, captured at API side:
LPTSTR lpCommandLine = "C:\Prg\Dev\Lazarus\fpc\3.1.1\bin\i386-win32\fpc.exe -MObjFPC -Scghi -O1 -g -gl -WG -l -vewnhibq -FiC:\Test\?????1\lib\i386-win32 -FuC:\Prg\Dev\Lazarus\lcl\units\i386-win32\win32 -FuC:\Prg\Dev\Lazarus\lcl\units\i386-win32 -FuC:\Prg\Dev\Lazarus\components\lazutils\lib\i386-win32 -FuC:\Prg\Dev\Lazarus\packager\units\i386-win32 -FuC:\Test\?????1\ -FUC:\Test\?????1\lib\i386-win32\ -dLCL -dLCLwin32 project.lpr"
LPCTSTR lpCurrentDirectory = "C:\Test\?????1\"

So these are replaced characters. This is not what happened in my test before with using TProcess directly, where I had seen raw UTF8. Looks like another issue before that then.
Pre-Build Commands are affected as well, so i'd guess in the exttools.pas? The CreateProcessA/W issue comes into play later.

Bart Broersma

2015-11-11 18:01

developer   ~0087239

I don't have a clue how to solve this. Unassigning.

Michl

2015-11-11 18:11

developer   ~0087241

It is a FPC compiler issue. See 0028962

Juha Manninen

2015-11-11 19:14

developer   ~0087242

Ok, 0028962 is about FPC trunk. Has anybody tested with FPC 3.0? It is the most relevant version now. Next Lazarus will be released with it.

Bart Broersma

2015-11-11 22:16

developer   ~0087249

Last edited: 2015-11-11 22:34

View 4 revisions

Same problem with 3..rc2 compiler.

Panic: internal error: unable to execute: Failed to execute : 267

The file in question is:
"C:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\empty.pp"
Locale: Dutch (so äëï is in my current codepage)
Lazarus compiled with ÜTF8 in RTL feature enabled.

Compile Project, Mode: debug, Target: empty.exe
Compile Reason: State file "C:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\empty.compiled" of Project is missing.

(Note: the file is actually there after the attempt to compile)

Executable:
C:\devel\fpc\3.0.0\bin\i386-Win32\fpc.exe
CurrentDirectory:
C:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\
Parameters:
-B
-MObjFPC
-Scgi
-Cirot
-g
-gl
-gh
-l
-vewnhibq
-FuC:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\
empty.pp
Command Line:
C:\devel\fpc\3.0.0\bin\i386-Win32\fpc.exe -B -MObjFPC -Scgi -Cirot -g -gl -gh -l -vewnhibq -FuC:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\ empty.pp
ProcessID:
0
ExitStatus:
0
Error: unable to execute: Failed to execute : 267

If I try again, it says:
Compile Reason: Last compile was incomplete.
  State file=C:\Users\Bart\LazarusProjecten\ConsoleProjecten\bugs\äëï\empty.compiled

Juha Manninen

2015-11-11 23:52

developer   ~0087252

I remember it was already working at some point. Maybe I misunderstood.
This will be a beauty-spot in the otherwise improved Unicode support of next Lazarus release.

Michl

2015-11-15 10:54

developer   ~0087288

Last edited: 2015-11-15 11:05

View 4 revisions

I found some time and I check that issue.

I was wrong with that statement, that this is a compiler issue. It is a problem of Lazarus, cause the DefaultSystemCodePage is changed to UTF8 (changed in revision 50129). Martok was near the answer. Till now, the FPC compiler only accept ACP strings and Lazarus have to follow it.

The Lazarus compiler calls a TProcessUTF8, where are some string conversions:
 
SomeString:=UTF8ToSys(String)

These strings aren't converted any more, cause with the changing (DefaultSystemCodePage:=CP_UTF8), following happens:

SomeString:=UTF8ToUTF8(String)

In the calling of TProcess all strings are now UTF8 encoded. There are also some string conversions:

SomeString:=PChar(SomeString)

PChar convert a string to a null terminated string defined through DefaultSystemCodePage, which means just about:

SomeUTF8EncodedString:=PChar(SomeUTF8EncodedString)

The FPC compiler don't understand UTF8 encoded parameters, so the current problem accrues.


I changed the function UTF8ToSys (unit LazUTF8) to:

function UTF8ToSys(const s: string): string;
begin
  {$IFDEF ReallyUseUTF8RTL}
  Result:=s;
  SetCodePage(RawByteString(Result), RealSystemCodePage, True);
  {$ELSE}
...
  {$ENDIF}
end;

and set the ACP codepage for the TProcessUTF8 execution:

procedure TProcessUTF8.Execute;
begin
  UpdateEnvironment;
  DefaultSystemCodePage := RealSystemCodePage;
  inherited Execute;
  DefaultSystemCodePage := CP_UTF8;
end;

Now a project with ACP chars in the path are compiled again, but the debugger crash. Till now, I don't know why. Maybe I find some time later to check this too.

I added a patch - not to apply it - only as a hint, that you can see, what goes wrong. Maybe you have a better solution for that issue.

It is related to 0028857

Michl

2015-11-15 10:55

developer  

lazutils.patch (2,541 bytes)
Index: components/lazutils/fpcadds.pas
===================================================================
--- components/lazutils/fpcadds.pas	(revision 50331)
+++ components/lazutils/fpcadds.pas	(working copy)
@@ -32,6 +32,11 @@
 function AlignToPtr(const p: Pointer): Pointer;
 function AlignToInt(const p: Pointer): Pointer;
 
+{$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
+var
+  RealSystemCodePage:TSystemCodePage;
+{$ENDIF}
+
 implementation
 
 function StrToWord(const s: string): word;
@@ -66,6 +71,7 @@
 
 {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
 initialization
+  RealSystemCodePage:=DefaultSystemCodePage;
   SetMultiByteConversionCodePage(CP_UTF8);
   // SetMultiByteFileSystemCodePage(CP_UTF8); not needed, this is the default under Windows
   SetMultiByteRTLFileSystemCodePage(CP_UTF8);
Index: components/lazutils/lazutf8.pas
===================================================================
--- components/lazutils/lazutf8.pas	(revision 50331)
+++ components/lazutils/lazutf8.pas	(working copy)
@@ -53,6 +53,7 @@
 function ConsoleToUTF8(const s: string): string;
 // converts UTF8 string to console encoding (used by Write, WriteLn)
 function UTF8ToConsole(const s: string): string;
+
 {$IFDEF MSWindows}
 // for all Windows supporting 8bit codepages (e.g. not WinCE)
 // converts string in Windows code page to UTF8 (used with some Windows specific functions)
@@ -253,6 +254,7 @@
 begin
   {$IFDEF ReallyUseUTF8RTL}
   Result:=s;
+  SetCodePage(RawByteString(Result), RealSystemCodePage, True);
   {$ELSE}
   if NeedRTLAnsi and (not IsASCII(s)) then
     Result:=UTF8ToAnsi(s)
Index: components/lazutils/utf8process.pp
===================================================================
--- components/lazutils/utf8process.pp	(revision 50331)
+++ components/lazutils/utf8process.pp	(working copy)
@@ -22,7 +22,8 @@
 interface
 
 uses
-  Classes, SysUtils, Process, FileUtil, LazFileUtils, LazUTF8, LazUtilsStrConsts;
+  Classes, SysUtils, Process, FileUtil, LazFileUtils, LazUTF8, LazUtilsStrConsts,
+  FPCAdds;
 
 type
   { TProcessUTF8 }
@@ -244,7 +245,13 @@
 procedure TProcessUTF8.Execute;
 begin
   UpdateEnvironment;
+  {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
+  DefaultSystemCodePage := RealSystemCodePage;
+  {$ENDIF}
   inherited Execute;
+  {$IF (FPC_FULLVERSION >= 30000) AND NOT DEFINED(DisableUTF8RTL)}
+  DefaultSystemCodePage := CP_UTF8;
+  {$ENDIF}
 end;
 
 function FindFilenameOfCmd(ProgramFilename: string): string;
lazutils.patch (2,541 bytes)

Mattias Gaertner

2015-12-03 16:06

manager   ~0087774

Patch was not stable.

Issue History

Date Modified Username Field Change
2015-11-11 08:20 Vasily I.Volchenko New Issue
2015-11-11 08:21 Vasily I.Volchenko Tag Attached: IDE
2015-11-11 08:21 Vasily I.Volchenko Tag Attached: lazbuild
2015-11-11 08:21 Vasily I.Volchenko Tag Attached: localization
2015-11-11 14:18 Vasily I.Volchenko Note Added: 0087229
2015-11-11 14:19 Vasily I.Volchenko Note Edited: 0087229 View Revisions
2015-11-11 14:51 Juha Manninen Relationship added duplicate of 0019483
2015-11-11 14:52 Juha Manninen Assigned To => Bart Broersma
2015-11-11 14:52 Juha Manninen Status new => assigned
2015-11-11 14:53 Juha Manninen Note Added: 0087230
2015-11-11 14:58 Vasily I.Volchenko Note Added: 0087231
2015-11-11 14:58 Vasily I.Volchenko Note Edited: 0087231 View Revisions
2015-11-11 15:01 Vasily I.Volchenko Note Added: 0087232
2015-11-11 15:02 Vasily I.Volchenko Note Edited: 0087232 View Revisions
2015-11-11 15:14 Martok Note Added: 0087233
2015-11-11 15:24 Vasily I.Volchenko Note Added: 0087234
2015-11-11 15:31 Vasily I.Volchenko Note Edited: 0087234 View Revisions
2015-11-11 16:21 Martok Note Added: 0087236
2015-11-11 16:46 Vasily I.Volchenko Note Added: 0087237
2015-11-11 17:06 Martok Note Added: 0087238
2015-11-11 18:01 Bart Broersma LazTarget => -
2015-11-11 18:01 Bart Broersma Note Added: 0087239
2015-11-11 18:01 Bart Broersma Assigned To Bart Broersma =>
2015-11-11 18:01 Bart Broersma Status assigned => acknowledged
2015-11-11 18:11 Michl Note Added: 0087241
2015-11-11 19:09 Juha Manninen Relationship added related to 0028962
2015-11-11 19:14 Juha Manninen Note Added: 0087242
2015-11-11 19:17 Juha Manninen Relationship replaced has duplicate 0019483
2015-11-11 22:16 Bart Broersma Note Added: 0087249
2015-11-11 22:17 Bart Broersma Note Edited: 0087249 View Revisions
2015-11-11 22:18 Bart Broersma Note Edited: 0087249 View Revisions
2015-11-11 22:34 Bart Broersma Note Edited: 0087249 View Revisions
2015-11-11 23:52 Juha Manninen Note Added: 0087252
2015-11-15 10:54 Michl Note Added: 0087288
2015-11-15 10:55 Michl File Added: lazutils.patch
2015-11-15 10:56 Michl Note Edited: 0087288 View Revisions
2015-11-15 10:58 Michl Note Edited: 0087288 View Revisions
2015-11-15 11:05 Michl Note Edited: 0087288 View Revisions
2015-11-15 12:06 Bart Broersma Relationship added child of 0028857
2015-12-03 15:50 Mattias Gaertner Fixed in Revision => 50586
2015-12-03 15:50 Mattias Gaertner Status acknowledged => resolved
2015-12-03 15:50 Mattias Gaertner Resolution open => fixed
2015-12-03 15:50 Mattias Gaertner Assigned To => Mattias Gaertner
2015-12-03 16:06 Mattias Gaertner Note Added: 0087774
2015-12-03 16:06 Mattias Gaertner Status resolved => confirmed
2015-12-04 18:38 Mattias Gaertner Fixed in Revision 50586 => 50586,50596-50605
2015-12-04 18:38 Mattias Gaertner Status confirmed => resolved
2019-08-21 00:19 Martin Friebe Relationship added related to 0035991