View Issue Details

IDProjectCategoryView StatusLast Update
0037354LazarusTAChartpublic2020-07-23 18:08
Reporterdonovaly Assigned Towp  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformWindows 64bitOSWindows 
Product Version2.0.8 
Summary0037354: When fitting ConfidenceLevel cannot be changed
DescriptionI use the fit demo that comes with Lazarus. It has the nice feature to show the confidence interval.
In the docs I read that I can change the confidence interval by setting the variable
ConfidenceLevel

However, when I do this and set it te.g. to 0.5 I still get exactly the same result as with a level of 0.95.

So instead of setting ConfidenceLevel in the program, I tried to set it directly in TAChart. Thus I changed in TAFuncSeres.pas the line
FConfidenceLevel := 0.9;
to
FConfidenceLevel := 0.5;

But the result is the same - nothing changes, the shown interval is always the same (shown in the attached screenshot.
TagsNo tags attached.
Fixed in Revision63566
LazTarget2.2
WidgetsetWin32/Win64
Attached Files

Activities

donovaly

2020-07-15 03:07

reporter   ~0124023

fitdemo_b363fXOUb7.png (30,148 bytes)   
fitdemo_b363fXOUb7.png (30,148 bytes)   

wp

2020-07-15 15:55

developer   ~0124043

Fixed. Please test and close if ok.

donovaly

2020-07-16 02:25

reporter   ~0124070

Last edited: 2020-07-16 02:30

View 4 revisions

I tested it by upgrading to Lazarus 2.0.10, then I copied the TAChart files I found at Github:
https://github.com/graemeg/lazarus/commit/ee4d4295af83acf6a58daf644d8341509387a5e6?diff=unified

The bug is basically fixed, many thanks!

However, when decreasing the confidence level, the resulting lines are narrower to the fit. I expected them to become wider since e.g. a confidence of only 0.10 means less confidence that the fit follows the data points.

I cannot close this bug because your fix introduced new issues when executing the fit demo:

1. start the fit demo and simply change the fit function to "Linear"
result: see the error attached as screenshot

2. increase the confidence level to 1.0
result: "Invalid floating point operation"

3. decrease the confidence to 0.0
result: the program crashes

When compiling the TAChart package I also get now these warnings that you might have a look at:

Compile package TAChartLazarusPkg 1.0: Success, Warnings: 24
tachartutils.pas(1005,19) Warning: function result variable of a managed type does not seem to be initialized
tageometry.pas(127,19) Warning: function result variable of a managed type does not seem to be initialized
tageometry.pas(695,19) Warning: function result variable of a managed type does not seem to be initialized
tachartaxis.pas(441,41) Warning: Symbol "OnMarkToText" is deprecated: "Use "OnGetMarkText"
tagraph.pas(625,84) Warning: Symbol "OnBeforeDrawBackground" is deprecated: "Use OnBeforeCustomDrawBackground instead"
tagraph.pas(626,27) Warning: Symbol "OnBeforeDrawBackground" is deprecated: "Use OnBeforeCustomDrawBackground instead"
tagraph.pas(634,83) Warning: Symbol "OnAfterDrawBackground" is deprecated: "Use OnAfterCustomDrawBackground instead"
tagraph.pas(635,26) Warning: Symbol "OnAfterDrawBackground" is deprecated: "Use OnAfterCustomDrawBackground instead"
tagraph.pas(954,76) Warning: Symbol "OnExtentChanging" is deprecated: "Used OnExtentValidate instead"
tagraph.pas(955,23) Warning: Symbol "OnExtentChanging" is deprecated: "Used OnExtentValidate instead"
tagraph.pas(980,26) Warning: Symbol "OnAfterDraw" is deprecated: "Use OnAfterCustomDraw instead"
tagraph.pas(981,16) Warning: Symbol "OnAfterDraw" is deprecated: "Use OnAfterCustomDraw instead"
tagraph.pas(1011,82) Warning: Symbol "OnBeforeDrawBackWall" is deprecated: "Use OnBeforeCustomDrawBackWall instead"
tagraph.pas(1012,25) Warning: Symbol "OnBeforeDrawBackWall" is deprecated: "Use OnBeforeCustomDrawBackWall instead"
tagraph.pas(1027,81) Warning: Symbol "OnAfterDrawBackWall" is deprecated: "Use OnAfterCustomDrawBackWall instead"
tagraph.pas(1028,24) Warning: Symbol "OnAfterDrawBackWall" is deprecated: "Use OnAfterCustomDrawBackWall instead"
taseries.pas(63,42) Warning: Symbol "TBeforeDrawBarEvent" is deprecated
taseries.pas(1259,79) Warning: Symbol "OnBeforeDrawBar" is deprecated: "Use OnCustomDrawBar instead"
taseries.pas(1260,22) Warning: Symbol "OnBeforeDrawBar" is deprecated: "Use OnCustomDrawBar instead"
tatools.pas(2134,21) Warning: Symbol "OnDraw" is deprecated: "Use OnCustomDraw"
tatools.pas(2135,11) Warning: Symbol "OnDraw" is deprecated: "Use OnCustomDraw"
tafitlib.pas(134,19) Warning: function result variable of a managed type does not seem to be initialized
tafitlib.pas(219,19) Warning: function result variable of a managed type does not seem to be initialized
tafuncseries.pas(1949,19) Warning: function result variable of a managed type does not seem to be initialized
lazarus_lvl661IjJw.png (4,109 bytes)   
lazarus_lvl661IjJw.png (4,109 bytes)   

wp

2020-07-16 13:37

developer   ~0124091

> when decreasing the confidence level, the resulting lines are narrower to the fit. I expected them to become wider since e.g. a confidence of only 0.10 means less confidence that the fit follows the data points.

I was surprised about this, too. For comparison I only have LazStats available, but it shows the same behavior. (I do have some doubt in LazStats, though, because the conficence lines have a different distance than in TAChart, but in TAChart I see the same confidence lines as in Excel). If you have access to a "trust-worthy" statistics program I would appreciate if you could test this.

On the other hand, I think that the conficence lines could be understood in a different way compatible with the observation: The starting point is the scatter of the data points around the fitted line. Requesting a a high confidence level means that the confidence lines must have a LARGE distance from the fitted line because almost all the deviations must be within the confidence bands. Conversely, when the conficence level is reduced you accept more data points being outside the confidence band and the lines can be drawn narrower to the fitted line (because the scatter in the data is the same, it does not depend on the choice of the confidence level).

-------------

>1. start the fit demo and simply change the fit function to "Linear"
> result: see the error attached as screenshot

I use trunk and cannot see this. There were a lot of changes within TAChart which did not make it into the 2.0.x bug fix releases. Therefore, I think that your way of partial patching 2.0.10 is not suffcient. Try to replace the entire TAChart folder, but I cannot guarantee that this combination still compiles. Ideally you should use Laz trunk for this purpose.

-------------

> 2. increase the confidence level to 1.0
> result: "Invalid floating point operation"
>
> 3. decrease the confidence to 0.0
> result: the program crashes

OK, of course I do not check all error conditions in a demo program. The cases you mention are due to limitations of the underlying statistical functions which are defined only between 0 and 1 (interval ends excluced) - a probability of exactly 0% or 100% simply does not exist in the statistical sense. I modified the limits of the spin edit to limit the input range at 0.01 and 0.99 (of course I could move much closer to the limits by setting Min = 1E-6 and Max = 0.999999, but then the GUI would display "0" and "1" which is not correct either .

donovaly

2020-07-16 16:48

reporter   ~0124098

> Requesting a a high confidence level means that the confidence lines must have a LARGE distance from the fitted line because almost all the deviations must be within the confidence bands.

I also had this thought, but when you do this with the demo, you see that with 1% confidence there are still many data points out of the boundary lines. So this cannot be the explanation.

> Try to replace the entire TAChart folder,

I did so. I installed Lazarus 2.0.10 and then replaced its whole TA Chart component folder. As you can see I can compile, just gets the warnings I reported.

> The cases you mention are due to limitations of the underlying statistical functions which are defined only between 0 and 1

I can understand for a level of 0.0 (division by zero), but I wondered about the value 1.0. However, my point was that the demo should demonstrate what is possible and not what is not ;-). And now you you already did it right:
> I modified the limits of the spin edit to limit the input range at 0.01 and 0.99
Thanks.

> If you have access to a "trust-worthy" statistics program I would appreciate if you could test this.

I trust the program JASP (free, OpenSource, developed by the Uni Amsterdam). I am in contact with them and will ask them.

wp

2020-07-16 22:13

developer   ~0124102

Ah, I forgot to discuss the warnings: I removed the "not initialized" warning. As for the "deprecated" warnings, It is our policy to keep them for the current and next release versions. Since, IIRC, the related code was marked as such during the life time of v2.0.x it will be removed in trunk of v2.3 and be in the product of v2.4

donovaly

2020-07-19 18:13

reporter   ~0124173

> Requesting a a high confidence level means that the confidence lines must have a LARGE distance from the fitted line because almost all the deviations must be within the confidence bands.

OK, I got it now: the confidence interval shows the area in which the optimal fit is in. So for a confidence level of 0.99, the area is wider while for 0.01 ist is super narrow because then with a probability of 99% the optimal fit is not in this area.

Besides this, what about the new exception introduced by fixing this bug? I attached a screencast to demonstrate the issue. This is with current SVN trunk and directly after opening the fit demo.

(Attached is also the list of unused variables that might be removed from the fit demo.)
ekjwJRlMnR.gif (190,602 bytes)   
ekjwJRlMnR.gif (190,602 bytes)   
lazarus_xDqicXACff.png (8,954 bytes)   
lazarus_xDqicXACff.png (8,954 bytes)   

wp

2020-07-19 18:36

developer   ~0124174

> what about the new exception introduced by fixing this bug?

You did not mention that I must check one of the "Show intervals" boxes before activating the linear fit. No error occurs without it.
I'll have a look...

wp

2020-07-20 10:57

developer   ~0124185

Done

donovaly

2020-07-20 16:30

reporter   ~0124189

Many thanks.

Will this bugfix be backported to Lazarus 2.2 and/or 2.0.12?

How does sending a patch work for Lazarus? Can I e.g. simply upload a unified diff file to a bug report? I am asking because I only know Git and Lazarus is using SVN.

wp

2020-07-20 20:08

developer   ~0124192

2.0.12, if it ever would be released, will be a bug fix release. Since there were too many changes it will not contain the current bugfix.
2.2, however, will be a new version containing the state of trunk at the time of tagging, i.e. it will contain the bugfix.

I'm no git specialist. Please ask in the forum or on the Lazarus mailing list.

donovaly

2020-07-21 01:03

reporter   ~0124197

> I'll have a look...

Using latest SVN trunk, I get another crash, see the attached screencast. So using the power model crashes.

> 2.2, however, will be a new version containing the state of trunk at the time of tagging, i.e. it will contain the bugfix.

Thank you.

> I'm no git specialist. Please ask in the forum or on the Lazarus mailing list.

OK. I was only asking because Maybe I can fix this fit demo bug https://bugs.freepascal.org/view.php?id=37369
teLeKtHZ1Y.gif (154,043 bytes)   
teLeKtHZ1Y.gif (154,043 bytes)   

wp

2020-07-21 16:39

developer   ~0124210

> I get another crash, see the attached screencast. So using the power model crashes.

Fixed, as well as some other nuisances. Can't guarantee that they're all...
--------------------
> I was only asking because Maybe I can fix this fit demo bug

I thought this was a general question. I myself do not have problems receiving a git diff file.

donovaly

2020-07-22 01:17

reporter   ~0124220

> Fixed

I tested using current SVN master but I still get a crash. The error massage however, is now different, see attached.
QpBCT6BZ1L.gif (142,357 bytes)   
QpBCT6BZ1L.gif (142,357 bytes)   

wp

2020-07-22 09:55

developer   ~0124221

Done. No idea why this happens only on 64-bit.

donovaly

2020-07-23 01:44

reporter   ~0124246

Many thanks. Everything works now for me.

I think the bug should be open as reference until the fix is released with Lazarus 2.2.

wp

2020-07-23 14:19

developer   ~0124268

No. This does not make sense, For 2,2, the state of trunk will be used at the time of tagging - this is automatic. 2.2 it will contain the current fix, no matter which state the report has. Leaving the report open, on the other hand, will cause clutter of various bug tracker views in which the close reports normally are removed.

donovaly

2020-07-23 15:05

reporter   ~0124271

> This does not make sense,

To me it does: I a normal user I update to Lazarus 2.10. so the latest stable release I can get. there I find a bug. Searching the bug databases does not show me that my bug has been maybe already fixed for a further release nor that it is already known.
So the bug is in the latest stable release but marked fixed. For me as user it is not because I won't take an unreleased version.

So at least in order to find confirmed bugs in the latest stable release, these bug reports should stay open. Other projects first close them the day the a release was published in which the fix is included and that is very sensible in my opinion.

wp

2020-07-23 15:23

developer   ~0124272

If this were correct, no bug report should ever be closed before the release of a new version.

You can always filter the report list by your name and you can also reset the filters to show the closed reports. And when you search for a bug ID by typing it into the box at the top/right of the bugtracker form you will also find closed reports.

donovaly

2020-07-23 17:39

reporter   ~0124275

> If this were correct, no bug report should ever be closed before the release of a new version.

Yes. This is the case in other projects. Bugs gets the label/status "fixed" but the reports are kept open.

> can also reset the filters to show the closed reports.

My experience after over a decade with OpenSource programming (project www.lyx.org) is that especially new users don't do this and/or don't know this. Now I am a developer of the project FreeCAD and there, we have the same problem.

However, I am not part of the Lazarus community and changing your current policy would require a brief debate.

wp

2020-07-23 18:03

developer   ~0124277

This is from wiki article about bugtracker: https://wiki.freepascal.org/How_do_I_create_a_bug_report#Understanding_the_Report_Status

"Understanding the report status: ... Closed: the reporter tested the fix and agrees with the fix. Periodically resolved issues that have not been closed by the reporter, will be a closed by the bug tracker administrator."

No word about porting to a release.

Anyway, I do not want to lead an endless discussion here. Close it nor keep it open, as you like...

donovaly

2020-07-23 18:08

reporter   ~0124279

> from wiki article about bugtracker

I understood meanwhile. I just wanted to point out that I think this should be changed.

As I wrote, I will follow your rules thus closing.

Issue History

Date Modified Username Field Change
2020-07-15 03:06 donovaly New Issue
2020-07-15 03:07 donovaly Note Added: 0124023
2020-07-15 03:07 donovaly File Added: fitdemo_b363fXOUb7.png
2020-07-15 10:23 wp Assigned To => wp
2020-07-15 10:23 wp Status new => assigned
2020-07-15 15:55 wp Status assigned => resolved
2020-07-15 15:55 wp Resolution open => fixed
2020-07-15 15:55 wp Fixed in Revision => 63566
2020-07-15 15:55 wp LazTarget => -
2020-07-15 15:55 wp Widgetset Win32/Win64 => Win32/Win64
2020-07-15 15:55 wp Note Added: 0124043
2020-07-16 02:25 donovaly Note Added: 0124070
2020-07-16 02:25 donovaly File Added: lazarus_lvl661IjJw.png
2020-07-16 02:26 donovaly Note Edited: 0124070 View Revisions
2020-07-16 02:27 donovaly Note Edited: 0124070 View Revisions
2020-07-16 02:30 donovaly Note Edited: 0124070 View Revisions
2020-07-16 13:37 wp Note Added: 0124091
2020-07-16 16:48 donovaly Note Added: 0124098
2020-07-16 22:13 wp Note Added: 0124102
2020-07-19 18:13 donovaly Note Added: 0124173
2020-07-19 18:13 donovaly File Added: ekjwJRlMnR.gif
2020-07-19 18:13 donovaly File Added: lazarus_xDqicXACff.png
2020-07-19 18:36 wp Note Added: 0124174
2020-07-20 10:57 wp Note Added: 0124185
2020-07-20 16:30 donovaly Note Added: 0124189
2020-07-20 20:08 wp Note Added: 0124192
2020-07-21 01:03 donovaly Note Added: 0124197
2020-07-21 01:03 donovaly File Added: teLeKtHZ1Y.gif
2020-07-21 16:39 wp Note Added: 0124210
2020-07-22 01:17 donovaly Note Added: 0124220
2020-07-22 01:17 donovaly File Added: QpBCT6BZ1L.gif
2020-07-22 09:55 wp Note Added: 0124221
2020-07-23 01:44 donovaly Note Added: 0124246
2020-07-23 14:14 wp Status resolved => assigned
2020-07-23 14:14 wp Resolution fixed => open
2020-07-23 14:14 wp Status assigned => resolved
2020-07-23 14:14 wp Resolution open => fixed
2020-07-23 14:14 wp LazTarget - => 2.2
2020-07-23 14:14 wp Widgetset Win32/Win64 => Win32/Win64
2020-07-23 14:19 wp Note Added: 0124268
2020-07-23 15:05 donovaly Note Added: 0124271
2020-07-23 15:23 wp Note Added: 0124272
2020-07-23 17:39 donovaly Note Added: 0124275
2020-07-23 18:03 wp Note Added: 0124277
2020-07-23 18:08 donovaly Note Added: 0124279