This project has moved and is read-only. For the latest updates, please go here.

Distribution.FitToSample

Nov 7, 2014 at 11:04 AM
Hi,
First of all I wanna to congratulate you for the fantastic work on Meta Numerics. It is a wonderful package.
I am trying to use the FitToSample in the Weibull distribution but I am getting errors on that procedure.
Maybe this is an error (Or maybe I don't know yet how to use it properly).
My code is in VB and is describe below:
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click

    Dim parametros(1) As Double

    Dim números As New Statistics.Sample
    números.Add(12.824) : números.Add(12.855) : números.Add(12.861) : números.Add(12.862) : números.Add(12.863)
    números.Add(12.864) : números.Add(12.865) : números.Add(12.866) : números.Add(12.866) : números.Add(12.866)
    números.Add(12.867) : números.Add(12.867) : números.Add(12.868) : números.Add(12.868) : números.Add(12.87)
    números.Add(12.871) : números.Add(12.871) : números.Add(12.871) : números.Add(12.871) : números.Add(12.872)
    números.Add(12.876) : números.Add(12.878) : números.Add(12.879) : números.Add(12.879) : números.Add(12.881)

    parametros = Statistics.Distributions.WeibullDistribution.FitToSample(números).Parameters()
    Dim distribuição As New Statistics.Distributions.WeibullDistribution(parametros(0), parametros(1))
    MessageBox.Show(distribuição.InverseLeftProbability(0.135 / 100)) 'quantil 0,135%
    MessageBox.Show(distribuição.InverseLeftProbability(99.865 / 100)) 'quantil 99,865%


End Sub
I've tried to debug it, and I saw that the code is stopping on the following Statement
double k1 = FunctionMath.FindZero(kFunction, k0); within WeibullDistribution.cs wich call the findzero function on FunctionMath_Roots.cs

The funny thing is when I change the sample number for the following ones It works properly
    números.Add(0.782792) : números.Add(0.086726) : números.Add(0.010857) : números.Add(0.368851) : números.Add(0.28748)
    números.Add(0.909071) : números.Add(0.150298) : números.Add(0.608511) : números.Add(0.490219) : números.Add(0.151538)
    números.Add(0.156621) : números.Add(0.261364) : números.Add(0.615852) : números.Add(0.183577) : números.Add(0.071625)
    números.Add(0.318152) : números.Add(0.103415) : números.Add(0.478834) : números.Add(0.41213) : números.Add(0.192791)
    números.Add(0.208925) : números.Add(0.234034) : números.Add(0.433043) : números.Add(0.788636) : números.Add(0.84719)
    números.Add(0.459984) : números.Add(0.293006) : números.Add(0.648248) : números.Add(0.154507) : números.Add(0.320434)
    números.Add(0.340069) : números.Add(0.351005) : números.Add(0.37279) : números.Add(0.331414) : números.Add(0.357962)
    números.Add(0.358099) : números.Add(0.560374) : números.Add(0.828459) : números.Add(0.565067) : números.Add(0.626277)
    números.Add(0.054402) : números.Add(0.387835) : números.Add(0.306073) : números.Add(0.303861) : números.Add(0.174402)
    números.Add(0.338571) : números.Add(0.226616) : números.Add(0.371329) : números.Add(0.551373) : números.Add(0.187531)
Also I've tried to change the Ditribution to BetaDistribution and WaldDistribution and both had also problem

For GammaDistribution and LognormalDistribution and NormalDistribution, the code is working OK.

Could you please check this issue?

Best regards
Nov 11, 2014 at 3:00 PM
Edited Nov 11, 2014 at 3:11 PM
Hello,
Just a few more info that I found out.

Software R computed the Shape and Scale as 1812,59359 and 12,87145852 respectively for the first set of data informed.

Looking into \lambda function (Executed from Findzero function), the var s goes to infinity after math.pow(12.824, 1783.95) because the k0 value is very high due the global.logtwo / math.log(x2/x1) function. This infinity value could not be used to calculate the zeros of function (The Findzero loop keep stepping until change the function sign, but this never happened)

Maybe this helps a bit.

kind regards
Dec 2, 2014 at 10:54 AM
Hello

Sorry to ask again, but is there somebody who could enlight us in this issue?

Thanks!
Feb 9, 2015 at 1:32 AM
You are right in your diagnosis. WeibullDistribution.FitToSample reduces the maximum likelihood problem to numerically finding the root of a 1D function. Part of computing this function is finding (<x^k>)^(1/k). This fails because intermediate values are (12.87)^1812 ~ 10^{2023}, which overflows a Double.

Fixing this issue would require us to work out a large-k approximation to our function and apply it in appropriate cases. I will add a bug to do this.

That said, it is very clear that a Weibull distribution is a very poor fit to this sample for any parameters. Weibull just doesn't produce values narrowly clustered around some non-zero value.
Feb 11, 2015 at 8:52 AM
I have checked in a fix. The basic idea is to use the variables w = log(x) - <log(x)> instead of x, so we are no longer taking large numbers to large powers. We ended up being able to speed up the root-finding too. Thanks, this was a great bug.

Oh, and I was wrong about the fit being bad. The goodness-of-fit test indicates that the fit is fine.
Feb 11, 2015 at 7:23 PM
Hi

Thanks for your update!

I will download the patch for that.

Best regards