This project has moved and is read-only. For the latest updates, please go here.

FindMinimum Nonconvergance exception in strange case #2

Mar 7, 2013 at 7:04 PM
Ok, so I'm not going to post this as an issue at the moment simply because one of you guys probably should translate it into something more concise.

I ran the FindMinimum on a complex function that has several "barriers" in it. That is, if a parameter steps beyond a specific value, then the cost function returns double.MaxValue. This has worked fine in the vast majority of the cases, but here is a case where it failed. In the specific situation, there are two parameters to optimize, and they are specified along with the cost for those parameter values below for each iteration. Note that in order to display the actual problem that was occurring below, I had to make sure that the parameter values were displayed using "value.ToString("r")". Otherwise, the string.Format method would round the numbers to 15 decimal places and the truth about what was happening just before iteration #74 would be masked.

Iter 0 of costFunc for Optimization: 2.83790194656588. Parameters:0.77, 0.20000000000000076,
Iter 1 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.8277575, 0.20000000000000076,
Iter 2 of costFunc for Optimization: 4.61886888724584. Parameters:0.7122425, 0.20000000000000076,
Iter 3 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.79206140189477792, 0.20000000000000076,
Iter 4 of costFunc for Optimization: 3.43800035851101. Parameters:0.74793859810522212, 0.20000000000000076,
Iter 5 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77842670568433381, 0.20000000000000076,
Iter 6 of costFunc for Optimization: 3.05542243573483. Parameters:0.76157329431566623, 0.20000000000000076,
Iter 7 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.7732187151582236, 0.20000000000000076,
Iter 8 of costFunc for Optimization: 2.91928085353884. Parameters:0.76678128484177643, 0.20000000000000076,
Iter 9 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.771229439790337, 0.20000000000000076,
Iter 10 of costFunc for Optimization: 2.86873694418091. Parameters:0.76877056020966306, 0.20000000000000076,
Iter 11 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77046960421278721, 0.20000000000000076,
Iter 12 of costFunc for Optimization: 2.84964354208257. Parameters:0.76953039578721283, 0.20000000000000076,
Iter 13 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77017937284802462, 0.20000000000000076,
Iter 14 of costFunc for Optimization: 2.84238153714782. Parameters:0.76982062715197541, 0.20000000000000076,
Iter 15 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77006851433128654, 0.20000000000000076,
Iter 16 of costFunc for Optimization: 2.8396122246787. Parameters:0.7699314856687135, 0.20000000000000076,
Iter 17 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.770026170145835, 0.20000000000000076,
Iter 18 of costFunc for Optimization: 2.83855510186147. Parameters:0.76997382985416507, 0.20000000000000076,
Iter 19 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000999610621845, 0.20000000000000076,
Iter 20 of costFunc for Optimization: 2.83815141322963. Parameters:0.76999000389378158, 0.20000000000000076,
Iter 21 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000381817282026, 0.20000000000000076,
Iter 22 of costFunc for Optimization: 2.83799723195101. Parameters:0.76999618182717977, 0.20000000000000076,
Iter 23 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000145841224243, 0.20000000000000076,
Iter 24 of costFunc for Optimization: 2.83793834199402. Parameters:0.7699985415877576, 0.20000000000000076,
Iter 25 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.770000557063907, 0.20000000000000076,
Iter 26 of costFunc for Optimization: 2.83791584833128. Parameters:0.769999442936093, 0.20000000000000076,
Iter 27 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000021277947861, 0.20000000000000076,
Iter 28 of costFunc for Optimization: 2.83790725656031. Parameters:0.76999978722052143, 0.20000000000000076,
Iter 29 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000008127452868, 0.20000000000000076,
Iter 30 of costFunc for Optimization: 2.83790397480218. Parameters:0.76999991872547136, 0.20000000000000076,
Iter 31 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000003104410752, 0.20000000000000076,
Iter 32 of costFunc for Optimization: 2.83790272128305. Parameters:0.76999996895589251, 0.20000000000000076,
Iter 33 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.770000011857794, 0.20000000000000076,
Iter 34 of costFunc for Optimization: 2.8379022424815. Parameters:0.769999988142206, 0.20000000000000076,
Iter 35 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000452927431, 0.20000000000000076,
Iter 36 of costFunc for Optimization: 2.83790205959559. Parameters:0.76999999547072573, 0.20000000000000076,
Iter 37 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.7700000017300288, 0.20000000000000076,
Iter 38 of costFunc for Optimization: 2.83790198973939. Parameters:0.76999999826997123, 0.20000000000000076,
Iter 39 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.7700000006608122, 0.20000000000000076,
Iter 40 of costFunc for Optimization: 2.83790196305669. Parameters:0.76999999933918784, 0.20000000000000076,
Iter 41 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000025240778, 0.20000000000000076,
Iter 42 of costFunc for Optimization: 2.83790195286482. Parameters:0.76999999974759226, 0.20000000000000076,
Iter 43 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000009641123, 0.20000000000000076,
Iter 44 of costFunc for Optimization: 2.83790194897187. Parameters:0.76999999990358881, 0.20000000000000076,
Iter 45 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000003682578, 0.20000000000000076,
Iter 46 of costFunc for Optimization: 2.83790194748489. Parameters:0.76999999996317425, 0.20000000000000076,
Iter 47 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000001406621, 0.20000000000000076,
Iter 48 of costFunc for Optimization: 2.83790194691692. Parameters:0.76999999998593383, 0.20000000000000076,
Iter 49 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000537283, 0.20000000000000076,
Iter 50 of costFunc for Optimization: 2.83790194669997. Parameters:0.7699999999946272, 0.20000000000000076,
Iter 51 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000205226, 0.20000000000000076,
Iter 52 of costFunc for Optimization: 2.83790194661711. Parameters:0.76999999999794777, 0.20000000000000076,
Iter 53 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.770000000000784, 0.20000000000000076,
Iter 54 of costFunc for Optimization: 2.83790194658545. Parameters:0.76999999999921609, 0.20000000000000076,
Iter 55 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000029944, 0.20000000000000076,
Iter 56 of costFunc for Optimization: 2.83790194657336. Parameters:0.76999999999970059, 0.20000000000000076,
Iter 57 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000011437, 0.20000000000000076,
Iter 58 of costFunc for Optimization: 2.83790194656874. Parameters:0.76999999999988566, 0.20000000000000076,
Iter 59 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000004365, 0.20000000000000076,
Iter 60 of costFunc for Optimization: 2.83790194656697. Parameters:0.76999999999995639, 0.20000000000000076,
Iter 61 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000001667, 0.20000000000000076,
Iter 62 of costFunc for Optimization: 2.8379019465663. Parameters:0.76999999999998336, 0.20000000000000076,
Iter 63 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000000635, 0.20000000000000076,
Iter 64 of costFunc for Optimization: 2.83790194656605. Parameters:0.76999999999999369, 0.20000000000000076,
Iter 65 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000000246, 0.20000000000000076,
Iter 66 of costFunc for Optimization: 2.83790194656594. Parameters:0.76999999999999758, 0.20000000000000076,
Iter 67 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000000091, 0.20000000000000076,
Iter 68 of costFunc for Optimization: 2.83790194656591. Parameters:0.76999999999999913, 0.20000000000000076,
Iter 69 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000000035, 0.20000000000000076,
Iter 70 of costFunc for Optimization: 2.8379019465659. Parameters:0.76999999999999968, 0.20000000000000076,
Iter 71 of costFunc for Optimization: 1.79769313486232E+308. Parameters:0.77000000000000013, 0.20000000000000076,
Iter 72 of costFunc for Optimization: 2.83790194656589. Parameters:0.76999999999999991, 0.20000000000000076,
Iter 73 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
Iter 74 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
Iter 75 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
Iter 76 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
Iter 77 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
.
.
.
Iter 252 of costFunc for Optimization: 2.83790194656589. Parameters:0.77, 0.20000000000000076,
Mar 8, 2013 at 7:03 PM
Thanks, Skybluecodeflier. Is it possible for me to get the cost function that is doing this, perhaps via email, or do business considerations make that not possible?

I will try to see what I can do to improve the behavior in this case, but I think the real solution here is for us to implement a minimization algorithm which knows about constraints. An unconstrained algorithm is always going to be making some assumptions about smoothness that are going to be violated by the barrier approach, and I would expect problems when the minimum is close the the barrier.
Mar 9, 2013 at 12:04 AM
Edited Mar 9, 2013 at 12:06 AM
Unfortunately there is no easy way for you to get access to the cost function that is causing this functionality. There is a lot of proprietary code involved, and it really is an extremely complex cost function. I might be able to create a cost function that is much simpler that causes the same kind of issue. We'll see.

One idea though on the bug indicated above- perhaps the minimization algorithm is trying to reduce the step size for the first parameter to a value that is smaller than can be noticably added or subtracted from 0.77 (given the limits of the double structure)? Then of course there would be no change in the result, since the step size is just too small for the double to hold it given only 16 significant figures. I would imagine that would be a source of having iterations 77-252 appear to do absolutely nothing.

But ultimately I would agree that the real solution is to use a constrained minimization algorithm. Have you taken a look at the link I mentioned in the other discussion? Someone else made a constrained minimization function that may work for the purposes of this library. It seems to work pretty well and has a nice set of parameters for changing the behavior of the minimization. But if you have a better solution than that, not a problem.
Mar 12, 2013 at 8:52 PM
Thanks again for your engagement. I replied to the other thread with positive thoughts on the algorithm used there.

There are a lot of minor features and bug fixes in the current tree that we would like to get out before tackling a complete revamp of the minimization system. Therefore you should expect a release soon that does not contain a constrained minimization algorithm (but does address bug 7306!), followed by a next release that does.
Mar 12, 2013 at 9:58 PM
Thanks. I will see if I can create a cost function that will reproduce the above problem then, assuming that perhaps we can still get a fix for it into the next release. Let me know if anything changes in your plans on this note.
May 23, 2013 at 12:16 AM
Just to let you know, I have not been able to create a simpler cost function that reproduces this problem.