code generates different result using snopt_cwrap.h and f2c interface

[hongkai-dai]

I have a simple optimization problem, and I implemented one function that uses the f2c.h, and another one using snopt_cwrap.h. They generate different output. The one with f2c is more accurate than the one with snopt_cwrap.h (as can be seen in the "Optimal" column of the print out file). The two print out files are
f2c_out.txt
snopt_cwrap_out.txt

The only difference in the setup part is the "Unbounded objective", and I think the function evaluation part are the same. Should I expect that snopt_cwrap.h and f2c.h produce different result? I am using SNOPT.7.6

[gnowzil]

One thing I notice is the first LU factors for each run don't seem to match.

Do you think you could provide the example? If not, can you rerun with "System information yes". That might provide some extra info.

[hongkai-dai]

Thanks Elizabeth! I added "System information yes", and here is the output file
f2c_out.txt
snopt_cwrap_out.txt

One thing I notice is that at iteration 0, snopt_cwrap_out.txt says the Hessian is set to scaled identity matrix, while f2c_out.txt doesn't say so. Would this be the reason?

I am writing a wrapper for SNOPT to be called through drake, and the current code that reproduce this problem is very complicated. Do you think the print out file provides enough information? If not, I will try to create a small example to reproduce the problem.

[gnowzil]

No I think that's a red herring. Extra output messages were added in the last few years. Sorry, I forgot to include some other options:
Print frequency 1
Summary frequency 1
Minor print level 1
Major print level 11
System information yes

It kinda just looks like a different basis is chosen initially (thus different LU factors). Or maybe a different initial point? Note the different values in the "optimal" column at iteration 0.

[hongkai-dai]

Thanks!

Here are the print out files.
snopt_cwrap_out.txt
f2c_out.txt

I think the input values are the same. I checked x, xmul, xstate, all are set to 0.

[hongkai-dai]

Here is the code to reproduce the output, using snopt_cwrap.h
lp.c.zip
I also need to make a change to snopt/interfaces/Submakefile to compile lp.c

--- a/interfaces/Submakefile
+++ b/interfaces/Submakefile
@@ -23,7 +23,7 @@ SNOPT_C_LO   = $(SNOPT_C_FILES:%=$(API_OBJDIR)/%.lo)
 SNOPT_CPP_LO = $(SNOPT_CPP_FILES:%=$(API_OBJDIR)/%.lo)

 #-----------------------------------------------------------------------
-SNOPT_C_EX   = toyA  toyB  toyC  hs118
+SNOPT_C_EX   = toyA  toyB  toyC  hs118 lp
 SNOPT_CPP_EX = sntoya  sntoyb  sntoyc  catmixa  hs118

 SNOPT_C_EXAMPLES   = $(SNOPT_C_EX:%=$(CEX_OBJDIR)/%)

BTW, how should I build the examples in snopt/cexamples folder, such as csntoya? This example uses f2c interface. I tried the following commands

$ configure --enable-f2c --with-c --with-cxx
$ make cexamples

but I only see the executables from snopt/interfaces/cexamples; I do not see the target from snopt/cexamples.

[gnowzil]

I don't think there's a bug. I rewrote the problem in Fortran to call snopt directly and I get the same results. A lot of stuff changed internally in 2014 that is causing the differences you're seeing. I will check with my colleagues to see what they think, but they are currently out-of-office.

Out of curiosity, why do you specify the derivatives of the objective as nonlinear even though they're all constant/linear? If you formulate the problem entirely as an LP, you would "fix" this issue.

I stopped doing anything to support the f2c version years ago. I would just manually compile/link the example. --with-c and --with-cpp are not related to the f2c version.

[hongkai-dai]

I see, so I won't worry about it any more.

You are absolutely right that I should have written the objective as a linear function. I currently wrote it as a nonlinear function due to a drawback in my wrapper, that treats any cost function as nonlinear functions. I will fix that bug soon. But there are other problems with actual nonlinear cost, that I also observe that the f2c interface and snopt_cwrap interface give different results.

Apologize for my ignorance on compilation, to compile the f2c version, I tried

$ gcc -I$HOME/snopt/build/include -L$HOME/snopt/build/lib -lf2c -lsnopt7 MY_TEST_FILE.c -o MY_TEST_FILE

but the compiler complains "undefined reference to snopenappend_'", should I link to some other libraries? I thought snopenappend_` is in the f2c library.

[hongkai-dai]

Sorry to bother you again, but we have ran into more problems when we switched to snopt-interface from f2c interface. Some of our problems that can be solved using f2c interface, is not solved any more with snopt-interface. Here are the outputs logs.

f2c.txt
snopt_interface.txt

It seems to me that at the beginning before the major iteration, the LU factorization generates different results.

Should I add some more flags to generate more info in the output file? Should I set "System info" to Yes?

[hongkai-dai]

One thing I do not quite understand, is that in the log snopt_interface.txt, it takes 1000 major iterations, but only 13 Itns. Should Itns be the number of minor iterations? I would suppose that the number of minor iterations should be more than the number of major iterations?

[hongkai-dai]

Here are two logs with more information.
f2c.txt
snopt_interface.txt

[gnowzil]

Can you tell me what version/date of SNOPT you're running in the f2c problems? Your output always cuts off the output displaying that info.

[hongkai-dai]

It is snopt 7.6.1

[gnowzil]

For the f2c'd version? I didn't think we got past 7.5 with f2c. Can you check -- in file csrc/sn02lib.c, line 54 should be the line that prints the version.

[hongkai-dai]

I see, the f2c version is 7.4-12

We have several other problems, that are also solved very quickly by the f2c interface (with 7.4-12), but is not solved with 7.6.1 using c interface.