Backend='numpy'

[Peter230655]

I made a comparison between the backends, using mosty examples for examples-gallery and got these results for running ``Problem.
Currently, the default backend is cyipopt.
Based on my data, backend='numpy' has a huge advantage for low number of operations in the eoms, and backend='cyipopt' has a big one for large numbers.
I suppose the cutoff may be around 50,000 - 75,000 operations.

I suggest this:

• change the default value of backend in Problem to None
• if None, set backend = 'numpy' if count_ops < 50,000, else to cyipot

I think, the exact number is not very crucial: for small op_count, say, 2,000 numpy is MUCH faster, for large op_counts, say 300,000 cyipot is.

If #393 will be successful before the next release of opty, we can easily change this number or make 'numpy' the default, if it beats cyipot on any number.

25-03-05 backend_comparisons.xlsx

[moorepants]

Opty has existed for almost 10 years and it ran the Cython backend. So changing to use the NumPy backend (even for low operation count) is (sort of) a backwards incompatible change. I'm not sure I want to do that.

There is also a computational cost to running count_ops(). If the equations are really huge, count_ops() may take a very long time. So we'd need a function that short circuits once 50,000 operations are counted. Maybe that already exists in SymPy.

Given that someone can set backend='numpy' very easily, I'm not sure why change the expected behavior that is 10 years old.

[moorepants]

Nice analysis btw, it is useful for the conference abstract we are writing. Sad you aren't interested in being an author :)

[Peter230655]

:-)
I will continue my list for a while. I need to find simulations with 50,000 < count_ops(eom) < 100,000

[Peter230655]

I am basically a VERY interested user of opty, how could I be a co author?? Being acknowledged is an honor for me!

Then we should maybe put it in the docstrings that backend='numpy' is better for small op_counts?

Yes, count_ops takes time. But nothing compared to 'cyipopt'
vs. 'numpy' to select the better backend.

[moorepants]

You are no longer a user of opty, you have made the second most contributions to opty. You are a developer of opty.

You can definitely improve the docstrings with info about numpy backend.

You may be also missing that once you make a model, you can cache that model and never have to do it again. This means that the cyipopt backend is always better than the numpy backend. It seems to be important because you build the model over and over again when you don't have to do so. The issue we have about #107 is about this.

[Peter230655]

Thanks! I like to make them - within my limitations.

I pushed a change in the docstrings.

I did not think about backwards compatibility.
I think, backend='numpy' is particularly helpful if one has to iterate form a simpler problem to a harder one and Problem has to be in the loop, like my example particle_on_line. I remember #107, but I was not too clear what it meant.

I will close this comment as taken care of, I will continue my list.

[moorepants]

Yes, when building a model, you have to iterate and it is annoying to wait for the evaluation, so yes, just use numpy backend when doing that.

[Peter230655]

I expanded the comparison list as follows:
I had an old 2D n-link pendulum simulation. So, I simply added a control torque at each link, and set up Problem.
I ran it on Jupyter notebook, as it was there. Each run started from scratch by using the restart button.
Of course, this is a very special model, with n links, there are 2n state variables and n controls, so this may not reflect the 'average' simulation, but easiest to get any number of count_ops(eom)
I did not add the counter reactions in the eom, so the model is mechanically incorrect - I do not think, Problem cares.
25-03-05 backend_comparisons.xlsx
@moorepants Would it be of interest to also get the time it takes to solve the problem, or is the simulation too special to be meaningful.
If yes, of course I can do, may take me a few days.

NB:
As always, I check the time with time.time()
I recall I read, that this is accurate enough for this kind of rough idea.

[Peter230655]

We have come a long way in those 10+ years!!

Not sure that is so true. That all is basically the same as what is in SymPy, PyDy, and Opty.

But the ease of using the programs! Was opty this easy to use then?

[Peter230655]

For examples-gallery simulations, where we use a stored solution, the speed of Problem matters, otherwise the speed od prob.solve(..) matters only.

For the examples gallery, yes every step in the computation matters. But that isn't what matters when you want to solve what I would call "research grade" problems, i.e. problems that are on the cutting of of being solvable or not.

Re-running the examples. Adding the time to solve the problem.

[moorepants]

You may want to see what we are doing here: mechmotum/ISBTGCS2025#1

[Peter230655]

For examples-gallery simulations, where we use a stored solution, the speed of Problem matters, otherwise the speed od prob.solve(..) matters only.

For the examples gallery, yes every step in the computation matters. But that isn't what matters when you want to solve what I would call "research grade" problems, i.e. problems that are on the cutting of of being solvable or not.

Of course I have never done research type stuff.
If one wants to change anything that is in the signature of Problem, e.g. num_nodes, bounds, instance_constraints, would one not have to run Problem again?

[moorepants]

Ideally not, but practically no. Like I have this issue open #112 and #107 which, if solved, would allow the user to change things without having to know the internals. Right now you can cache anything up to solve if you want to, but you have to manually do this.

[Peter230655]

I ran my simple comparisons again, checking the time to solve the problem, too.
25-03-05 backend_comparisons.xlsx

• It seeems, that with backend='numpy' it always takes longer to solve the problem, no matter how small it is
• In two cases it did not converge, while the default backend did converge.
• the pendulum was no good, with more than 3 links, neither backend found a way to get it upright, at least not with linear initial guesses.

Seems my #398 was a bit premature. Can you easily de-merge it or is it better I make a new PR where I take it out?
Or I change it about like this: for smaller numbers in the eoms numpy builds the Problem part very fast, but the solution part is gernerally faster with cyipopt?

[moorepants]

I reverted it.

[moorepants]

It seeems, that with backend='numpy' it always takes longer to solve the problem, no matter how small it is

Yes, that should always be true.