Use C/C++ extensions for file writers

[whart222]

The key idea is to use C extension libraries to create problem files (LP, NL, GAMS, etc). Preliminary work with POEK and COEK indicates that this approach could significantly reduce the time needed to process a Pyomo model and generate a standard data file.

Specifically, here's a concrete experiment we should be able to easily do:

1. Generate a model with Pyomo
2. Populate a POEK model from the Pyomo model
3. Create a SymbolMap in POEK, and then use COEK to write the LP file

Note that step (2) does not involve creating a canonical expression. Further, step (3) pushes all file I/O to C/C++. Both of these will likely lead to significant improvement in the total runtime using LP files.

Some limitations of this approach:

• COEK currently only supports LP files. Adding support for NL would require additional effort, but given the complexity of that writer I think the potential performance improvements are likely greater.
• I wouldn't expect this approach to support symbolic labels.
• This would create an optional dependency on POEK/COEK.

Note that this approach does not require the use of COEK solver interfaces. This is merely using COEK's ability to (a) express linear/quadratic models, and (b) write them in standard formats.

[qtothec]

To clarify, what are POEK and COEK? Google doesn't turn up useful results.

[whart222]

POEK and COEK are some experimental codes for expressing optimization models in C++ and Python, with a goal towards reassessing the computational bottleneck's we've seen in Pyomo. These have not been released (yet), but they'll be available for use at the hackathon.

[whart222]

The perf_poek_writer branch includes a preliminary demonstration. Comparing the writing time of pmedian8 to files foo.lp and foo.poeklp, the writing time of poeklp is about 17% faster. This demonstration (a) creates a Pyomo model, (b) creates a POEK model, and (c) calls C++ to write the POEK model to an LP file.

This performance test used Pybind11, which is slower than the cffi interface (which I'm not sure is working yet for COEK).

[michaelbynum]

On this front, I am particularly interested in having a persistent interface to poek/coek which can then be used to write lp and nl files. This would essentially give us a fast persistent interface to all of the most used solvers.

[whart222]

FWIW, I testing with the cppyy binding, and writing poeklp was 24% faster for pmedian.

[michaelbynum]

@whart222 Do you happen to have the comparison of just translating the model to an lp file (i.e., excluding the time to generate the pyomo/poek models)?

[whart222]

The perf_poek_writer branch now includes a preliminary demonstration for writing NL files with POEK. Comparing the writing time of pmedian8 to files foo.nl and foo.poeknl, the writing time of poeknl is about 20% faster. This demonstration (a) creates a Pyomo model, (b) creates a POEK model, and (c) calls C++ to write the POEK model to an NL file.

[whart222]

@michaelbynum I'm not sure what "just translating" the model means. The comparison here is fair in that I'm comparing the time it takes for Pyomo to write with the time it takes to convert to POEK and then write. The conversion to POEK is automatic.

[jsiirola]

Archived on the master Performance Proposals Issue (#1430). Closing this performance proposal until active development has begun.