Batch operations / Improve transactions

[fxprunayre]

Is your feature request related to a problem?

For batch operations on a selection, proxies may trigger a timeout after X sec (depending on proxy or web server configuration) API calls may be terminated but the processing continue. As discussed with @juanluisrp and @josegar74 , the database transaction handled by the servlet get the exception at the end of the processing phase and rollback. Database and index may be inconsistent in such cases.

User usually notice that by reindexing processed records and the initial XML from the database is reindexed and user notice that the changes were not applied.

To mimic that, configure an apache proxy with a timeout of 5 sec

        ProxyPass /geonetwork http://localhost:8080/geonetwork timeout=5
        ProxyPassReverse /geonetwork http://localhost:8080/geonetwork

In the short term, a workaround in such situation:

• process records one by one (or pages by pages) when the number of records is too high
• call process from the server to avoid the proxy

Describe the solution you'd like

First, to avoid inconsistencies, create a dedicated transaction (to not rely on the default servlet transaction manager) in order to commit the process results in the end. Eg.

core-geonetwork/services/src/main/java/org/fao/geonet/api/records/InspireValidationRunnable.java

Line 40 in 9504f94

TransactionManager.runInTransaction("inspire-validation", ApplicationContextHolder.get(),

. Apply this first to XSL and batch editing (https://github.com/geonetwork/core-geonetwork/blob/main/services/src/main/java/org/fao/geonet/api/processing/XslProcessApi.java#L327-L350) which are the main one affected by this issue.

After that change, user will not be notified of the termination of the processing. So we could improve the UI to follow the progress of the processing using the existing GET /processes/report operation.

Then we can check if it also applies to operation like changing ownership, sharing, ...

Describe alternatives you've considered

Then on the long run, we may:

• run those operations asynchronously https://spring.io/guides/gs/async-method/
• try to globally improve all batch operations and make more consistent reporting of the progress of processing (currently we have various approaches in harvesters, indexing, link analysis using Jolokia, processing, ...)