opengrok-sync don't index some projects

[shusterboris]

Hello, all!
I installed OpenGrok into container, it works. My next task is starting re-indexing of project on-demand, but not by schedule. I created sync.conf. When I start it for several similar projects, for some projects it works, for some shows error messages, like this.

opengrok-sync -c /opengrok/etc/sync.conf -P p90_integ
command 'opengrok-reindex-project -J=-Xmx22G -J=-Xms22G -J=-server -j /opt/java/openjdk/bin/java --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties -p %PROJ% -d /opengrok/logs/p90_integ -P p90_integ -U http://localhost:8080 -- --progress -R /opengrok/etc/readonly_configuration.xml -G -m 256 -H p90_integ -U http://localhost:8080' for project p90_integ failed with code 1, breaking
processing of project 'p90_integ' failed
  failed commands:
    'opengrok-reindex-project -J=-Xmx22G -J=-Xms22G -J=-server -j /opt/java/openjdk/bin/java --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties -p %PROJ% -d /opengrok/logs/p90_integ -P p90_integ -U http://localhost:8080 -- --progress -R /opengrok/etc/readonly_configuration.xml -G -m 256 -H p90_integ -U http://localhost:8080': 1
      15:41:55 INFO: Reading configuration from /tmp/tmpyi_wbw17
      15:41:56 INFO: Reading configuration from /opengrok/etc/readonly_configuration.xml
      15:41:56 INFO: Indexer options: [-R, /tmp/tmpyi_wbw17, --progress, -R, /opengrok/etc/readonly_configuration.xml, -G, -m, 256, -H, p90_integ, -U, http://localhost:8080]
      The path /-U does not correspond to a project
      The path /http:/localhost:8080 does not correspond to a project
      The path /p90_integ does not correspond to a project
      None of the paths were added, exiting
      Indexer command for project p90_integ failed (return code 1)

My sync.conf is...

commands:
- command:
  call:
    uri: 'http://localhost:8080/api/v1/messages'
    method: POST
    data:
      messageLevel: warning
      duration: PT1H
      tags: ['%PROJECT%']
      text: reindex in progress
- command: [opengrok-reindex-project,
    -J=-Xmx22G,-J=-Xms22G,-J=-server,
    -j,/opt/java/openjdk/bin/java,
    --jar,/opengrok/lib/opengrok.jar,
    -t,/opengrok/etc/logging.properties,-p,'%PROJ%',-d,'/opengrok/logs/%PROJECT%',
    -P,'%PROJECT%',-U,'http://localhost:8080',
    --,
    --progress,
    -R, /opengrok/etc/readonly_configuration.xml,
    -G,-m,'256',-H,'%PROJECT%',-U,'http://localhost:8080']
- command:
  call:
    uri: 'http://localhost:8080/api/v1/messages?tag=%PROJECT%'
    method: DELETE
- command: [python3,/opengrok/scripts/wva_finally.py,'%PROJECT%',success]
cleanup:
- command:
  call:
    uri: 'http://localhost:8080/api/v1/messages?tag=%PROJECT%'
    method: DELETE
- command: [python3,/opengrok/scripts/wva_finally.py,'%PROJECT%',failure]

[vladak]

What version are you running ? The above will not pass the opengrok-sync configuration parsing in the recent versions:

Invalid configuration: command value not a dictionary: ['opengrok-reindex-project', '-J=-Xmx22G', '-J=-Xms22G', '-J=-server', '-j', '/opt/java/openjdk/bin/java', '--jar', '/opengrok/lib/opengrok.jar', '-t', '/opengrok/etc/logging.properties', '-p', '%PROJ%', '-d', '/opengrok/logs/%PROJECT%', '-P', '%PROJECT%', '-U', 'http://localhost:8080', '--', '--progress', '-R', '/opengrok/etc/readonly_configuration.xml', '-G', '-m', '256', '-H', '%PROJECT%', '-U', 'http://localhost:8080']

Also, the -H indexer option does not take an argument. That's why the rest of the arguments is treated as paths.

[shusterboris]

Dear Vladimir! Thank you for the rapid response. I'm using version 1.8.4. The main reason for THIS issue was the use of -P and -p command line options at the same time. I was happy, but it did not last long enough :-). Now I have another error(you can see log below): "http://localhost:8080/" is not reachable and the -U option was specified. But curl http://localhost:8080/api/v1/configuration returns configuration without any problems. The good thing is that after your hint I can start opengrok-reindex-project. It is still working, but, at least, it's working and maybe I can create a workaround... But if you have any idea about the problem, it would be good. The error message: opengrok-sync -c /opengrok/etc/sync.conf -P p90_integ command 'opengrok-reindex-project -J=-Xmx22G -J=-Xms15G -J=-server -j /opt/java/openjdk/bin/java --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties -d /opengrok/logs/p90_integ -P p90_integ -U http://localhost:8080/ -- --progress -R /opengrok/etc/readonly_configuration.xml -G -m 256 -U http://localhost:8080/' for project p90_integ failed with code 1, breaking processing of project 'p90_integ' failed failed commands: 'opengrok-reindex-project -J=-Xmx22G -J=-Xms15G -J=-server -j /opt/java/openjdk/bin/java --jar /opengrok/lib/opengrok.jar -t /opengrok/etc/logging.properties -d /opengrok/logs/p90_integ -P p90_integ -U http://localhost:8080/ -- --progress -R /opengrok/etc/readonly_configuration.xml -G -m 256 -U http://localhost:8080/': 1 Oct 23, 2023 2:26:11 PM org.opengrok.indexer.configuration.Configuration read INFO: Reading configuration from /tmp/tmpqo7z02hv Oct 23, 2023 2:26:12 PM org.opengrok.indexer.configuration.Configuration read INFO: Reading configuration from /opengrok/etc/readonly_configuration.xml Oct 23, 2023 2:26:12 PM org.opengrok.indexer.index.Indexer parseOptions INFO: Indexer options: [-R, /tmp/tmpqo7z02hv, --progress, -R, /opengrok/etc/readonly_configuration.xml, -G, -m, 256, -U, http://localhost:8080/] Oct 23, 2023 2:26:12 PM org.opengrok.indexer.util.HostUtil isWebAppReachable SEVERE: could not connect to http://localhost:8080/ jakarta.ws.rs.ProcessingException: java.net.SocketTimeoutException: Read timed out at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:270) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313) at org.opengrok.indexer.util.HostUtil.isWebAppReachable(HostUtil.java:82) at org.opengrok.indexer.util.HostUtil.isReachable(HostUtil.java:114) at org.opengrok.indexer.index.Indexer.main(Indexer.java:175) Respectfully, Boris пн, 23 окт. 2023 г. в 15:13, Vladimir Kotal ***@***.***>:

…

What version are you running ? The above will not pass the opengrok-sync configuration parsing in the recent versions: Invalid configuration: command value not a dictionary: ['opengrok-reindex-project', '-J=-Xmx22G', '-J=-Xms22G', '-J=-server', '-j', '/opt/java/openjdk/bin/java', '--jar', '/opengrok/lib/opengrok.jar', '-t', '/opengrok/etc/logging.properties', '-p', '%PROJ%', '-d', '/opengrok/logs/%PROJECT%', '-P', '%PROJECT%', '-U', 'http://localhost:8080', '--', '--progress', '-R', '/opengrok/etc/readonly_configuration.xml', '-G', '-m', '256', '-H', '%PROJECT%', '-U', 'http://localhost:8080'] Also, the -H indexer option does not take an argument. That's why the rest of the arguments is treated as paths. — Reply to this email directly, view it on GitHub <#4449 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGDXRGNITJSTJSI2XPECN4LYAZNNJAVCNFSM6AAAAAA6K7TVXGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNJYGIZTC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[vladak]

I think you are bumping into #4228, fixed in https://github.com/oracle/opengrok/releases/tag/1.9.0

[shusterboris]

Hi, Vladimir. I admire the support. I think you are right. I think I'll try changing this in the source code (there are some breaking changes that I made as requested by my team...) Seems like it would be easier... But in any case, thank you пн, 23 окт. 2023 г. в 18:03, Vladimir Kotal ***@***.***>:

…

I think you are bumping into #4228 <#4228>, fixed in https://github.com/oracle/opengrok/releases/tag/1.9.0 — Reply to this email directly, view it on GitHub <#4449 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGDXRGL4KNYOYKAVNOEDMRTYA2BN3AVCNFSM6AAAAAA6K7TVXGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNRQGA3DA> . You are receiving this because you authored the thread.Message ID: ***@***.***>