[Epic] Agent dynamic attach

[andrewazores]

Currently, the only way to get the Cryostat Agent attached and running on a target application is with the -javaagent JVM flag. This means the end user must be able to modify their application launch setup to add this flag, and the user must get the Agent JAR into the application's filesystem somehow, ex. building it into the application image or supplying it in a mounted volume. The user must also decide before launching their application whether they would like to use the Cryostat Agent or not, and if they decide to add the Cryostat Agent later, their application must be restarted.

It should be possible to decouple the Cryostat Agent from these assumptions for the most part.

1. the user might obtain their application image from another team or a separate vendor and may not be able to modify the launcher setup to add a -javaagent flag, but would be able to spawn a process within the container or launch a sidecar container
2. the user might be able to supply the Agent as a mounted volume, but not built in to the application image (similar to above). Part of the story here could be an integration with the cryostat-operator that can bundle the Agent JAR and supply it as a mounted volume if the user simply adds some annotation to their application Deployment, for example.
3. the user might have a long-running "pet" (not cattle) application and wish to install the Cryostat Agent without taking the application down

If the Agent implements dynamic attach and provides some additional mechanisms to supply its required configurations then the major potential deployment blockers in the above scenarios can be alleviated.

[andrewazores]

As seen done by another project recently, the Agent JAR could be built into a simple JAR-carrier container image, ex:

FROM scratch
COPY ./target/cryostat-agent.jar /cryostat/

This would be an easy way for application images to consume the Agent JAR - rather than manually downloading the JAR to the build machine and copying it into the container build, they could use a multi-stage container build like:

FROM cryostat/agent-carrier:latest as cryostat-agent
RUN echo '' # no-op, can this be removed?

FROM base-image
COPY --from=cryostat-agent /cryostat/cryostat-agent.jar /app/deployment/lib
... copy application artifacts in

---

Alternatively, this JAR-carrier image can be used as an init container in the application's Deployment to achieve the same result dynamically, if there is a volume shared between the init container and the application container:

        initContainers:
        - name: download-agent
          image: agent-carrier:latest
          imagePullPolicy: Always
          command: [ 'cp', '/cryostat/cryostat-agent.jar','/etc/agent/ ]
          volumeMounts: # this must also be defined on the container that runs the host JVM which the Agent should attach to, so that the JAR is available on its filesystem as well
            - name: shared
              mountPath: /etc/agent/

[andrewazores]

https://github.com/open-telemetry/opentelemetry-operator/blob/16e11344ae27db6f2bfb0a8953d908827c7b9019/pkg/instrumentation/javaagent.go#L28

This is a good reference for the same/similar technique, which also includes the technique of modifying the target container's JAVA_TOOL_OPTIONS environment variable to include -javaagent to launch the agent via static attach. This would necessitate a target container restart for agent instrumentation to begin.

Discussion on dynamic attach mechanisms over here: cryostatio/cryostat-operator#784

[andrewazores]

Since the init container needs at least cp in order to copy the carried JAR into the shared volume, FROM scratch alone won't be quite enough. It could be a FROM scratch that also has a standalone cp binary copied into it, or else a very minimal base image with some coreutils in it instead:

FROM docker.io/library/busybox:latest
COPY ./target/cryostat-agent.jar /cryostat/