charts: Add headlamp-plugin sidecar for automated plugin management

[Faakhir30]

Implemented headlamp-plugin sidecar container for plugin management.

• charts: Added --watch-plugins-changes option to charts for the main headlamp container
• headlamp-plugin: Added watch option for config file changes in bulk install
• headlamp-plugin: Added cleanup for removal of plugins (in case the user removes a plugin from the config file)
• charts: Mount plugin.yml using a configMap-based volume
• charts: do a global @kinvolk/headlamp-plugin install and watch the config file

Testing

For docs, see https://github.com/kubernetes-sigs/headlamp/pull/2983/files#diff-6ba75da13a137e0dc291f25e5514187d41c825abf7cc74cee9e238cffbf7d539

I've tested by using my published npm package from my account https://www.npmjs.com/package/@faakhir/headlamp-plugin
and building the headlamp server image locally, and overriding it by minikube image load ghcr.io/headlamp-k8s/headlamp:v0.30.0.

sample plugin.yml to try:

plugins:
  - name: test-app-catalog
    source: https://artifacthub.io/packages/headlamp/test-123/appcatalog_headlamp_plugin
    version: 0.0.3
  - name: ai-plugin
    source: https://artifacthub.io/packages/headlamp/test-123/ai_plugin
    version: 0.0.2
  - name: prometheus
    source: https://artifacthub.io/packages/headlamp/test-123/prometheus_headlamp_plugin
    version: 0.0.3
    dependencies:
      - test-app-catalog

installOptions:
  parallel: true
  maxConcurrent: 2

NOTES:

• The latest v0.30.0 version of ghcr.io/headlamp-k8s/headlamp image does not include the --watch-plugins-changes flag. So would require a release for that image.
• Also, the headlamp-plugin package would require a release, though I could not see an NPM package release workflow. Does the headlamp do this manually from the terminal with npm publish?
• The sidecar container watch was having some delay sometimes on my minikube cluster during testing, not sure though if it was due to resource contention on my end only, because it is working fine when updating multiple times locally using bin/headlamp-plugin.js install -c plugin.yml -w --folderName /home/f/w/oss/headlamp/plugins/headlamp-plugin/out

[k8s-triage-robot]

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

[github-advanced-security]

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

[Faakhir30]

@knrt10 an initial review here along with other bulk support PR might be helpful.

I've currently brought the image size to 100MB, further reduction of image size would require creating a binary IMO, I did try that also with pkg, but pkg does not support node20 binary, and plugins CLI is not working with node 18. So, I might have to try out some other solutions for making a binary. Just needed a signal: would a binary be required, or 100MB is fine for a CLI tool to be run in a sidecar container?

[illume]

Can you please add to the PR description why this is needed?

How about using npm global install?

Perhaps using global install is a good idea for the Dockerfile?

Dockerfile:

ARG BASE_IMAGE=node:lts
FROM $BASE_IMAGE
ARG VERSION=latest
WORKDIR /usr/src/app
RUN npm install -g @kinvolk/headlamp-plugin@$VERSION
ENTRYPOINT ["headlamp-plugin"]

To run it defaulting to latest version:

docker build -t headlamp-k8s/headlamp-plugin .
docker run -it --rm headlamp-k8s/headlamp-plugin

Or with a version:

docker build --build-arg VERSION=0.11.4 -t headlamp-k8s/headlamp-plugin:0.11.4 .
docker run -it --rm headlamp-k8s/headlamp-plugin:0.11.4

Watch bash script

I think it should be for windows containers that the watch commands runs in JavaScript rather than using bash. Maybe the watch command could be added to headlamp-plugin?

[Faakhir30]

Perhaps using global install is a good idea for the Dockerfile?

Yes, i did thought of installing using npm package. I was creating seperate containerization and headlamp-plugin container release configuration based on the initial description of the issue..

ARG BASE_IMAGE=node:lts
FROM $BASE_IMAGE
ARG VERSION=latest
WORKDIR /usr/src/app
RUN npm install -g @kinvolk/headlamp-plugin@$VERSION
ENTRYPOINT ["headlamp-plugin"]

This would create dependency on the npm package. If this is fine approach, then I guess we wont need the release management and maintainance overhead for this container image seperatly.

I think it should be for windows containers that the watch commands runs in JavaScript rather than using bash. Maybe the watch command could be added to headlamp-plugin?

And yes, this would be indeed better, thanks for the feedback.

---

@illume @knrt10 please leave a reaction/comment to confirm the approach:

• add watch commands to headlamp-plugin itself
• no dockerfile, release management needed
• The containerization part is only relevant to the in-cluster deployment, so lets do the above mentioned Dockerfile as in-line dockerfile of a sidecar container for plugin-management.

[illume]

Sounds good to me.

[Faakhir30]

@illume @knrt10
installing from npm package is causing a bulky layer in the image, irrespective of global or local install, I've tried with several base images like node:lts-alpine, node:lts, alpine:3.18.
size of some bulky deps of the npm package on installing with RUN npm install --omit=dev --no-cache @kinvolk/headlamp-plugin@$VERSION:

/usr/src/app/node_modules # du -sh * | sort -h | tail
7.7M    elkjs
9.3M    es-abstract
9.8M    @esbuild
10.8M   prettier
18.2M   @headlamp-k8s
18.7M   typescript
31.4M   @storybook
55.5M   @iconify
67.0M   monaco-editor
135.6M  @mui
/usr/src/app/node_modules #

a non-global install, not much difference in size though:

f@f:~/w/oss/headlamp/plugins/headlamp-plugin$ docker history headlamp-plugin:latest 
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
8876b566b74b   2 days ago     CMD ["headlamp-plugin"]                         0B        buildkit.dockerfile.v0
<missing>      2 days ago     ENV PATH=/app/node_modules/.bin:/usr/local/s…   0B        buildkit.dockerfile.v0
<missing>      2 days ago     RUN |1 VERSION=latest /bin/sh -c npm install…   663MB     buildkit.dockerfile.v0
<missing>      2 days ago     WORKDIR /usr/src/app                            0B        buildkit.dockerfile.v0
<missing>      2 days ago     ARG VERSION=latest                              0B        buildkit.dockerfile.v0
<missing>      2 days ago     RUN /bin/sh -c apk add --no-cache nodejs npm…   59.4MB    buildkit.dockerfile.v0
<missing>      2 months ago   CMD ["/bin/sh"]                                 0B        buildkit.dockerfile.v0
<missing>      2 months ago   ADD alpine-minirootfs-3.18.12-x86_64.tar.gz …   7.36MB    buildkit.dockerfile.v0
f@f:~/w/oss/headlamp/plugins/headlamp-plugin$ 

The total image size with node:lts-alpine and npm install is getting around 700 MB.
Does image size matter this much for our sidecar container? If yes, I'd prefer the original approach as I reached up to 100MB image by trimming node_modules, multistage builds etc.
While there is not much optimization available to reduce this from 700MB with npm package install.

[illume]

The headlamp-plugin is large because it needs all the packages as frontend/ for testing.

Is it only plugin management commands that need to be run? We did have a plan to make plugin-management its own package #2197 . Because it is also used by the app. plugin management by itself should be a much smaller image.

I don’t know how much time is left for your project, and if this headlamp-k8s/plugin-management package could be done in time?

it could include a cli with only the plugin management things in there. A copy of the headlamp-plugin with other stuff removed (no test, format, lint, etc).

If you don’t have enough time to do the plugin-management package… Maybe call the container image headlamp-plugin-management, and just use headlamp-plugin package in there for now. This way the npm package used could be swapped out fairly easily later.

Then it won’t matter too much that the image is larger for now.

I hope I am being clear?

—-

I’m not sure if it’s worth considering moving most dependencies into devDependencies? Maybe it’s possible to move most there that are needed for testing/tsc/build/lint. Then it could be installed with npm ci (or omit dev) and commands like creat and the plugin management ones could be done. But then the container image could only be used for these commands. So after thinking this through, I think plugin-management should be its own package?

[Faakhir30]

Maybe call the container image headlamp-plugin-management, and just use headlamp-plugin package in there for now. This way the npm package used could be swapped out fairly easily later.

Alright, understood, I'll be doing the above approach; however, I might probably come back to implement this plugin-management package later if time permits, or even post LFX next month.

[Faakhir30]

@illume I've implemented the above discussed approach, tested by publishing package temporarily on my account and building image of headlamp server locally.

NOTES:
The latest v0.30.0 version of ghcr.io/headlamp-k8s/headlamp image does not include the --watch-plugins-changes flag. So would require a release for that image.
Also, the headlamp-plugin package would require a release, though I could not see an NPM package release workflow. Does the headlamp do this manually from the terminal with npm publish?

Updated PR description, please let me know about the release of docker image and npm package mentioned in the notes. Let me know if I need to separate out PR for headlamp-plugin changes for it.

[illume]

🎉🎈 thanks

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Faakhir30, illume

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [illume]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[illume]

@Faakhir30 There is now a @kinvolk/[email protected]

The release is delayed because of some certificate signing issues. I don't have any clarity on when the container image will be released again. But I'll let you know when I do.