feat: make model metrics endpoints configurable

[nayihz]

fix: #16

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nayihz
Once this PR has been reviewed and has the lgtm label, please assign danehans for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	d86effa
🔍 Latest deploy log	https://app.netlify.com/projects/gateway-api-inference-extension/deploys/6854b9cdb5b6cb000878ff17
😎 Deploy Preview	https://deploy-preview-1000--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[nirrozenbaum]

I have some doubts about adding additional fields to InferencePool.
there are alternative ways to achieve the same goal, like using command line args.
changes to CRDs should be discussed and get broad agreement using proposals.

cc @kfswain @ahg-g @robscott @danehans @elevran

/hold for others to comment.

[nayihz]

@nirrozenbaum , thanks for you advise. I think you are right.
Command line args or environment variable, which way do you prefer?

[nirrozenbaum]

@nirrozenbaum , thanks for you advise. I think you are right. Command line args or environment variable, which way do you prefer?

@nayihz I would start with command-line args with default values (the existing ones).
we can always iterate if needed.

[elevran]

@nayihz @nirrozenbaum this introduces a fixed endpoint for all model servers in the pool.
Would it make sense to use the Prometheus-formatted annotations as the source of truth when present and fallback to the configuration when missing?
For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"

[nirrozenbaum]

@nayihz @nirrozenbaum this introduces a fixed endpoint for all model servers in the pool. Would it make sense to use the Prometheus-formatted annotations as the source of truth when present and fallback to the configuration when missing? For example:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080”

@elevran we already have a fixed endpoint, so this PR is not introducing it :). the intention was to make that endpoint configurable.
but your suggestion makes sense as an improvement.

[nirrozenbaum]

/unhold

[nirrozenbaum]

metrics port default value is 0?

[nayihz]

default metrics port should be equal to pool.Spec.TargetPortNumberif user did not specify a port number.

[ahg-g]

Can you please explain that it defaults to the target pool number on pool?

btw, perhaps those two parameters are candidates to be added to the InferencePool API, but we should leave this for later.

[nirrozenbaum]

I wouldn't add them to the InfPool API cause it implies that all pods in the pool are necessarily exposing metrics on the same path + port.
while that's what is done today in the reference implementation, the current API doesn't restrict that.
a different implementation may have each pod specify it's metrics path and port using Prometheus format on the pod as annotations and the EPP may store this info when reconciling the pod and use it for scraping.

leaving this outside of the CRD keeps this more flexible.

[ahg-g]

I mean having it as a flag is just as restrictive as having it on the pool, in fact, if we ever make epp multi-pool, then it is more restrictive, but agree that this is certainly not very important to get into now.

[kfswain]

implies that all pods in the pool are necessarily exposing metrics on the same path + port.

Similar assumptions are often made. We make the assumption that the serving port is the same across the pool also.

[nayihz]

Can you please explain that it defaults to the target pool number on pool?

Added some explanations to the usage string of -modelServerMetricsPort flag.

[nirrozenbaum]

seems to me that GetMetricEndpoint and FetchMetrics calls always come together (one must calculate the url before calling FetchMetrics).
does it make sense to make GetMetricEndpoint function private, remove the url arg from the FetchMetrics function, and get the url inside FetchMetrics (for example in the first line of the function)?
(and then GetMetricEndpoint function can also be removed from the interface)

[nayihz]

Done