Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Routing table is not synchronized #744

Closed
Mickeyrourkeske opened this issue Jul 26, 2023 · 5 comments
Closed

Routing table is not synchronized #744

Mickeyrourkeske opened this issue Jul 26, 2023 · 5 comments
Labels
bug Something isn't working

Comments

@Mickeyrourkeske
Copy link

Mickeyrourkeske commented Jul 26, 2023 •
edited
Loading

Report

Installed keda-core and the http-add-on and created a scaled object.
The deployment is scaled to 0, but never scaled up.

After looking around I found an error in the keda-add-ons-http-interceptor

"msg":"health check function failed","error":"table has not synced","errorVerbose":"table has not synced...

Expected Behavior

The deployment get scaled to 1

Actual Behavior

The deployment is never scaled to 1

Steps to Reproduce the Problem

Logs from KEDA HTTP operator

{"level":"info","ts":1690376567.5835505,"caller":"interceptor/main.go:52","msg":"starting interceptor","timeoutConfig":{"Connect":500000000,"KeepAlive":1000000000,"ResponseHeader":500000000,"DeploymentReplicas":20000000000,"ForceHTTP2":false,"MaxIdleConns":100,"IdleConnTimeout":90000000000,"TLSHandshakeTimeout":10000000000,"ExpectContinueTimeout":1000000000},"servingConfig":{"CurrentNamespace":"develop","WatchNamespace":"","ProxyPort":8080,"AdminPort":9090,"ConfigMapCacheRsyncPeriod":3600000000000,"DeploymentCachePollIntervalMS":250}}W0726 13:02:47.584099       1 shared_informer.go:586] resyncPeriod 250ms is too small. Changing it to the minimum allowed value of 1s{"level":"info","ts":1690376567.5844886,"caller":"interceptor/main.go:95","msg":"Interceptor starting"}
{"level":"info","ts":1690376567.5845597,"caller":"build/version.go:26","msg":"Interceptor Version: 0.5.0"}
{"level":"info","ts":1690376567.5846007,"caller":"build/version.go:27","msg":"Interceptor Commit: ce372282d11a28a409086f1493ca1be8e9256ad9"}
{"level":"info","ts":1690376567.5846298,"caller":"build/version.go:28","msg":"Go Version: go1.20.4"}
{"level":"info","ts":1690376567.5846615,"caller":"build/version.go:29","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1690376567.5849004,"caller":"interceptor/main.go:121","msg":"starting the admin server","port":9090}{"level":"info","ts":1690376567.5849507,"caller":"interceptor/main.go:140","msg":"starting the proxy server","port":8080}
{"level":"info","ts":1690376567.5850782,"logger":"runAdminServer.pkg.queue.AddCountsRoute","caller":"queue/queue_rpc.go:17","msg":"adding queue counts route","path":"/queue"}
{"level":"info","ts":1690376567.585136,"logger":"runAdminServer","caller":"interceptor/main.go:181","msg":"admin server starting","address":"0.0.0.0:9090"}
{"level":"error","ts":1690376567.5851088,"logger":"Probe","caller":"handler/probe.go:63","msg":"health check function failed","error":"table has not synced","errorVerbose":"table has not synced\ngithub.com/kedacore/http-add-on/pkg/routing.init\n\tgithub.com/kedacore/http-add-on/pkg/routing/table.go:25\nruntime.doInit\n\truntime/proc.go:6506\nruntime.doInit\n\truntime/proc.go:6483\nruntime.doInit\n\truntime/proc.go:6483\nruntime.main\n\truntime/proc.go:233\nruntime.goexit\n\truntime/asm_amd64.s:1598","stacktrace":"github.com/kedacore/http-add-on/interceptor/handler.(*Probe).check\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:63\ngithub.com/kedacore/http-add-on/interceptor/handler.(*Probe).Start\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:44"}
{"level":"info","ts":1690376567.5850503,"caller":"interceptor/main.go:226","msg":"proxy server starting","address":"0.0.0.0:8080"}
{"level":"error","ts":1690376568.5852768,"logger":"Probe","caller":"handler/probe.go:63","msg":"health check function failed","error":"table has not synced","errorVerbose":"table has not synced\ngithub.com/kedacore/http-add-on/pkg/routing.init\n\tgithub.com/kedacore/http-add-on/pkg/routing/table.go:25\nruntime.doInit\n\truntime/proc.go:6506\nruntime.doInit\n\truntime/proc.go:6483\nruntime.doInit\n\truntime/proc.go:6483\nruntime.main\n\truntime/proc.go:233\nruntime.goexit\n\truntime/asm_amd64.s:1598","stacktrace":"github.com/kedacore/http-add-on/interceptor/handler.(*Probe).check\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:63\ngithub.com/kedacore/http-add-on/interceptor/handler.(*Probe).Start\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:44"}

HTTP Add-on Version

0.5.0

Kubernetes Version

1.26

Platform

Other

Anything else?

No response
@Mickeyrourkeske Mickeyrourkeske added the bug Something isn't working label Jul 26, 2023
@Mickeyrourkeske Mickeyrourkeske changed the title Roting table is not synche Roting table is not synched Jul 26, 2023
@tomkerkhove
Copy link
Member

@t0rr3sp3dr0 can you please have a look please?

@tomkerkhove tomkerkhove changed the title Roting table is not synched Routing table is not synched Aug 10, 2023
@tomkerkhove tomkerkhove changed the title Routing table is not synched Routing table is not synced Aug 10, 2023
@tomkerkhove tomkerkhove changed the title Routing table is not synced Routing table is not synchronized Aug 10, 2023
@t0rr3sp3dr0
Copy link
Contributor

Hey @Mickeyrourkeske, can you share your HTTPScaledObject definition and the full logs of the operator, scaler, and interceptor? Can you send me the output of kubectl get all -n keda as well?

The log message you are seeing is normal and doesn't indicate a problem. When the interceptor starts, it takes a while for the routing table to be synced. The first few readiness probes fail, logging the error that caused the failures.

If there is a problem and the routing table never gets synced, the pod would never become ready and wouldn't receive any traffic. Otherwise, the pod becomes ready no more error logs should be seen from that point onwards.

@Mickeyrourkeske
Copy link
Author

Hey,

thanks for your reply. I think you are right. There seems to be some other problem. I don't know how I missed the errors last time.

HTTPScaledObject

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  finalizers:
    - httpscaledobject.http.keda.sh
  generation: 5
  name: scale-test
  namespace: develop
  resourceVersion: '33012935558'
  uid: 49b4bd2c-bae8-4233-82e3-41a508fdd2d7
spec:
  hosts:
    - foo.bar
  pathPrefixes:
    - /test
  replicas:
    max: 1
    min: 0
  scaleTargetRef:
    deployment: scale-test
    port: 8080
    service: scale-test
  targetPendingRequests: 100

keda-admission-webhooks

2023-08-11T11:51:37Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-08-11T11:51:37Z    INFO    setup   Starting admission webhooks
2023-08-11T11:51:37Z    INFO    setup   KEDA Version: 2.11.2
2023-08-11T11:51:37Z    INFO    setup   Git Commit: 517ed30cc311b0b81e39112cff0fa8e3251aed69
2023-08-11T11:51:37Z    INFO    setup   Go Version: go1.20.5
2023-08-11T11:51:37Z    INFO    setup   Go OS/Arch: linux/amd64
2023-08-11T11:51:37Z    INFO    setup   Running on Kubernetes 1.26      {"version": "v1.26.5"}
2023-08-11T11:51:37Z    INFO    controller-runtime.builder      skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called       {"GVK": "keda.sh/v1alpha1, Kind=ScaledObject"}
2023-08-11T11:51:37Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "keda.sh/v1alpha1, Kind=ScaledObject", "path": "/validate-keda-sh-v1alpha1-scaledobject"}
2023-08-11T11:51:37Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-keda-sh-v1alpha1-scaledobject"}
2023-08-11T11:51:37Z    INFO    controller-runtime.webhook.webhooks     Starting webhook server
2023-08-11T11:51:37Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-08-11T11:51:37Z    INFO    starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-08-11T11:51:37Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate
2023-08-11T11:51:37Z    INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2023-08-11T11:51:37Z    INFO    controller-runtime.certwatcher  Starting certificate watcher

keda-operator

2023-08-11T13:11:54Z    INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-08-11T13:11:54Z    INFO    setup   Starting manager
2023-08-11T13:11:54Z    INFO    setup   KEDA Version: 2.11.2
2023-08-11T13:11:54Z    INFO    setup   Git Commit: 517ed30cc311b0b81e39112cff0fa8e3251aed69
2023-08-11T13:11:54Z    INFO    setup   Go Version: go1.20.5
2023-08-11T13:11:54Z    INFO    setup   Go OS/Arch: linux/amd64
2023-08-11T13:11:54Z    INFO    setup   Running on Kubernetes 1.26      {"version": "v1.26.5"}
2023-08-11T13:11:54Z    INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-08-11T13:11:54Z    INFO    starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
I0811 13:11:54.831818       1 leaderelection.go:245] attempting to acquire leader lease keda/operator.keda.sh...
I0811 13:12:10.749425       1 leaderelection.go:255] successfully acquired lease keda/operator.keda.sh
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v1alpha1.ScaledObject"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "source": "kind source: *v2.HorizontalPodAutoscaler"}
2023-08-11T13:12:10Z    INFO    cert-rotation   starting cert rotator controller
2023-08-11T13:12:10Z    INFO    Starting Controller     {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "source": "kind source: *v1alpha1.ClusterTriggerAuthentication"}
2023-08-11T13:12:10Z    INFO    Starting Controller     {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "source": "kind source: *v1alpha1.TriggerAuthentication"}
2023-08-11T13:12:10Z    INFO    Starting Controller     {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2023-08-11T13:12:10Z    INFO    Starting Controller     {"controller": "cert-rotator"}
2023-08-11T13:12:10Z    INFO    Starting EventSource    {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "source": "kind source: *v1alpha1.ScaledJob"}
2023-08-11T13:12:10Z    INFO    Starting Controller     {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob"}
2023-08-11T13:12:10Z    INFO    cert-rotation   no cert refresh needed
2023-08-11T13:12:10Z    INFO    cert-rotation   certs are ready in /certs
2023-08-11T13:12:10Z    INFO    Starting workers        {"controller": "cert-rotator", "worker count": 1}
2023-08-11T13:12:10Z    INFO    cert-rotation   Ensuring CA cert        {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2023-08-11T13:12:10Z    INFO    cert-rotation   Ensuring CA cert        {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2023-08-11T13:12:11Z    INFO    cert-rotation   Ensuring CA cert        {"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2023-08-11T13:12:11Z    INFO    cert-rotation   Ensuring CA cert        {"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2023-08-11T13:12:11Z    INFO    Starting workers        {"controller": "scaledjob", "controllerGroup": "keda.sh", "controllerKind": "ScaledJob", "worker count": 1}
2023-08-11T13:12:11Z    INFO    Starting workers        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "worker count": 5}
2023-08-11T13:12:11Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scale-test","namespace":"develop"}, "namespace": "develop", "name": "scale-test", "reconcileID": "0e8b2d70-302a-432c-8347-f87b0ee37a02"}
2023-08-11T13:12:11Z    INFO    Reconciling ScaledObject        {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"keda"}, "namespace": "keda", "name": "keda-add-ons-http-interceptor", "reconcileID": "6580ee06-5df6-43b6-b6e3-2fb7466f7e5e"}
2023-08-11T13:12:11Z    INFO    Starting workers        {"controller": "clustertriggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "ClusterTriggerAuthentication", "worker count": 1}
2023-08-11T13:12:11Z    INFO    Starting workers        {"controller": "triggerauthentication", "controllerGroup": "keda.sh", "controllerKind": "TriggerAuthentication", "worker count": 1}
2023-08-11T13:12:11Z    ERROR   external_scaler error   {"type": "ScaledObject", "namespace": "keda", "name": "keda-add-ons-http-interceptor", "error": "rpc error: code = Unknown desc = httpscaledobject.http.keda.sh \"keda-add-ons-http-interceptor\" not found"}
github.com/kedacore/keda/v2/pkg/scalers.(*externalScaler).GetMetricSpecForScaling
        /workspace/pkg/scalers/external_scaler.go:172
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricSpecForScaling
        /workspace/pkg/scaling/cache/scalers_cache.go:90
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).getScaledObjectMetricSpecs
        /workspace/controllers/keda/hpa.go:215
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).newHPAForScaledObject
        /workspace/controllers/keda/hpa.go:75
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).updateHPAIfNeeded
        /workspace/controllers/keda/hpa.go:153
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).ensureHPAForScaledObjectExists
        /workspace/controllers/keda/scaledobject_controller.go:417
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).reconcileScaledObject
        /workspace/controllers/keda/scaledobject_controller.go:254
github.com/kedacore/keda/v2/controllers/keda.(*ScaledObjectReconciler).Reconcile
        /workspace/controllers/keda/scaledobject_controller.go:177
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226
2023-08-11T13:12:11Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"scale-test","namespace":"develop"}, "namespace": "develop", "name": "scale-test", "reconcileID": "0e8b2d70-302a-432c-8347-f87b0ee37a02"}
2023-08-11T13:12:11Z    INFO    Updated HPA according to ScaledObject   {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"keda"}, "namespace": "keda", "name": "keda-add-ons-http-interceptor", "reconcileID": "6580ee06-5df6-43b6-b6e3-2fb7466f7e5e", "HPA.Namespace": "keda", "HPA.Name": "keda-hpa-keda-add-ons-http-interceptor"}
2023-08-11T13:12:11Z    INFO    Initializing Scaling logic according to ScaledObject Specification      {"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"keda-add-ons-http-interceptor","namespace":"keda"}, "namespace": "keda", "name": "keda-add-ons-http-interceptor", "reconcileID": "6580ee06-5df6-43b6-b6e3-2fb7466f7e5e"}
2023-08-11T13:12:11Z    ERROR   external_scaler error   {"type": "ScaledObject", "namespace": "keda", "name": "keda-add-ons-http-interceptor", "error": "rpc error: code = Unknown desc = httpscaledobject.http.keda.sh \"keda-add-ons-http-interceptor\" not found"}
github.com/kedacore/keda/v2/pkg/scalers.(*externalScaler).GetMetricSpecForScaling
        /workspace/pkg/scalers/external_scaler.go:172
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricSpecForScalingForScaler
        /workspace/pkg/scaling/cache/scalers_cache.go:104
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledObjectState
        /workspace/pkg/scaling/scale_handler.go:552
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:236
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:175

keda-operator-metrics-apiserver

I0811 11:51:35.111325       1 welcome.go:34] keda_metrics_adapter "msg"="Starting metrics server" 
I0811 11:51:35.111503       1 welcome.go:35] keda_metrics_adapter "msg"="KEDA Version: 2.11.2" 
I0811 11:51:35.111567       1 welcome.go:36] keda_metrics_adapter "msg"="Git Commit: 517ed30cc311b0b81e39112cff0fa8e3251aed69" 
I0811 11:51:35.111608       1 welcome.go:37] keda_metrics_adapter "msg"="Go Version: go1.20.5" 
I0811 11:51:35.111635       1 welcome.go:38] keda_metrics_adapter "msg"="Go OS/Arch: linux/amd64" 
I0811 11:51:35.111696       1 welcome.go:39] keda_metrics_adapter "msg"="Running on Kubernetes 1.26" "version"={"major":"1","minor":"26","gitVersion":"v1.26.5","gitCommit":"890a139214b4de1f01543d15003b5bda71aae9c7","gitTreeState":"clean","buildDate":"2023-05-17T14:08:49Z","goVersion":"go1.19.9","compiler":"gc","platform":"linux/amd64"}
I0811 11:51:35.112511       1 listener.go:44] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8080"
I0811 11:51:35.112802       1 main.go:120] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.keda.svc.cluster.local:9666"
I0811 11:51:35.121303       1 provider.go:62] keda_metrics_adapter/provider "msg"="starting" 
I0811 11:51:35.121333       1 main.go:211] keda_metrics_adapter "msg"="starting adapter..." 
I0811 11:51:35.124468       1 server.go:50] keda_metrics_adapter "msg"="starting server" "addr"={"IP":"::","Port":8080,"Zone":""} "kind"="metrics" "path"="/metrics"
I0811 11:51:35.124562       1 client.go:88] keda_metrics_adapter/provider "msg"="Waiting for establishing a gRPC connection to KEDA Metrics Server" 
W0811 11:51:35.134187       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
I0811 11:51:35.452044       1 secure_serving.go:210] Serving securely on [::]:6443
I0811 11:51:35.452270       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0811 11:51:35.452330       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0811 11:51:35.452417       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/certs/tls.crt::/certs/tls.key"
I0811 11:51:35.452615       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0811 11:51:35.452846       1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/certs/ca.crt"
I0811 11:51:35.453091       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0811 11:51:35.453170       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0811 11:51:35.554830       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0811 11:51:35.554868       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
W0811 11:51:36.137972       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
W0811 11:51:37.680387       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
W0811 11:51:40.645898       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
W0811 11:51:45.074143       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
W0811 11:51:50.858513       1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.233.21.97:9666: connect: connection refused"
I0811 11:52:01.156743       1 provider.go:70] keda_metrics_adapter/provider "msg"="Connection to KEDA Metrics Service gRPC server has been successfully established" "server"="keda-operator.keda.svc.cluster.local:9666"

keda-add-ons-http-controller-manager

I0811 11:54:30.284535       1 main.go:186] Valid token audiences: 
I0811 11:54:30.284617       1 main.go:316] Generating self signed cert as no cert is provided
I0811 11:54:31.267723       1 main.go:366] Starting TCP socket on 0.0.0.0:8443
I0811 11:54:31.268032       1 main.go:373] Listening securely on 0.0.0.0:8443

keda-add-ons-http-external-scaler

{"level":"info","ts":1691759905.2902954,"caller":"build/version.go:26","msg":"Scaler Version: 0.5.0"}
{"level":"info","ts":1691759905.2907681,"caller":"build/version.go:27","msg":"Scaler Commit: ce372282d11a28a409086f1493ca1be8e9256ad9"}
{"level":"info","ts":1691759905.2909207,"caller":"build/version.go:28","msg":"Go Version: go1.20.4"}
{"level":"info","ts":1691759905.2910154,"caller":"build/version.go:29","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1691759905.2905178,"caller":"scaler/main.go:148","msg":"starting grpc server","address":"0.0.0.0:9090"}
{"level":"error","ts":1691759905.9777708,"logger":"GetMetricSpec","caller":"scaler/handlers.go:117","msg":"unable to get HTTPScaledObject","name":"keda-add-ons-http-interceptor","namespace":"keda","error":"httpscaledobject.http.keda.sh \"keda-add-ons-http-interceptor\" not found","stacktrace":"main.(*impl).GetMetricSpec\n\tgithub.com/kedacore/http-add-on/scaler/handlers.go:117\ngithub.com/kedacore/keda/v2/pkg/scalers/externalscaler._ExternalScaler_GetMetricSpec_Handler\n\tgithub.com/kedacore/keda/[email protected]/pkg/scalers/externalscaler/externalscaler_grpc.pb.go:190\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/[email protected]/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/[email protected]/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/[email protected]/server.go:966"}
{"level":"error","ts":1691759906.9758782,"logger":"GetMetricSpec","caller":"scaler/handlers.go:117","msg":"unable to get HTTPScaledObject","name":"keda-add-ons-http-interceptor","namespace":"keda","error":"httpscaledobject.http.keda.sh \"keda-add-ons-http-interceptor\" not found","stacktrace":"main.(*impl).GetMetricSpec\n\tgithub.com/kedacore/http-add-on/scaler/handlers.go:117\ngithub.com/kedacore/keda/v2/pkg/scalers/externalscaler._ExternalScaler_GetMetricSpec_Handler\n\tgithub.com/kedacore/keda/[email protected]/pkg/scalers/externalscaler/externalscaler_grpc.pb.go:190\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/[email protected]/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/[email protected]/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golang.org/[email protected]/server.go:966"}

keda-add-ons-http-interceptor

{"level":"info","ts":1691754877.5041065,"caller":"interceptor/main.go:52","msg":"starting interceptor","timeoutConfig":{"Connect":500000000,"KeepAlive":1000000000,"ResponseHeader":500000000,"DeploymentReplicas":20000000000,"ForceHTTP2":false,"MaxIdleConns":100,"IdleConnTimeout":90000000000,"TLSHandshakeTimeout":10000000000,"ExpectContinueTimeout":1000000000},"servingConfig":{"CurrentNamespace":"keda","WatchNamespace":"","ProxyPort":8080,"AdminPort":9090,"ConfigMapCacheRsyncPeriod":3600000000000,"DeploymentCachePollIntervalMS":250}}
W0811 11:54:37.504740       1 shared_informer.go:586] resyncPeriod 250ms is too small. Changing it to the minimum allowed value of 1s
{"level":"info","ts":1691754877.5048883,"caller":"interceptor/main.go:95","msg":"Interceptor starting"}
{"level":"info","ts":1691754877.50494,"caller":"build/version.go:26","msg":"Interceptor Version: 0.5.0"}
{"level":"info","ts":1691754877.5049608,"caller":"build/version.go:27","msg":"Interceptor Commit: ce372282d11a28a409086f1493ca1be8e9256ad9"}
{"level":"info","ts":1691754877.5049853,"caller":"build/version.go:28","msg":"Go Version: go1.20.4"}
{"level":"info","ts":1691754877.5050085,"caller":"build/version.go:29","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1691754877.505057,"caller":"interceptor/main.go:140","msg":"starting the proxy server","port":8080}
{"level":"info","ts":1691754877.5051093,"caller":"interceptor/main.go:226","msg":"proxy server starting","address":"0.0.0.0:8080"}
{"level":"info","ts":1691754877.5052373,"caller":"interceptor/main.go:121","msg":"starting the admin server","port":9090}
{"level":"info","ts":1691754877.5059428,"logger":"runAdminServer.pkg.queue.AddCountsRoute","caller":"queue/queue_rpc.go:17","msg":"adding queue counts route","path":"/queue"}
{"level":"info","ts":1691754877.5061834,"logger":"runAdminServer","caller":"interceptor/main.go:181","msg":"admin server starting","address":"0.0.0.0:9090"}
{"level":"error","ts":1691754877.5055249,"logger":"Probe","caller":"handler/probe.go:63","msg":"health check function failed","error":"table has not synced","errorVerbose":"table has not synced\ngithub.com/kedacore/http-add-on/pkg/routing.init\n\tgithub.com/kedacore/http-add-on/pkg/routing/table.go:25\nruntime.doInit\n\truntime/proc.go:6506\nruntime.doInit\n\truntime/proc.go:6483\nruntime.doInit\n\truntime/proc.go:6483\nruntime.main\n\truntime/proc.go:233\nruntime.goexit\n\truntime/asm_amd64.s:1598","stacktrace":"github.com/kedacore/http-add-on/interceptor/handler.(*Probe).check\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:63\ngithub.com/kedacore/http-add-on/interceptor/handler.(*Probe).Start\n\tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:44"}
fra@pc-006 ~/gsoft/git/com.gs.vd.cluster (main*) $

kubectl get all -n keda

NAME                                                        READY   STATUS    RESTARTS   AGE
pod/keda-add-ons-http-controller-manager-7d89b48bff-5df8f   2/2     Running   0          86m
pod/keda-add-ons-http-external-scaler-77d7f974ff-lwzlk      1/1     Running   0          2m14s
pod/keda-add-ons-http-interceptor-7bfbd664c5-4666f          1/1     Running   0          86m
pod/keda-add-ons-http-interceptor-7bfbd664c5-lvhsb          1/1     Running   0          86m
pod/keda-add-ons-http-interceptor-7bfbd664c5-q4vlx          1/1     Running   0          86m
pod/keda-admission-webhooks-854ccc65c4-psg8v                1/1     Running   0          89m
pod/keda-operator-6594fb5794-q72vk                          1/1     Running   0          8m46s
pod/keda-operator-metrics-apiserver-6c4ccc975d-rls64        1/1     Running   0          89m

NAME                                                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/keda-add-ons-http-controller-manager-metrics-service   ClusterIP   10.233.53.139   <none>        8443/TCP            86m
service/keda-add-ons-http-external-scaler                      ClusterIP   10.233.53.243   <none>        9090/TCP,9091/TCP   86m
service/keda-add-ons-http-interceptor-admin                    ClusterIP   10.233.57.7     <none>        9090/TCP            86m
service/keda-add-ons-http-interceptor-proxy                    ClusterIP   10.233.63.148   <none>        8080/TCP            86m
service/keda-admission-webhooks                                ClusterIP   10.233.22.111   <none>        443/TCP             89m
service/keda-operator                                          ClusterIP   10.233.21.97    <none>        9666/TCP            89m
service/keda-operator-metrics-apiserver                        ClusterIP   10.233.31.140   <none>        443/TCP,8080/TCP    89m

NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/keda-add-ons-http-controller-manager   1/1     1            1           86m
deployment.apps/keda-add-ons-http-external-scaler      1/1     1            1           86m
deployment.apps/keda-add-ons-http-interceptor          3/3     3            3           86m
deployment.apps/keda-admission-webhooks                1/1     1            1           89m
deployment.apps/keda-operator                          1/1     1            1           8m47s
deployment.apps/keda-operator-metrics-apiserver        1/1     1            1           89m

NAME                                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/keda-add-ons-http-controller-manager-7d89b48bff   1         1         1       86m
replicaset.apps/keda-add-ons-http-external-scaler-5d9cdf7795      0         0         0       86m
replicaset.apps/keda-add-ons-http-external-scaler-77d7f974ff      1         1         1       2m15s
replicaset.apps/keda-add-ons-http-interceptor-7bfbd664c5          3         3         3       86m
replicaset.apps/keda-admission-webhooks-854ccc65c4                1         1         1       89m
replicaset.apps/keda-operator-6594fb5794                          1         1         1       8m47s
replicaset.apps/keda-operator-metrics-apiserver-6c4ccc975d        1         1         1       89m

NAME                                                                         REFERENCE                                  TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/keda-hpa-keda-add-ons-http-interceptor   Deployment/keda-add-ons-http-interceptor   1%/80%    3         50        3          86m

Thanks for looking into this

@t0rr3sp3dr0
Copy link
Contributor

The errors you are getting on the operator and scaler logs are expected because of #718, but there should be no impact on the autoscaling behaviour of the add-on.

What is the response, including headers, you get when sending the request to the interceptor using the host and path defined on the HTTPScaledObject? Can you send me the command you are using?

@Mickeyrourkeske
Copy link
Author

After trying out the example which worked, I found out that my ingress was wrong. It was still pointing to my service and not to the keda-add-ons-http-interceptor-proxy.

Also it is worth noticing that the ingress needs to be in the same namespace as the keda-add-ons-http-interceptor-proxy

Tank you very much for your help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Roadmap - KEDA HTTP Add-On
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Mickeyrourkeske @tomkerkhove @t0rr3sp3dr0