Issue getting Istio control plane stats and k8s/self obv stats to run at same time

[andrewgkew]

Hi

I have been banging my head against the wall for the last 2 days on this and think I found a possible issue but wanted to see I am not loosing the plot.

In isolation I have got the following working

1. k8s monitoring
2. self observability of OAP server
3. Istio control and data plane stats

Now what I am doing is putting it all together so that they are all running at the same time and everything mostly worked, expect for the control plane stats, they kept not displaying. So after hours of pulling my configuration apart I have located the culprit.

If I configure skywalking to monitor Istio with the following parameters I get CP and DP stats

  --set oap.env.SW_OTEL_RECEIVER=default \
  --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \
  --set oap.envoy.als.enabled=true

As soon as I add the OC rules then the control plane does not process

 --set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="oap"

So if I install skywalking with the following command, I get DP stats self obs of OAP but NO CP stats

helm upgrade -i "${SKYWALKING_RELEASE_NAME}" \
skywalking -n "${SKYWALKING_RELEASE_NAMESPACE}" \
  --set oap.storageType='h2' \
--set ui.image.tag=8.7.0 \
--set oap.image.tag=8.7.0-es7 \
--set oap.env.JAVA_OPTS='-Dmode=' \
--set elasticsearch.enabled=false \
  --set oap.env.SW_OTEL_RECEIVER=default \
  --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \
  --set oap.envoy.als.enabled=true \
--set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="oap"

NOTE: I am using h2 in the above command just to speed up the deployment and pull down for testing as well as ensure I am not getting confused with ES persistence thinking its working but its old data I am seeing.

Have I completely missed something here? Any reasons why I am seeing this behaviour?

Thanks

[wu-sheng]

--set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="oap"

This is a rule for self observability. Do you misuse the rule?

[andrewgkew]

Sorry @wu-sheng - I have edited my post above, I copied the config and missed out the OAP self obs rule

[kezhenxu94]

The default value of SW_OTEL_RECEIVER_ENABLED_OC_RULES is istio-controlplane which processes the CP metrics, when you want to add oap to it, remember to also add istio-controlplane back (not override the entire config), e.g istio-controlplane,oap, with comma separated style, in helm command I think you need \, to separate.

[andrewgkew]

Thanks @kezhenxu94 - that is exactly what I am missing. I am actually setting my OC rules to the following

--set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="oap\,k8s-cluster\,k8s-node\,k8s-service" 

So I will update them to

--set oap.env.SW_OTEL_RECEIVER_ENABLED_OC_RULES="oap\,k8s-cluster\,k8s-node\,k8s-service\,istio-controlplane" \

It does clearly say this in your docs, I totally missed that!

Tested and is now working, thanks for your help.