Failing gRPC xDS Fallback - Multiple xDS servers #8057
Comments
|
@axkum10 the grpc version you have mentioned in the issue is 1.67.1 but the fallback support was added in 1.69.0 and it was enabled by default in 1.70.0. See the release notes https://github.com/grpc/grpc-go/releases. Could you try upgrading your grpc library version? The bootstrap config looks correct to me. You can also refer to these internal e2e tests to get more idea https://github.com/grpc/grpc-go/blob/master/xds/internal/xdsclient/tests/fallback_test.go. @easwars for providing more guidance here. |
purnesh42H
added
Type: Question
Area: xDS
|
Thanks @purnesh42H, upgraded version, still failing to connect to 2nd xDS server. I do not see client making a call with new port of xDS server. It does pick, however I do not see a connect call with different port going out in debug. Test/Scenario: Only one xDS server running and mentioned as 2nd server in the client bootstrap file. Trying to find, if gRPC client can connect to 2nd xDS server if 1st is down. Setup refresh: Client works fine if I put one xDS server in the bootstrap file (tried both 8001 & 8003) Software version: Please refer below 2 debug logs with fallback flag false & true:
This one line (from above debug), shows "8001" port instead of "8003": Please let me know if I am missing something or more information is required. |
|
@axkum10 from the logs, it seems that grpc did initiate the xDS server fallback process. The fallback begins when the channel reports TRANSIENT_FAILURE. I suspect you're not using |
The above line tells me that the connection to the first server mentioned in the bootstrap has filed. The above lines indicate that the xDS client has attempted fallback and has created a channel to the second server mentioned in the bootsttrap. The above lines indicate that the xDS client is still trying to see if it can get a successful connection to the primary server, which is by design. The xDS client will always try to use the highest priority server that is available and therefore will keep retrying the connection to a higher priority server, if one exists. These lines indicate that xDS client is attempting to subscribe to the same resources from the secondary server. All of the above are WAI. I don't see the grpc channel to the secondary server Also, @arjan-bal @axkum10 Could you please confirm if |
|
Allow me to divide conversation in 2 parts, as it may help track an continue discussion on both. A. @easwars your comment: I do not see that as well (debug level is enabled) and program exits. B. Suggestion from @arjan-bal to use grpc.WaitForReady moved the initialization. xDS client was able to get listerner & route information, then terminated after sending cluster info. ---- What I have tested so far ----
Am I missing something. |
|
@easwars the following log line lead me to believe the xDS resolver reported an error to the channel before producing a valid
IIUC, this would put the channel in TRANSIENT_FAILURE due to the following code in gracefulswitch: grpc-go/internal/balancer/gracefulswitch/gracefulswitch.go Lines 197 to 211 in ee3e8d9 Assigning to you since you've mentioned this may be a bug. |
|
Just merged an update to the gRFC that describes the xDS fallback implementation in grpc. See: grpc/proposal#473 Will send a PR to fix the issue shortly. Thanks for filing this issue @axkum10. |
|
Fixed by #8075 |
I have setup gRPC xDS server with client and server working fine. I am trying to introduce multiple gRPC xDS server for high availability.
As per A71 xDS fallback introducing multiple xDS server detail in client bootstrap should have enabled client to look pick a good xDS server, which is not happening. Anyone tried multiple gRPC xDS server and tested client fallback.
Here is client bootstrap file:
{ "xds_servers": [ { "server_uri": "xds.domain.com:8001", "channel_creds": [ { "type": "insecure" } ], "server_features": ["xds_v3"] }, { "server_uri": "xds.domain.com:8003", "channel_creds": [ { "type": "insecure" } ], "server_features": ["xds_v3"] } ], "node": { "id": "test123", "metadata": { "md": "123456789012" } } }Setup: I have just one xDS server running on port 8003.
Steps:
Started xDS server using port 8003 - good
exported client bootstarp file - good
Starting client - error (client startup is not looking at 2nd xDS server detail) Error:
I have not touched "GRPC_EXPERIMENTAL_XDS_FALLBACK", as the comment says, failover is enabled by default.
Software:
The text was updated successfully, but these errors were encountered: