Allow multiple endpoints in Envoy clusters configured with hostnames

[t-davies]

Description

Currently when Consul generates configuration for Envoy, it behaves the same for both LOGICAL_DNS and STRICT_DNS discovery types and provides only a single endpoint to Envoy in both modes. This means that when multiple hostnames are configured in STRICT_DNS mode, requests are not load balanced over all available hosts.

This change configures Envoy with all available hostnames when the discovery type is set to STRICT_DNS. In LOGICAL_DNS mode, Envoy supports only a single endpoint and so the current behaviour of Consul remains unchanged.

Testing & Reproduction steps

Tested manually, as follows:

1. Create Consul cluster

   consul agent -dev
   

2. Create 2 instances of the same service ("example-service"), with different hostnames.

   curl --request PUT \
     --url http://localhost:8500/v1/catalog/register \
     --header 'Content-Type: application/json' \
     --data '{
       "Datacenter": "dc1",
       "Node": "ext-headers.jsontest.com",
       "ID": "adb6d185-300c-40f3-b81b-799cc57f889a",
       "Address": "headers.jsontest.com",
       "NodeMeta": {
         "external-node": "true",
         "external-probe": "false"
       },
       "Service": {
         "ID": "example-service-a063a65b",
         "Service": "example-service",
         "Tags": [],
         "Address": "headers.jsontest.com",
         "Port": 80
       },
       "Checks": [{
         "CheckID": "ext-fff3e68d",
         "Name": "ext-default",
         "Status": "passing",
         "ServiceID": "example-service-a063a65b",
         "Definition": {
           "TCP": "headers.jsontest.com:80",
           "Interval": "15s",
           "Timeout": "5s",
           "DeregisterCriticalServiceAfter": "60m"
          }
       }]
     }'
   

   curl --request PUT \
     --url http://localhost:8500/v1/catalog/register \
     --header 'Content-Type: application/json' \
     --data '{
       "Datacenter": "dc1",
       "Node": "ext-ip.jsontest.com",
       "ID": "b676fcbd-44c8-4bf8-ac05-a3b346e8b3fb",
       "Address": "ip.jsontest.com",
       "NodeMeta": {
         "external-node": "true",
         "external-probe": "false"
       },
       "Service": {
         "ID": "example-service-d94bc56d",
         "Service": "example-service",
         "Tags": [],
         "Address": "ip.jsontest.com",
         "Port": 80
       },
       "Checks": [{
         "CheckID": "ext-49daef29",
         "Name": "ext-default",
         "Status": "passing",
         "ServiceID": "example-service-d94bc56d",
         "Definition": {
           "TCP": "ip.jsontest.com:80",
           "Interval": "15s",
           "Timeout": "5s",
           "DeregisterCriticalServiceAfter": "60m"
          }
       }]
     }'
   

3. Create terminating gateway, configured with "example-service" as a destination.

   curl --request PUT \
     --url http://localhost:8500/v1/config \
     --header 'Content-Type: application/json' \
     --data '{
    "Kind": "terminating-gateway",
    "Name": "ext-terminating-gateway",
    "Services": [
      {
        "Name": "example-service"
      }
    ]
   }'
   

   curl --request PUT \
     --url http://localhost:8500/v1/agent/service/register \
     --header 'Content-Type: application/json' \
     --data '{
       "Kind": "terminating-gateway",
       "Name": "ext-terminating-gateway",
       "Port": 22500
   }'
   

   consul connect envoy -gateway=terminating -service ext-terminating-gateway
   

4. Fetch the Envoy config from the terminating gateway and observe that only a single hostname is present in lb_endpoints (since the default discovery type is LOGICAL_DNS).

   curl --request GET --url http://localhost:19000/config_dump
   

5. Set envoy_dns_discovery_type to "STRICT_DNS" in proxy-defaults.

   curl --request PUT \
     --url http://localhost:8500/v1/config \
     --header 'Content-Type: application/json' \
     --data '{
    "Kind": "proxy-defaults",
    "Name": "global",
    "Config": {
      "protcol": "http",
      "envoy_dns_discovery_type": "STRICT_DNS"
    }
   }'
   

6. Refetch config and observe that the Envoy config now contains both hostnames in lb_endpoints and will load balance incoming requests across all configured endpoints.

   curl --request GET --url http://localhost:19000/config_dump
   

Links

• Envoy service discovery types
• https://support.hashicorp.com/hc/en-us/requests/157078 (related support ticket)

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

[t-davies]

Pretty sure we do still need to break early here to maintain the same behaviour that we had before with regards to how fallbacks are determined, but happy to be corrected so we can remove this slightly clunky check.

[t-davies]

@blake - anyone that can take a look please?