Skip to content

Latest commit

 

History

History
352 lines (250 loc) · 22.9 KB

using-openstack-cloud-controller-manager.md

File metadata and controls

352 lines (250 loc) · 22.9 KB
Raw

Table of Contents generated with DocToc

Get started with external openstack-cloud-controller-manager in Kubernetes

External cloud providers were introduced as an Alpha feature in Kubernetes release 1.6. openstack-cloud-controller-manager is the implementation of external cloud provider for OpenStack clusters. An external cloud provider is a kubernetes controller that runs cloud provider-specific loops required for the functioning of kubernetes. These loops were originally a part of the kube-controller-manager, but they were tightly coupling the kube-controller-manager to cloud-provider specific code. In order to free the kubernetes project of this dependency, the cloud-controller-manager was introduced.

cloud-controller-manager allows cloud vendors and kubernetes core to evolve independent of each other. In prior releases, the core Kubernetes code was dependent upon cloud provider-specific code for functionality. In future releases, code specific to cloud vendors should be maintained by the cloud vendor themselves, and linked to cloud-controller-manager while running Kubernetes.

For more information about cloud-controller-manager, please see:

NOTE: Now, the openstack-cloud-controller-manager implementation is based on OpenStack Octavia, Neutron-LBaaS has been removed in openstack-cloud-controller-manager since v1.26.0. So make sure to use Octavia if upgrade to the latest openstack-cloud-controller-manager docker image.

Deploy a Kubernetes cluster with openstack-cloud-controller-manager using kubeadm

The following guide has been tested to install Kubernetes v1.17 on Ubuntu 18.04.

Prerequisites

  • docker, kubeadm, kubelet and kubectl has been installed.

Steps

  • Create the kubeadm config file according to manifests/controller-manager/kubeadm.conf

  • Bootstrap the cluster, make sure to install the CNI network plugin as well.

    kubeadm init --config kubeadm.conf

  • Bootstrap worker nodes. You need to set --cloud-provider=external for kubelet service before running kubeadm join.

  • Create a secret containing the cloud configuration. You can find an example config file in manifests/controller-manager/cloud-config. If you have certs you need put the cert file into folder /etc/ssl/certs/ and update ca-file in the configuration file, refer to ca-file option here for further information. After that, Save the configuration to a file named cloud.conf, then:

    kubectl create secret -n kube-system generic cloud-config --from-file=cloud.conf

  • Create RBAC resources and openstack-cloud-controller-manager daemonset.

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml

  • Waiting for all the pods in kube-system namespace up and running.

Migrating from in-tree openstack cloud provider to external openstack-cloud-controller-manager

If you are already running a Kubernetes cluster (installed by kubeadm) but using in-tree openstack cloud provider, switching to openstack-cloud-controller-manager is easy by following the steps in the demo below.

asciicast

Also, checkout the guide on Migrate to CCM

Config openstack-cloud-controller-manager

Implementation of openstack-cloud-controller-manager relies on several OpenStack services.

Service API Version(s) Deprecated Required
Identity (Keystone) v3 No Yes
Compute (Nova) v2 No Yes
Load Balancing (Octavia) v2 No Yes
Key Manager (Barbican) v1 No No

NOTE:

  • Block Storage is not needed for openstack-cloud-controller-manager in favor of cinder-csi-plugin.
  • Barbican is required to support creating Service of LoadBalancer type with TLS termination.

Global

The options in Global section are used for openstack-cloud-controller-manager authentication with OpenStack Keystone, they are similar to the global options when using openstack CLI, see more information in openstack man page.

  • auth-url Required. Keystone service URL, e.g. http://128.110.154.166/identity

  • os-endpoint-type Optional. Specify which type of endpoint to use from the service catalog. If not set, public endpoints are used.

  • ca-file Optional. CA certificate bundle file for communication with Keystone service, this is required when using the https protocol in the Keystone service URL.

  • cert-file Optional. Client certificate path used for the client TLS authentication.

  • key-file Optional. Client private key path used for the client TLS authentication.

  • username Keystone user name. If you are using Keystone application credential, this option is not required.

  • password Keystone user password. If you are using Keystone application credential, this option is not required.

  • region Required. Keystone region name.

  • domain-id Keystone user domain ID. If you are using Keystone application credential, this option is not required.

  • domain-name Keystone user domain name, not required if domain-id is set.

  • tenant-id Keystone project ID. When using Keystone V3 - which changed the identifier tenant to project - the tenant-id value is automatically mapped to the project construct in the API.

    tenant-id is not needed when using trust-id or Keystone application credential

  • tenant-name Keystone project name, not required if tenant-id is set.

  • tenant-domain-id Keystone project domain ID.

  • tenant-domain-name Keystone project domain name.

  • user-domain-id Keystone user domain ID.

  • user-domain-name Keystone user domain name.

  • trust-id Keystone trust ID. A trust represents a user's (the trustor) authorization to delegate roles to another user (the trustee), and optionally allow the trustee to impersonate the trustor. Available trusts are found under the /v3/OS-TRUST/trusts endpoint of the Keystone API.

  • trustee-id Keystone trustee user ID.

  • trustee-password Keystone trustee user password.

  • use-clouds Set this option to true to get authorization credentials from a clouds.yaml file. Options explicitly set in this section are prioritized over values read from clouds.yaml, the file path can be set in clouds-file option. Otherwise, the following order is applied:

    1. A file path stored in the environment variable OS_CLIENT_CONFIG_FILE
    2. The directory pkg/openstack
    3. The directory ~/.config/openstack
    4. The directory /etc/openstack

  • clouds-file File path of a clouds.yaml file, used together with use-clouds=true.

  • cloud Used to specify which named cloud in the clouds.yaml file that you want to use, used together with use-clouds=true.

  • application-credential-id The ID of an application credential to authenticate with. An application-credential-secret has to be set along with this parameter.

  • application-credential-name The name of an application credential to authenticate with. If application-credential-id is not set, the user name and domain need to be set.

  • application-credential-secret The secret of an application credential to authenticate with.

  • tls-insecure If set to true, then the server’s certificate will not be verified. Default is false.

Networking

  • ipv6-support-disabled Indicates whether or not IPv6 is supported. Default: false

  • public-network-name The name of Neutron external network. openstack-cloud-controller-manager uses this option when getting the external IP of the Kubernetes node. Can be specified multiple times. Specified network names will be ORed. Default: ""

  • internal-network-name The name of Neutron internal network. openstack-cloud-controller-manager uses this option when getting the internal IP of the Kubernetes node, this is useful if the node has multiple interfaces. Can be specified multiple times. Specified network names will be ORed. Default: ""

  • address-sort-order This configuration key influences the way the provider reports the node addresses to the Kubernetes node resource. The default order depends on the hard-coded order the provider queries the addresses and what the cloud returns, which does not guarantee a specific order.

    To override this behavior it is possible to specify a comma separated list of CIDRs. Essentially, this will sort and group all addresses matching a CIDR in a prioritized manner, where the first item having a higher priority than the last. All non-matching addresses will remain in the same order they are already in.

    For example, this option can be useful when having multiple or dual-stack interfaces attached to a node and needing a user-controlled, deterministic way of sorting the addresses. Default: ""

Route

  • router-id Specifies the Neutron router ID to activate route controller to manage Kubernetes cluster routes.

    NOTE: This require openstack-cloud-controller-manager's --cluster-cidr flag to be set.

Load Balancer

Although the openstack-cloud-controller-manager was initially implemented with Neutron-LBaaS support, Octavia is mandatory now because Neutron-LBaaS has been deprecated since Queens OpenStack release cycle and no longer accepted new feature enhancements. As a result, since v1.26.0 the Neutron-LBaaS is not supported in openstack-cloud-controller-manager and removed from code repo.

  • enabled Whether or not to enable the LoadBalancer type of Services integration at all. Default: true

  • floating-network-id Optional. The external network used to create floating IP for the load balancer VIP. If there are multiple external networks in the cloud, either this option must be set or user must specify loadbalancer.openstack.org/floating-network-id in the Service annotation.

  • floating-subnet-id Optional. The external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotation loadbalancer.openstack.org/floating-subnet-id.

  • floating-subnet Optional. A name pattern (glob or regexp if starting with ~) for the external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotation loadbalancer.openstack.org/floating-subnet. If multiple subnets match the first one with still available IPs is used.

  • floating-subnet-tags Optional. Tags for the external network subnet used to create floating IP for the load balancer VIP. Can be overridden by the Service annotation loadbalancer.openstack.org/floating-subnet-tags. If multiple subnets match the first one with still available IPs is used.

  • lb-method The load balancing algorithm used to create the load balancer pool.

    If lb-provider is set to "amphora" or "octavia" the value can be one of:

    • ROUND_ROBIN (default)
    • LEAST_CONNECTIONS
    • SOURCE_IP

    If lb-provider is set to "ovn" the value must be set to SOURCE_IP_PORT.

  • lb-provider Optional. Used to specify the provider of the load balancer, e.g. "amphora" (default), "octavia" (deprecated alias for "amphora"), or "ovn". Only the "amphora", "octavia", and "ovn" providers are officially tested, other providers will cause a warning log.

  • lb-version Optional. If specified, only "v2" is supported.

  • subnet-id ID of the Neutron subnet on which to create load balancer VIP. This ID is also used to create pool members, if member-subnet-id is not set. For dual-stack deployments it's recommended to not set this option and let cloud-provider-openstack autodetect which subnet to use for which load balancer.

  • member-subnet-id ID of the Neutron network on which to create the members of the load balancer. The load balancer gets another network port on this subnet. Defaults to subnet-id if not set.

  • network-id ID of the Neutron network on which to create load balancer VIP, not needed if subnet-id is set. If not set network will be autodetected based on the network used by cluster nodes.

  • manage-security-groups If the Neutron security groups should be managed separately. Default: false

  • create-monitor Indicates whether or not to create a health monitor for the service load balancer. A health monitor required for services that declare externalTrafficPolicy: Local. Default: false

    NOTE: Health monitors for the ovn provider are only supported on OpenStack Wallaby and later.

  • monitor-delay The time, in seconds, between sending probes to members of the load balancer. Default: 5

  • monitor-max-retries The number of successful checks before changing the operating status of the load balancer member to ONLINE. A valid value is from 1 to 10. Default: 1

  • monitor-max-retries-down The number of unsuccessful checks before changing the operating status of the load balancer member to ERROR. A valid value is from 1 to 10. Default: 3

  • monitor-timeout The maximum time, in seconds, that a monitor waits to connect backend before it times out. Default: 3

  • internal-lb Determines whether or not to create an internal load balancer (no floating IP) by default. Default: false.

  • node-selector A comma separated list of key=value annotations used to filter nodes for targeting by the load balancer. When defined, only nodes that match all the specified key=value annotations will be targeted. If an annotation includes only a key without a value, the filter will check only for the existence of the key on the node. When node-selector is not set (default value), all nodes will be added as members to a load balancer pool.

    Note: This configuration option can be overridden with the loadbalancer.openstack.org/node-selector service annotation. Refer to Exposing applications using services of LoadBalancer type

    Example: To filter nodes with the labels env=production and region=default, set the node-selector as follows:

    node-selector="env=production, region=default"

    Example: To filter nodes that have the key env with any value and the key region specifically set to default, set the node-selector as follows:

    node-selector="env, region=default"

  • cascade-delete Determines whether or not to perform cascade deletion of load balancers. Default: true.

  • flavor-id The id of the loadbalancer flavor to use. Uses octavia default if not set.

  • availability-zone The name of the loadbalancer availability zone to use. The Octavia availability zone capabilities will not be used if it is not set. The parameter will be ignored if the Octavia version doesn't support availability zones yet.

  • LoadBalancerClass "ClassName" This is a config section including a set of config options. User can choose the ClassName by specifying the Service annotation loadbalancer.openstack.org/class. The following options are supported:

    • floating-network-id. The same with floating-network-id option above.
    • floating-subnet-id. The same with floating-subnet-id option above.
    • floating-subnet. The same with floating-subnet option above.
    • floating-subnet-tags. The same with floating-subnet-tags option above.
    • network-id. The same with network-id option above.
    • subnet-id. The same with subnet-id option above.
    • member-subnet-id. The same with member-subnet-id option above.

  • enable-ingress-hostname

    Used with proxy protocol (set by annotation loadbalancer.openstack.org/proxy-protocol: "true") by adding a dns suffix (nip.io) to the load balancer IP address. Default false.

    This option is currently a workaround for the issue kubernetes/ingress-nginx#3996, should be removed or refactored after the Kubernetes KEP-1860 is implemented.

  • ingress-hostname-suffix

    The dns suffix to the load balancer IP address when using proxy protocol. Default nip.io

    This option is currently a workaround for the issue kubernetes/ingress-nginx#3996, should be removed or refactored after the Kubernetes KEP-1860 is implemented.

  • default-tls-container-ref Reference to a tls container or secret. This option works with Octavia, when this option is set then the cloud provider will create an Octavia Listener of type TERMINATED_HTTPS for a TLS Terminated loadbalancer.

    Accepted format for tls container ref are https://{keymanager_host}/v1/containers/{uuid} and https://{keymanager_host}/v1/secrets/{uuid}. Check container-store parameter if you want to disable validation.

  • container-store Optional. Used to specify the store of the tls-container-ref, e.g. "barbican" or "external" - other store will cause a warning log. Default value - barbican - existence of tls container ref would always be performed.

    If set to external format for tls container ref will not be validated.

  • max-shared-lb The maximum number of Services that share a load balancer. Default: 2

  • provider-requires-serial-api-calls Some Octavia providers do not support creating fully-populated loadbalancers using a single API call. Setting this option to true will create loadbalancers using serial API calls which first create an unpopulated loadbalancer, then populate its listeners, pools and members. This is a compatibility option at the expense of increased load on the OpenStack API. Default: false

NOTE:

  • environment variable OCCM_WAIT_LB_ACTIVE_STEPS is used to provide steps of waiting loadbalancer to be ready. Current default wait steps is 23 and setup the environment variable overrides default value. Refer to Backoff.Steps for further information.

Metadata

  • search-order This configuration key influences the way that the provider retrieves metadata relating to the instance(s) in which it runs. The default value of configDrive,metadataService results in the provider retrieving metadata relating to the instance from the config drive first if available and then the metadata service. Alternative values are:

    • configDrive - Only retrieve instance metadata from the configuration drive.
    • metadataService - Only retrieve instance metadata from the metadata service.
    • metadataService,configDrive - Retrieve instance metadata from the metadata service first if available, then the configuration drive.

    Not all OpenStack clouds provide both configuration drive and metadata service though and only one or the other may be available which is why the default is to check both. Especially, the metadata on the config drive may grow stale over time, whereas the metadata service always provides the most up to date data.

Multi region support (alpha)

  • environment variable OS_CCM_REGIONAL is set to true - allow CCM to set ProviderID with region name ${ProviderName}://${REGION}/${instance-id}. Default: false.

Exposing applications using services of LoadBalancer type

Refer to Exposing applications using services of LoadBalancer type

Metrics

Refer to Metrics for openstack-cloud-controller-manager

Limitation

OpenStack availability zone must not contain blank

topology.kubernetes.io/zone is used to label node and its value comes from availability zone of the node, according to label spec it does not support blank (' ') but OpenStack availability zone supports blank. So your OpenStack availability zone must not contain blank otherwise it will lead to node that belongs to this availability zone register failure, see #1379 for further information.