Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Attaching additional port to instance: novaclient.exceptions.ClientException: Unexpected API Error #768

Open
freyes opened this issue May 23, 2022 · 2 comments
Open

Attaching additional port to instance: novaclient.exceptions.ClientException: Unexpected API Error #768

freyes opened this issue May 23, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@freyes
Copy link
Member

freyes commented May 23, 2022

when running a CI job the z-o-t test that adds a second nic to a instance may fail due to undercloud (transient) issues, so it would be appropriate to make the method create_additional_port_for_machines() idempotent and wrapped with a retry decorator.

Issue found at https://openstack-ci-reports.ubuntu.com/artifacts/afd/841979/1/check/ceph-focal-wallaby_nova/afd2b80/

Relevant stacktrace:

2022-05-19 11:00:50.939845 | focal-medium | 2022-05-19 11:00:50 [INFO] AUTH_URL: http://10.245.161.156:5000/v3, api_ver: 3
2022-05-19 11:00:50.940030 | focal-medium | 2022-05-19 11:00:50 [INFO] Using keystone API V3 (or later) for undercloud auth
2022-05-19 11:01:00.810025 | focal-medium | 2022-05-19 11:01:00 [INFO] looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
2022-05-19 11:01:08.272579 | focal-medium | 2022-05-19 11:01:08 [WARNING] Adding second interface for dataport to guest netplan for bionic-queens and later
2022-05-19 11:01:08.272842 | focal-medium | 2022-05-19 11:01:08 [INFO] Configuring network for OpenStack undercloud/provider
2022-05-19 11:01:26.640403 | focal-medium | 2022-05-19 11:01:26 [INFO] Attaching additional port to instance ("f6959d28-95a3-45fa-b96a-089cf4435a9a"), connected to net id: 94a87d6a-6d82-4931-878d-aedcf0e62962
2022-05-19 11:02:35.824977 | focal-medium | 2022-05-19 11:02:35 [ERROR] {'ceph': 'zaza-3ac3c2b5764f'}
2022-05-19 11:02:35.825279 | focal-medium | 2022-05-19 11:02:35 [ERROR] Model ceph (zaza-3ac3c2b5764f)
2022-05-19 11:02:37.419028 | focal-medium | Traceback (most recent call last):
2022-05-19 11:02:37.419319 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/bin/functest-run-suite", line 8, in <module>
2022-05-19 11:02:37.419350 | focal-medium |     sys.exit(main())
2022-05-19 11:02:37.419408 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 364, in main
2022-05-19 11:02:37.419486 | focal-medium |     func_test_runner(
2022-05-19 11:02:37.419521 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 277, in func_test_runner
2022-05-19 11:02:37.419536 | focal-medium |     run_env_deployment(env_deployment, keep_model=preserve_model,
2022-05-19 11:02:37.419563 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 151, in run_env_deployment
2022-05-19 11:02:37.419579 | focal-medium |     configure.configure(
2022-05-19 11:02:37.419591 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
2022-05-19 11:02:37.419604 | focal-medium |     run_configure_list(functions)
2022-05-19 11:02:37.419616 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
2022-05-19 11:02:37.419627 | focal-medium |     utils.get_class(func)()
2022-05-19 11:02:37.419642 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 103, in basic_overcloud_network
2022-05-19 11:02:37.419661 | focal-medium |     network.setup_gateway_ext_port(network_config,
2022-05-19 11:02:37.419673 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/configure/network.py", line 286, in setup_gateway_ext_port
2022-05-19 11:02:37.419685 | focal-medium |     openstack_utils.configure_gateway_ext_port(
2022-05-19 11:02:37.419703 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 1047, in configure_gateway_ext_port
2022-05-19 11:02:37.419715 | focal-medium |     macs = create_additional_port_for_machines(
2022-05-19 11:02:37.419726 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 945, in create_additional_port_for_machines
2022-05-19 11:02:37.419738 | focal-medium |     server.interface_attach(port_id=port['port']['id'],
2022-05-19 11:02:37.419750 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-05-19 11:02:37.419761 | focal-medium |     return methods[-1].func(obj, *args, **kwargs)
2022-05-19 11:02:37.419773 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/v2/servers.py", line 613, in interface_attach
2022-05-19 11:02:37.419785 | focal-medium |     return self.manager.interface_attach(self, port_id, net_id, fixed_ip)
2022-05-19 11:02:37.419797 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
2022-05-19 11:02:37.419880 | focal-medium |     return methods[-1].func(obj, *args, **kwargs)
2022-05-19 11:02:37.419898 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/v2/servers.py", line 2065, in interface_attach
2022-05-19 11:02:37.419908 | focal-medium |     return self._create('/servers/%s/os-interface' % base.getid(server),
2022-05-19 11:02:37.419919 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/base.py", line 364, in _create
2022-05-19 11:02:37.419930 | focal-medium |     resp, body = self.api.client.post(url, body=body)
2022-05-19 11:02:37.419941 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 401, in post
2022-05-19 11:02:37.419951 | focal-medium |     return self.request(url, 'POST', **kwargs)
2022-05-19 11:02:37.419962 | focal-medium |   File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
2022-05-19 11:02:37.419972 | focal-medium |     raise exceptions.from_response(resp, body, url, method)
2022-05-19 11:02:37.419985 | focal-medium | novaclient.exceptions.ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2022-05-19 11:02:37.420069 | focal-medium | <class 'oslo_messaging.exceptions.MessagingTimeout'> (HTTP 500) (Request-ID: req-18ba7adb-f06c-46b3-9bd0-4ac06233d99f)
2022-05-19 11:02:37.785417 | focal-medium | ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-nova-compute/.tox/func-target/bin/functest-run-suite --keep-model --bundle ceph:focal-wallaby (exited with code 1)
@freyes freyes added the enhancement New feature or request label May 23, 2022
@freyes
Copy link
Member Author

freyes commented May 24, 2022

the approach I have in mind is to create the port using a set of tags that can identify the port, so in a subsequent call (a retry) we can look for a port with those tags to know if the port was really created in the background.

Tags:

model_name=$modelname
unit_id=$principal_unit

@freyes
Copy link
Member Author

freyes commented May 26, 2022

Another occurrence:

https://openstack-ci-reports.ubuntu.com/artifacts/125/841978/1/check/vault-focal-xena/12583a5/

2022-05-23 15:10:23 [INFO] Attaching additional port to instance ("69e1ec08-d847-4018-b13c-c56bd9804419"), connected to net id: 94a87d6a-6d82-4931-878d-aedcf0e62962
2022-05-23 15:11:53 [ERROR] {'vault': 'zaza-4b8742462d18'}
2022-05-23 15:11:53 [ERROR] Model vault (zaza-4b8742462d18)
Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1348, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 316, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 285, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 1022, in _send_request
    resp = self.session.request(method, url, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 364, in main
    func_test_runner(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 277, in func_test_runner
    run_env_deployment(env_deployment, keep_model=preserve_model,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 151, in run_env_deployment
    configure.configure(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 51, in configure
    run_configure_list(functions)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/charm_lifecycle/configure.py", line 37, in run_configure_list
    utils.get_class(func)()
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/charm_tests/neutron/setup.py", line 103, in basic_overcloud_network
    network.setup_gateway_ext_port(network_config,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/configure/network.py", line 286, in setup_gateway_ext_port
    openstack_utils.configure_gateway_ext_port(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 1047, in configure_gateway_ext_port
    macs = create_additional_port_for_machines(
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/zaza/openstack/utilities/openstack.py", line 944, in create_additional_port_for_machines
    port = neutronclient.create_port(body=body_value)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 803, in create_port
    return self.post(self.ports_path, body=body)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 358, in post
    return self.do_request("POST", action, body=body,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/v2_0/client.py", line 281, in do_request
    resp, replybody = self.httpclient.do_request(action, method, body=body,
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/client.py", line 340, in do_request
    return self.request(url, method, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/neutronclient/client.py", line 328, in request
    resp = super(SessionClient, self).request(*args, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 257, in request
    return self.session.request(url, method, **kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 931, in request
    resp = send(**kwargs)
  File "/home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/lib/python3.8/site-packages/keystoneauth1/session.py", line 1038, in _send_request
    raise exceptions.ConnectFailure(msg)
keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://10.245.161.159:9696/v2.0/ports: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
ERROR: InvocationError for command /home/ubuntu/src/review.opendev.org/openstack/charm-nova-cloud-controller/.tox/func-target/bin/functest-run-suite --keep-model --bundle vault:focal-xena (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   func-target: commands failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@freyes