New feature: Reconcile commands (CopyCommand, MigrateCommand, MigrateVolumeCommand)

[weizhouapache]

Description

This PR aims to improve the process on some agent commands and answers.

Current process

Many cloudstack operations require the communication between management server and cloudstack agent.
The normal process is

management server --> send commands to agents --> agents process the commands ->
agents send the answers to management server --> management server process the answers

Each operation might have one or more processes above.

Issues in some scenarios

Normally the process works fine. However, there are some issues in some scenarios

• agent lost connectivity to management server
• agent crashes
• agent is stopped or restarted
• management server is restarted (not in scope of FR doc)

Consider the following examples

• agent has processed the command completely, and sent the answer to management server but the answer is not received on management server (connectivity issue or worker threads are all in use), therefore timed out on management server
• agent has asked 3rd party (for example libvirt or scaleio) to process the command, but agent crashes, which leads to some inconsistent state of resources. For example vm has been migrated, volume has been copied to destination pool, but management server does not get the results.
• management is restarted while vm/volume is being migrated, which leads vm/volume to be stuck at Migrating state.

Operations to address

This FR focuses on the following operations

• migrate vm
• migrate vm with volumes
• migrate volume to another pool

The backend processes can be found at
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=337678693#AsyncAgentCommandReconciliation-4.1BackendcommandsofVMandvolumemigrations

Main changes

Design doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation

Global settings

• reconcile.commands.enabled (default: true)
• reconcile.commands.interval (default: 60)
• reconcile.commands.max.attempts (default: 30)
• reconcile.commands.workers (default: 100)

New terminology: Reconcile commands

• Add command state: CommandInfo.State
  • Management server: CREATED -> COMPLETED/FAILED or INTERRUPTED/TIMED_OUT -> RECONCILING -> RECONCILED/RECONCILE_RETRY/RECONCILE_FAILED
  • Agent: STARTED -> PROCESSING -> COMPLETED/FAILED/INTERRUPTED
• Add property to Command: isReconcile (false by default). true for 3 commands
  • CopyCommand
  • MigrateCommand
  • MigrateVolumeCommand

How it works

• management server creates a record in reconcile_commands table when create the command
• agent updates the command/answer in a JSON file while process the command
• agent syncs with management server every 60 seconds (ping.interval)
• when management server receives the update from agent, it updates the database
• when agent receives the PingAnswer form management server, it removes the JSON file with state=COMPLETED/FAILED

For reconcile commands, during stop/start of mgmt server and agent

• when agent is stopped/started, it updates the state (by agent) to INTERUPTED in JSON files and sync with mgmt server
• when mgmt server is stopped/started, it updates the state (by management) to INTERUPTED in database
• Every minute, management server loads the reconcile commands and reconcile the command in INTERUPTED or TIMED_OUT state

Improvement on management server when wait for the answer of reconcile commands

• meanwhile, every 60 seconds, it reads the state and answer from database
• if state is INTERUPTED or FAILED, it terminates the process
• if state is COMPLETED and answer is null, it processes the answer as normal
• This fixes the intermittent connection failure between agent and management server
• when times out, it updates the state (by mgmt) to TIMED_OUT in database for further reconciliation

Improvement on VM migration w/wo volumes

• If the operation fails with operation timeout exception, it might because of the connection failure between soource host and management server
• check if the vm is Running on destination host, if yes, consider the migration succceed
• if succeed, consider migration as success, and commit the network and volume changes.
• if not, destroy the vm on destination host and rollback the network/volume changes

Fixes after Volume migration

• on NFS, fix the new volumes at Creating state after failed migration
• on Powerflex, fix the read-only VM issue by checking if volume has been migrated by checking via ScaleIO gateway

Improvement on Agent

• Add MigrationCancelHook to terminate vm migration jobs when agent is disconnected or restarted
• Add VolumeMigrationCancelHook to terminate block copy jobs when agent is disconnected or restarted

Test results

It has been tested by dev on NFS and Powerflex

Refer to https://cwiki.apache.org/confluence/display/CLOUDSTACK/Async+Agent+Command+Reconciliation#AsyncAgentCommandReconciliation-4.3Summaryoftestresults

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI
• test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

Attention: Patch coverage is 11.62540% with 1642 lines in your changes missing coverage. Please review.

Project coverage is 16.15%. Comparing base (1c1dad9) to head (3359ec8).

Files with missing lines	Patch %	Lines
...loudstack/command/ReconcileCommandServiceImpl.java	0.00%	711 Missing ⚠️
...source/wrapper/LibvirtReconcileCommandWrapper.java	0.62%	160 Missing ⚠️
.../apache/cloudstack/command/ReconcileCommandVO.java	0.00%	84 Missing ⚠️
...ache/cloudstack/command/ReconcileCommandUtils.java	37.30%	73 Missing and 6 partials ⚠️
...ervisor/kvm/resource/LibvirtComputingResource.java	26.16%	76 Missing and 3 partials ⚠️
...loudstack/command/dao/ReconcileCommandDaoImpl.java	0.00%	58 Missing ⚠️
...ain/java/com/cloud/agent/manager/AgentAttache.java	3.70%	52 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	3.03%	32 Missing ⚠️
...torage/motion/StorageSystemDataMotionStrategy.java	0.00%	31 Missing ⚠️
...atastore/driver/ScaleIOPrimaryDataStoreDriver.java	21.62%	28 Missing and 1 partial ⚠️
... and 43 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10514      +/-   ##
============================================
- Coverage     16.17%   16.15%   -0.02%     
- Complexity    13291    13326      +35     
============================================
  Files          5668     5687      +19     
  Lines        498179   499972    +1793     
  Branches      60290    60562     +272     
============================================
+ Hits          80581    80791     +210     
- Misses       408578   410137    +1559     
- Partials       9020     9044      +24     

Flag	Coverage Δ
uitests	3.99% <ø> (ø)
unittests	17.00% <11.62%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

@blueorangutan package

[blueorangutan]

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12678

[weizhouapache]

@blueorangutan test matrix

[blueorangutan]

@weizhouapache a [SL] Trillian-Jenkins matrix job (EL8 mgmt + EL8 KVM, Ubuntu22 mgmt + Ubuntu22 KVM, EL8 mgmt + VMware 7.0u3, EL9 mgmt + XCP-ng 8.2 ) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-12593)

[blueorangutan]

[SF] Trillian Build Failed (tid-12594)

[blueorangutan]

[SF] Trillian test result (tid-12592)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 58377 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12592-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	1521.32	test_network.py

[blueorangutan]

[SF] Trillian test result (tid-12595)
Environment: xcpng82 (x2), Advanced Networking with Mgmt server ol9
Total time taken: 77369 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10514-t12595-xcpng82.zip
Smoke tests completed. 135 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestClusterDRS>:setup	Error	0.00	test_cluster_drs.py
test_01_prepare_and_cancel_maintenance	Error	0.11	test_ms_maintenance_and_safe_shutdown.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:setup	Error	10.65	test_network.py
test_01_non_strict_host_anti_affinity	Error	241.59	test_nonstrict_affinity_group.py
test_02_non_strict_host_affinity	Error	121.60	test_nonstrict_affinity_group.py
test_02_create_volume	Error	5.30	test_resource_names.py
test_05_scale_vm_dont_allow_disk_offering_change	Failure	69.74	test_scale_vm.py