[Windows] Access is denied error when rotating fleet.enc file

[cmacknz]

Filing a bug from an internal report of this error.

C:\Program Files\Elastic\Agent\fleet.enc: rename C:\ProgramFiles\Elastic\Agent\fleet.enc.tmp C:\ProgramFiles\Elastic\Agent\fleet.enc: Access is denied

https://github.com/elastic/elastic-agent-libs/blob/4babafd5ed1e5079acf74212ed3da01740b22de7/file/helper_windows.go#L26-L49

The SafeFileRotate implementation on Windows makes a call to os.Remove that can fail with an Access is Denied error if a process still has the file open when the removal is attempted. It is unclear what process has the file open, potentially there is AV software performing a periodic scan of the file system.

We could add retries like we did for the uninstall command that used to hit this error more frequently:

elastic-agent/internal/pkg/agent/install/uninstall.go

Lines 246 to 253 in 30688d4

	// RemovePath helps with removal path where there is a probability
	// of running into an executable running that might prevent removal
	// on Windows.
	//
	// On Windows it is possible that a removal can spuriously error due
	// to an ERROR_SHARING_VIOLATION. RemovePath will retry up to 2
	// seconds if it keeps getting that error.
	func RemovePath(path string) error {

We could also switch to using Win32 calls directly. Then we could use the MoveFileExA directly with the MoveOnReboot flag. We would have to check what happens if you have multiple moves scheduled on reboot.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)