GH-49078: [FS][Azure] Fix lossy pickling of SubTreeFileSystem(base_path, AzureFileSystem(...))

[Tom-Newton]

Rationale for this change

Fix #49078

What changes are included in this PR?

• Implement getters on the C++ side of AzureOptions, for the values that are currently stored only on the python side.
  • This required adding some more member variables
  • I decided to add ClearCredentials , so that it can't get into strange states by configuring one credential type then another. IMO configuring the credentials during initialisation on the AzureOptions would be neater but I don't want to make this PR too big.
• Update the C++ side AzureOptions::Equals
• Remove python side self attributes and instead depend on getters from C++ side.

Are these changes tested?

• Updated tests on the C++ side for the updated Equals and newly added getter methods.
• Added a fixture pickle_with_and_without_subtree_filesystem, which can be used in place of the pickle_module. This adds test combinations with and without wrapping the filesystem in a SubTreeFilesystem before pickling it.

Are there any user-facing changes?

Only that pickling SubTreeFileSystem(base_path, AzureFileSystem(...)) now works properly.

• GitHub Issue: [Python][FS][Azure] Pickling SubTreeFileSystem(base_path, AzureFileSystem(...)) is lossy #49078

[github-actions]

⚠️ GitHub issue #49078 has been automatically assigned in GitHub to PR creator.

[Tom-Newton]

Sorry to be impatient, but please could someone review this? The test failure looks like a timeout, that I don't think is related. @AlenkaF please could you review?

[raulcd]

Hi @Tom-Newton thanks for being patient. There are lots of PRs and issues being opened and sometimes it just takes a while to find the time. Just a really minor nit but the changes look reasonable to me.

[Tom-Newton]

Rebasing on main in the hope it will fix a bunch of test failures that didn't look related.

[raulcd]

Thanks for the PR @Tom-Newton

[Tom-Newton]

Thanks for the PR @Tom-Newton

Thanks for reviewing 🙂

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 0124d5b.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 44 possible false positives for unstable benchmarks that are known to sometimes produce them.