HADOOP-19877: run s3a integration and scale tests in CI

[ajfabbri]

Description of PR

Run AWS integration tests in our CI for pull requests which modify hadoop-tools/hadoop-aws/**

How was this patch tested?

Running the actions here in github.

For code changes:

• Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
• [na] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
• [na] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
• [na] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

• [na] The PR includes the phrase "Contains content generated by "
  where is the name of the AI tool used.
• My use of AI contributions follows the ASF legal policy
  https://www.apache.org/legal/generative-tooling.html

Requirements

• test passes here in CI
• Need to add setup-localstack action to ASF Infra's whitelist: HADOOP-19877: add LocalStack/setup-localstack 0.3.2 infrastructure-actions#801

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
			_ trunk Compile Tests _
+1 💚	shadedclient	22m 13s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shadedclient	15m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	asflicense	0m 21s	/results-asflicense.txt	The patch generated 2 ASF License warnings.
		39m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/1/artifact/out/Dockerfile
GITHUB PR	#8467
Optional Tests	dupname asflicense codespell detsecrets yamllint
uname	Linux 0a1a6b04a029 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / ac00cb2
Max. process+thread count	705 (vs. ulimit of 10000)
modules	C: . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/1/console
versions	git=2.43.0 maven=3.9.11
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[ajfabbri]

FYSA @pan3793. This is still a work-in-progress, but note the composite actions: I copied your build image logic into a composite action(s), so it can be called from multiple reusable workflows. I plan to do this copy-paste for now, and later we can change your existing workflow to reuse the same composite actions.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	2m 7s		Maven dependency ordering for branch
+1 💚	mvninstall	28m 58s		trunk passed
+1 💚	shadedclient	16m 9s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s		Maven dependency ordering for patch
+1 💚	mvninstall	22m 6s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	16m 12s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	10m 27s		root in the patch passed.
-1 ❌	asflicense	0m 38s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		99m 27s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/2/artifact/out/Dockerfile
GITHUB PR	#8467
Optional Tests	dupname asflicense codespell detsecrets yamllint unit shellcheck shelldocs
uname	Linux 963aa91e99d0 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / 8c5e782
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/2/testReport/
Max. process+thread count	610 (vs. ulimit of 10000)
modules	C: hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/2/console
versions	git=2.43.0 maven=3.9.11 shellcheck=0.9.0
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	yamllint	0m 0s		yamllint was not available.
+0 🆗	shelldocs	0m 0s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	2m 0s		Maven dependency ordering for branch
+1 💚	mvninstall	27m 7s		trunk passed
+1 💚	shadedclient	15m 14s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 17s		Maven dependency ordering for patch
+1 💚	mvninstall	19m 56s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	15m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 16s		root in the patch passed.
-1 ❌	asflicense	0m 42s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		92m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/4/artifact/out/Dockerfile
GITHUB PR	#8467
Optional Tests	dupname asflicense codespell detsecrets yamllint unit shellcheck shelldocs
uname	Linux 89de55586062 5.15.0-141-generic #151-Ubuntu SMP Sun May 18 21:35:19 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / d03bce8
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/4/testReport/
Max. process+thread count	637 (vs. ulimit of 10000)
modules	C: hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/4/console
versions	git=2.43.0 maven=3.9.11 shellcheck=0.9.0
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[hadoop-yetus]

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	yamllint	0m 1s		yamllint was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	1m 55s		Maven dependency ordering for branch
+1 💚	mvninstall	30m 57s		trunk passed
+1 💚	shadedclient	16m 47s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 19s		Maven dependency ordering for patch
+1 💚	mvninstall	21m 28s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	shellcheck	0m 0s		No new issues.
+1 💚	shadedclient	16m 23s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	9m 37s		root in the patch passed.
-1 ❌	asflicense	0m 47s	/results-asflicense.txt	The patch generated 3 ASF License warnings.
		100m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/3/artifact/out/Dockerfile
GITHUB PR	#8467
Optional Tests	dupname asflicense codespell detsecrets yamllint unit shellcheck shelldocs
uname	Linux c1e7f63007ae 5.15.0-173-generic #183-Ubuntu SMP Fri Mar 6 13:29:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / abf52d5
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/3/testReport/
Max. process+thread count	629 (vs. ulimit of 10000)
modules	C: hadoop-tools/hadoop-aws . U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8467/3/console
versions	git=2.43.0 maven=3.9.11 shellcheck=0.9.0
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

[pan3793]

Have you tried this in your forked repo before the ASF infra team approves it?

[ajfabbri]

Yes I'm testing this here. @pan3793 I should have tested before I created a PR to add the action, because I ended up abandoning it. It tries to start localstack in the same container. I've switched this to use services: to pull the container directly instead of using their setup action.

I have a bunch of changes there. Once I get it working well I will force-push an update here. There are a bunch of changes since I'm iterating on it in my fork PR.