Ozone S3 + Spark (S3A filesystem) Issues with bucket operations

[mcgrawia]

Hi,

I'm trying to use Ozone for the first time and running into issues with Spark / S3 compatibility and was hoping someone here may be able to help me. Thanks in advance!

Summary of problem

My Spark job attempts to overwrite a Spark ModelPipeline object:

LOGGER.info("Saving pipeline")
val baseS3Path = getS3ModelBasePath(modelId, orgId)
endToEndPipelineModel.write.overwrite.save(baseS3Path + "isolation_forest_pipeline")

This jobs fails with the following errors:

24/08/08 15:12:56 INFO TrainIsolationForest: Saving pipeline
24/08/08 15:12:56 INFO FileSystemOverwrite: Path s3a://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline already exists. It will be overwritten.
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.129.1.229:51248 in memory (size: 7.9 KiB, free: 434.3 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.128.1.127:51248 in memory (size: 7.9 KiB, free: 116.9 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_59_piece0 on 10.128.1.126:51248 in memory (size: 7.9 KiB, free: 116.9 MiB)
24/08/08 15:12:57 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 3
24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:12:57 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.129.1.229:51248 in memory (size: 54.7 KiB, free: 434.4 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.128.1.127:51248 in memory (size: 54.7 KiB, free: 117.0 MiB)
24/08/08 15:12:57 INFO BlockManagerInfo: Removed broadcast_57_piece0 on 10.128.1.126:51248 in memory (size: 54.7 KiB, free: 117.0 MiB)
24/08/08 15:12:58 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 2
24/08/08 15:12:58 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:12:58 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:12:59 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 1
24/08/08 15:12:59 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:03 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
24/08/08 15:13:03 INFO HadoopMapRedCommitProtocol: Using output committer class org.apache.hadoop.mapred.FileOutputCommitter
24/08/08 15:13:03 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
24/08/08 15:13:03 INFO FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
24/08/08 15:13:03 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:03 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:06 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:11 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:22 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:13:30 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: Bulk delete operation failed to delete all objects; failure count = 7
24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/: Directory is not empty. Key:anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/isolation_forest/: Directory is not empty. Key:anomaly_detection/isolation_forest

24/08/08 15:14:04 INFO MultiObjectDeleteSupport: InternalError: anomaly_detection/: Directory is not empty. Key:anomaly_detection

When I list the contents of the bucket, I see a lot of empty files, which seem to be preventing the MultiObjectDelete command from succeeding:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/
2024-08-08 11:03:25          0 anomaly_detection/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/
2024-08-08 11:03:25          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/0/

Troubleshooting steps

I've also tried creating my bucket using the --layout OBJECT_STORE format. This seems to cause a different issue. I no longer see the error above, but my spark job eventually fails with this log:

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error (Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null

When I look into the S3G pod, I see the following logs:

2024-08-08 14:37:14,675 [qtp313082880-328] WARN  server.HttpChannel (HttpChannel.java:handleException(776)) - /bucket
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more
2024-08-08 14:37:14,675 [qtp313082880-328] WARN  server.HttpChannelState (HttpChannelState.java:onError(787)) - unhandled due to prior sendError
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more

Looking using the AWS CLI, I can see there seems to be a directory marker file in the bucket:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/
2024-08-08 10:00:56          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/

But when I attempt to get this file, the AWS CLI fails:

aws s3 ls --endpoint http://localhost:9878 s3://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=3c1e36b1-6be4-448c-87a2-de0def6edafc/

An error occurred (500) when calling the ListObjectsV2 operation (reached max retries: 2): Internal Server Error

Some Background / Setup Details

I'm running Ozone via the Ozone helm chart in Kubernetes with default values, using version 1.4.0:

helm upgrade --install -n default ozone ./charts/ozone

I created my bucket with the following commands:

ozone sh volume create /s3v
ozone sh volume create /vol1

ozone sh bucket create /vol1/bucket
ozone sh bucket link /vol1/bucket /s3v/bucket

and set the S3G domain in my ozone-site.xml:

<property><name>ozone.s3g.domain.name</name><value>s3g.internal</value></property>

My current Spark S3A settings are the following:

spark.hadoop.com.amazonaws.services.s3a.enableV4             true
spark.hadoop.fs.s3a.impl                                     org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.connection.ssl.enabled                   true
spark.hadoop.fs.s3a.path.style.access                        true
spark.hadoop.fs.s3a.change.detection.version.required        false
spark.hadoop.fs.s3a.change.detection.mode                    none
spark.hadoop.fs.s3a.aws.credentials.provider                 org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,com.amazonaws.auth.DefaultAWSCredentialsProviderChain

Additionally, I have also tried setting spark.hadoop.fs.s3a.directory.marker.retention=delete, and it did not seem to change anything.

Thanks for the help 🙏

[mcgrawia]

I also tested by setting spark.hadoop.fs.s3a.multiobjectdelete.enable=false, and it still fails with this following error:

24/08/08 16:10:54 ERROR Instrumentation: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory s3a://bucket/anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata already exists
	at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:131)
	at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.assertConf(SparkHadoopWriter.scala:299)
	at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:71)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopDataset$1(PairRDDFunctions.scala:1090)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1088)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$4(PairRDDFunctions.scala:1061)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1026)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$3(PairRDDFunctions.scala:1008)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1007)
	at org.apache.spark.rdd.PairRDDFunctions.$anonfun$saveAsHadoopFile$2(PairRDDFunctions.scala:964)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:962)
	at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$2(RDD.scala:1578)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1578)
	at org.apache.spark.rdd.RDD.$anonfun$saveAsTextFile$1(RDD.scala:1564)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
	at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1564)
	at org.apache.spark.ml.util.DefaultParamsWriter$.saveMetadata(ReadWrite.scala:413)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.$anonfun$saveImpl$1(Pipeline.scala:250)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.$anonfun$saveImpl$1$adapted(Pipeline.scala:247)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.Pipeline$SharedReadWrite$.saveImpl(Pipeline.scala:247)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.saveImpl(Pipeline.scala:346)
	at org.apache.spark.ml.util.MLWriter.save(ReadWrite.scala:168)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.super$save(Pipeline.scala:344)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$4(Pipeline.scala:344)
	at org.apache.spark.ml.MLEvents.withSaveInstanceEvent(events.scala:174)
	at org.apache.spark.ml.MLEvents.withSaveInstanceEvent$(events.scala:169)
	at org.apache.spark.ml.util.Instrumentation.withSaveInstanceEvent(Instrumentation.scala:42)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$3(Pipeline.scala:344)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.$anonfun$save$3$adapted(Pipeline.scala:344)
	at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
	at scala.util.Try$.apply(Try.scala:213)
	at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
	at org.apache.spark.ml.PipelineModel$PipelineModelWriter.save(Pipeline.scala:344)
	at ai.com.spark.jobs.TrainIsolationForest$.saveEndToEndPipeline(TrainIsolationForest.scala:270)
	at ai.com.spark.jobs.TrainIsolationForest$.trainAndSave(TrainIsolationForest.scala:236)
	at ai.com.spark.jobs.TrainIsolationForest$.main(TrainIsolationForest.scala:80)
	at ai.com.spark.jobs.TrainIsolationForest.main(TrainIsolationForest.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:966)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:191)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:214)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1054)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1063)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It seems like the empty directory markers are what's causing the issue:

aws s3 ls --recursive --endpoint http://localhost:9878 s3://bucket/anomaly_detection/
2024-08-08 12:07:35          0 anomaly_detection/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/
2024-08-08 12:07:35          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/metadata/_temporary/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/metadata/
2024-08-08 12:07:39          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/0_emptystringfiller_1be2e8dde774/metadata/_temporary/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/
2024-08-08 12:07:45          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/data/
2024-08-08 12:07:45          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/data/_temporary/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/metadata/
2024-08-08 12:07:42          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/1_strIdx_b35c52fd17a7/metadata/_temporary/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/
2024-08-08 12:07:49          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/data/
2024-08-08 12:07:49          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/data/_temporary/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/metadata/
2024-08-08 12:07:47          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/2_oneHotEncoder_2fe087f49c18/metadata/_temporary/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/metadata/
2024-08-08 12:07:52          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/3_vecAssembler_04a638b48e31/metadata/_temporary/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/
2024-08-08 12:07:57          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/data/
2024-08-08 12:07:57          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/data/_temporary/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/metadata/
2024-08-08 12:07:54          0 anomaly_detection/isolation_forest/org=048e86ed-9f47-46bb-9303-1a93730abfe0/model=4a25cffe-4ccf-4ec2-b758-b138bc699359/isolation_forest_pipeline/stages/4_isolation-forest_dfdfa72ad592/metadata/_temporary/

[adoroszlai]

@mcgrawia A couple of S3A compatibility issues were fixed since 1.4.0, please see HDDS-8423. Especially HDDS-10570 and HDDS-10572 seem to be related to your woes. Can you please try building Ozone from source (master branch), and see if it works better?

[mcgrawia]

Hi @adoroszlai thank you for the quick response. I can try building latest and see if that resolves the issues. Is there a target date for the 1.5.0 release (or maybe a 1.4.1 that has these fixes)? Thanks

[tanvipenumudy]

Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

The IndexOutOfBoundsException while listing OBS buckets I believe has been fixed by HDDS-10399 in Ozone 1.4.1 and 1.5.0.

A tentative workaround is to disable shallow listing by setting the property ozone.s3g.list-keys.shallow.enabled to false.

cc: @SaketaChalamchala, @kerneltime

[mcgrawia]

hi @tanvipenumudy,

I tried recreating my bucket with setting the ozone.s3g.list-keys.shallow.enabled setting to false, but I'm still seeing the same error in the S3G pod.

Bucket info:

ozone sh bucket info /vol1/bucket
2024-08-08 19:28:33,988 [main] WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(60)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
{
  "metadata" : { },
  "volumeName" : "vol1",
  "name" : "bucket",
  "storageType" : "DISK",
  "versioning" : false,
  "usedBytes" : 287264445,
  "usedNamespace" : 22,
  "creationTime" : "2024-08-08T19:03:43.332Z",
  "modificationTime" : "2024-08-08T19:03:43.332Z",
  "sourcePathExist" : true,
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "bucketLayout" : "OBJECT_STORE",
  "owner" : "hadoop",
  "link" : false
}

cat /etc/hadoop/ozone-site.xml
<configuration>
<property><name>ozone.s3g.domain.name</name><value>s3g.internal</value></property>
<property><name>ozone.s3g.list-keys.shallow.enabled</name><value>false</value></property>
<property><name>ozone.scm.block.client.address</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>hdds.datanode.dir</name><value>/data/storage</value></property>
<property><name>ozone.scm.client.address</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>hdds.scm.safemode.min.datanode</name><value>3</value></property>
<property><name>ozone.datanode.pipeline.limit</name><value>1</value></property>
<property><name>ozone.scm.datanode.id.dir</name><value>/data</value></property>
<property><name>ozone.metadata.dirs</name><value>/data/metadata</value></property>
<property><name>ozone.scm.names</name><value>ozone-scm-0.ozone-scm-headless</value></property>
<property><name>dfs.datanode.use.datanode.hostname</name><value>true</value></property>
<property><name>ozone.om.address</name><value>ozone-om-0.ozone-om-headless</value></property>
</configuration>

Error logs:

2024-08-08 19:22:28,707 [qtp313082880-160] WARN  server.HttpChannel (HttpChannel.java:handleException(776)) - /bucket/
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.servlet.ServletException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
	at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
	at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1665)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	... 17 more
Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70)
	at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248)
	at java.base/java.util.Objects.checkIndex(Objects.java:372)
	at java.base/java.util.ArrayList.remove(ArrayList.java:536)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextShallowListOfKeys(OzoneBucket.java:1244)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.getNextListOfKeys(OzoneBucket.java:1150)
	at org.apache.hadoop.ozone.client.OzoneBucket$KeyIterator.hasNext(OzoneBucket.java:1124)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.get(BucketEndpoint.java:212)
	at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
	at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
	... 51 more

[yazynin]

Hello @mcgrawia, maybe this instruction help you https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/ozone-storing-data/topics/ozone-config-spark-s3a.html

[mcgrawia]

Hi @yazynin, thanks for sending but I had already tried those settings.