Modified the output to specify the mode as overwrite.

rwilson-ebi · rwilson-ebi · commit 61f1ee8c1184 · 2025-09-26T10:23:22.000+01:00
diff --git a/impc_etl/jobs/load/impc_web_api/impc_datasets_metadata_mapper.py b/impc_etl/jobs/load/impc_web_api/impc_datasets_metadata_mapper.py
@@ -436,4 +436,4 @@ def impc_datasets_metadata_mapper():
         )
     stats_results_df.distinct().repartition(1000).write.option(
         "ignoreNullFields", "false"
-    ).json(datasets_metadata_service_output_asset.uri)
+    ).json(datasets_metadata_service_output_asset.uri, mode="overwrite")

Original file line number	Diff line number	Diff line change
`@@ -436,4 +436,4 @@ def impc_datasets_metadata_mapper():`
`436`	`436`	`)`
`437`	`437`	`stats_results_df.distinct().repartition(1000).write.option(`
`438`	`438`	`"ignoreNullFields", "false"`
`439`		`- ).json(datasets_metadata_service_output_asset.uri)`
	`439`	`+ ).json(datasets_metadata_service_output_asset.uri, mode="overwrite")`