[venkata91]

Summary

• Iceberg reports write counts and sizes per commit, but has no timer for how long tasks spent writing, so write throughput can't be derived from Iceberg metrics alone.

Changes

• Add writeDuration (time in row-level write calls) and writeCloseDuration (time closing writers, where buffered rows flush and files are finished). Nanoseconds.
• Both travel to the driver in the commit messages. Spark calls currentMetricsValues() before commit(), so the close, which happens inside commit(), can't be a CustomTaskMetric.
• Aggregate renders as total (min, med, max), matching MetricUtils.stringValue and scanDuration in Spark 4.1: Add per-task scanDuration metric #17562.
• Covers all three write families: SparkWrite (append, overwrite, copy-on-write, streaming), SparkPositionDeltaWrite (merge-on-read), and SparkPositionDeletesRewrite (DeleteWriter and DVWriter).

Testing done

• TestSparkWriteMetrics (7): unpartitioned and partitioned inserts, copy-on-write delete, merge-on-read delete and update, multi-task aggregation.
• TestRewritePositionDeleteFilesMetrics: rewrite path for v2 (DeleteWriter) and v3 (DVWriter).
• TestWriteTimer and TestNanoDurationMetric: phase accumulation, timing on throw, rendering, locale independence.
• Assertions discriminate: disabling either timer fails 7 of 9 tests in TestSparkWriteMetrics.
• Regression sweep over copy-on-write and merge-on-read deletes/updates, position deletes, streaming reads, and DataFrame writes: 0 failures. spotlessCheck and checkstyle pass.

Notes

• Scoped to spark/v4.1. v4.0 is near-identical and can follow.
• v3.5 has no write custom metrics wired at all, so it would need the driver-metrics plumbing backported first.

[venkata91]

@nastra @singhpk234 @manuzhang mind taking a look when you get a chance?

This adds writeDuration and writeCloseDuration task metrics, the write-side counterpart to #17562, extending the metrics UI work from #15104. Both durations travel to the driver in the commit messages, since Spark calls currentMetricsValues() before commit().

It sits on top of #17562, so that one is the better starting point if you are reviewing both.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[venkata91]

@nastra now that #17562 is in, could you take a look at this one? It adds per-task write duration metrics for Spark 4.1. It's rebased on main and CI is green.