[SPARK-54342][BUILD] Unify scalatest format reports between SBT and maven builds

[vrozov]

What changes were proposed in this pull request?

1. Update SBT scalatest settings to use named file reporter (same as maven scalatest plugin)
2. Update SBT scalatest settings to use the same directories for junit-style xml files as maven scalatest plugin
3. Use without color (W), show all durations (D), show full stack traces (F), show reminder of failed and canceled tests without stack traces (I) for the named file scalatest reporter both in SBT and maven.

Why are the changes needed?

Unify scalatest settings between SBT and maven. Produce SparkTestSuite.txt in the following format:

Run starting. Expected test count is: 15
ExecutorSuite:
- SPARK-15963: Catch `TaskKilledException` correctly in Executor.TaskRunner (1 second, 768 milliseconds)
- SPARK-19276: Handle FetchFailedExceptions that are hidden by user exceptions (938 milliseconds)
- Executor's worker threads should be UninterruptibleThread (380 milliseconds)
- SPARK-19276: OOMs correctly handled with a FetchFailure (91 milliseconds)
- SPARK-23816: interrupts are not masked by a FetchFailure (78 milliseconds)
- Gracefully handle error in task deserialization (15 milliseconds)
- Heartbeat should drop zero accumulator updates (248 milliseconds)
- Heartbeat should not drop zero accumulator updates when the conf is disabled (68 milliseconds)
- SPARK-39696: Using accumulators should not cause heartbeat to fail (1 second, 58 milliseconds)
- Send task executor metrics in DirectTaskResult (114 milliseconds)
- Send task executor metrics in TaskKilled (108 milliseconds)
- Send task executor metrics in ExceptionFailure (93 milliseconds)
- SPARK-34949: do not re-register BlockManager when executor is shutting down (67 milliseconds)
- SPARK-33587: isFatalError *** FAILED *** (2 milliseconds)
  false equaled false (ExecutorSuite.scala:508)
  org.scalatest.exceptions.TestFailedException:
  at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
  at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
  at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
  at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
  at org.apache.spark.executor.ExecutorSuite.testThrowable$1(ExecutorSuite.scala:508)
  at org.apache.spark.executor.ExecutorSuite.$anonfun$new$38(ExecutorSuite.scala:528)
  at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:256)
  at org.apache.spark.executor.ExecutorSuite.$anonfun$new$33(ExecutorSuite.scala:527)
  at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
  at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
  at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
  at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
  at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
  at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:68)
  at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:154)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
  at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:226)
  at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
  at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
  at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
  at org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
  at org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
  at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:68)
  at org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:234)
  at org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:227)
  at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:68)
  at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTests$1(AnyFunSuiteLike.scala:269)
  at org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:413)
  at scala.collection.immutable.List.foreach(List.scala:323)
  at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
  at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
  at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
  at org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:269)
  at org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:268)
  at org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1564)
  at org.scalatest.Suite.run(Suite.scala:1114)
  at org.scalatest.Suite.run$(Suite.scala:1096)
  at org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1564)
  at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$run$1(AnyFunSuiteLike.scala:273)
  at org.scalatest.SuperEngine.runImpl(Engine.scala:535)
  at org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:273)
  at org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:272)
  at org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:68)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
  at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
  at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
  at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:68)
  at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:47)
  at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13(Runner.scala:1321)
  at org.scalatest.tools.Runner$.$anonfun$doRunRunRunDaDoRunRun$13$adapted(Runner.scala:1315)
  at scala.collection.immutable.List.foreach(List.scala:323)
  at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1315)
  at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
  at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
  at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
  at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:970)
  at org.scalatest.tools.Runner$.main(Runner.scala:775)
  at org.scalatest.tools.Runner.main(Runner.scala)
- SPARK-40235: updateDependencies is interruptible when waiting on lock (18 milliseconds)
Run completed in 6 seconds, 86 milliseconds.
Total number of tests run: 15
Suites: completed 1, aborted 0
Tests: succeeded 14, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
ExecutorSuite:

- SPARK-33587: isFatalError *** FAILED *** (2 milliseconds)
  false equaled false (ExecutorSuite.scala:508)

Does this PR introduce any user-facing change?

No

How was this patch tested?

Run test from maven and sbt

Was this patch authored or co-authored using generative AI tooling?

No