test: Add rust write and spark read integration test

[lliangyu-lin]

Which issue does this PR close?

• Closes #.

What changes are included in this PR?

• Add spark.rs that tests write in rust client and verify the metadata and data files written can be read in spark
• Added a helper in docker.rs for executing commands in any container. This allows to run sql in spark container.

Are these changes tested?

Yes, spark.rs in integration tests

[liurenjie1024]

Thanks @lliangyu-lin for this pr, but I don't think this is the right direction. The integration tests written in rust is difficult to read and maintain, also this test contains hacky way to call spark using scripts. For integration tests we should move to sqllogictests.

[lliangyu-lin]

@liurenjie1024 Thank you for taking a look!
My current understanding is that, with sqllogictests, we can test spark connect integration with iceberg rust. But I guess my goal is to add some kind of tests making sure the data and metadata created from rust client can be read by iceberg java library (via Spark). I'm not sure if sqllogictest will be able to cover the cross client compatibility, do you have any suggestion to approach this with sqllogictests framework?

[liurenjie1024]

@liurenjie1024 Thank you for taking a look! My current understanding is that, with sqllogictests, we can test spark connect integration with iceberg rust. But I guess my goal is to add some kind of tests making sure the data and metadata created from rust client can be read by iceberg java library (via Spark). I'm not sure if sqllogictest will be able to cover the cross client compatibility, do you have any suggestion to approach this with sqllogictests framework?

When we use spark connect to access spark, it calles iceberg-spark-runtime to access iceberg table, which calls iceberg java library under the hood. So I don't think tests in this pr to call them directly.

[liurenjie1024]

I'll close this pr for now as this is not the right direction to go. Feel free to reopen it and continue discussion if necessary.