Hive metastore 4.0.1 remove deprecated thrift APIs

[mattheusv]

Apache Iceberg version

0.7.1

Please describe the bug 🐞

Starting at version 4.0.1, Hive metastore removed deprecated thrift APIs that py-iceberg is currently using. When trying to create a table with catalog.create_table_transaction using Hive metastore 4.0.1, py-iceberg raise an unexpected thrift.Thrift.TApplicationException: Invalid method name: 'get_table' error:

  File "iceberg.py", line 102, in create_table
    with self.catalog.create_table_transaction(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 289, in __exit__
    self.commit_transaction()
  File "/usr/local/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 766, in commit_transaction
    self._table._do_commit(  # pylint: disable=W0212
  File "/usr/local/lib/python3.12/site-packages/pyiceberg/table/__init__.py", line 1638, in _do_commit
    response = self.catalog._commit_table(  # pylint: disable=W0212
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyiceberg/catalog/hive.py", line 457, in _commit_table
    hive_table = self._get_hive_table(open_client, database_name, table_name)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pyiceberg/catalog/hive.py", line 331, in _get_hive_table
    return open_client.get_table(dbname=database_name, tbl_name=table_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/hive_metastore/ThriftHiveMetastore.py", line 4242, in get_table
    return self.recv_get_table()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/hive_metastore/ThriftHiveMetastore.py", line 4260, in recv_get_table
    raise x
thrift.Thrift.TApplicationException: Invalid method name: 'get_table'

This error seems related with this Hive PR(apache/hive#3599) that removed the get_table method

[kevinjqliu]

Thanks for reporting this issue!

I know there's an option to set hive.hive2-compatible to be compatible with Hive 2.x
https://py.iceberg.apache.org/configuration/#hive-catalog

Perhaps we need to do something similar for Hive 4.x

Do you know if there's a migration guide on interacting with Hive 4.x?

[kevinjqliu]

Ah we use hive 4.0.0 in the integration tests

iceberg-python/dev/docker-compose-integration.yml

Lines 90 to 92 in 24a0175

	hive:
	build: hive/
	container_name: hive

iceberg-python/dev/hive/Dockerfile

Line 26 in 24a0175

FROM apache/hive:4.0.0

[Fokko]

Thanks for reporting this @mattheusv

To get to the bottom of this:

• I would first suggest bumping the Hive container to 4.0.1
• Maybe regenerate the Hive Thrift files against the latest version, see https://github.com/apache/iceberg-python/tree/main/vendor

Let me add first-good-issue to see if anyone is interested in working on this 👍

[akshayah3]

@Fokko I can look into this

[Fokko]

@akshayah3 That would be great! Let me know if you run into anything!

[rcsmith27]

@akshayah3 Have you had a chance to look at this? I may be able to work on it if not