0.1.1 Create an Apache Iceberg Table

[tusharchou]

You'll need to configure an Iceberg catalog for your data. Iceberg can store data in various formats such as Parquet or Avro. First, you'll set up a local or cloud storage (e.g., S3, GCS) to store the Iceberg tables.

Load a catalog (Hadoop, AWS S3, GCS, etc.)

from pyiceberg.catalog import load_catalog
catalog = load_catalog("my_catalog")

Define the schema for the Iceberg table (based on the structure of your BigQuery dataset)

schema = {
'transaction_hash': 'string',
'signer_account_id': 'string',
'block_timestamp': 'long',
'actions': 'string'
}

Create an Iceberg table for Near transactions

transactions_table = catalog.create_table(
identifier="near.transactions",
schema=schema,
partition_spec=None # Partition if necessary, e.g., by date or block
)

[tusharchou]

local-data-platform/local-data-platform/nyc_yellow_taxi.py

Line 76 in b00fe29

print(len(local_table.scan().to_arrow()))

#26