0.1.1 Insert Data into Iceberg

[tusharchou]

After you have the data from BigQuery and an Iceberg table ready, you can insert the dataset into Iceberg for storage.

Load the Iceberg table

from pyiceberg.table import Table
table = catalog.load_table("near.transactions")

Write data into Iceberg (converting Pandas DataFrame to PySpark DataFrame)

import pyspark
spark = pyspark.sql.SparkSession.builder.appName("IcebergApp").getOrCreate()
spark_df = spark.createDataFrame(transactions_df)

Append the data to the Iceberg table

table.new_append(spark_df).commit()

[tusharchou]

#26

local-data-platform/local-data-platform/nyc_yellow_taxi.py

Line 76 in b00fe29

print(len(local_table.scan().to_arrow()))