some speed issues

[onedollor]

I have a string array has about 300k items => col_values

and defined a dbldatagen DataGenerator

df_spec = {
generator
.withColumn("sample", "string", values=col_values)
}

df = df_spec.build()

but it takes about 7 minutes to generate data, partitions=4.