[Question] thoughts on validations?

[danhosanee]

Hi @mitchelllisle,

Wasn't too sure if this is out of scope for this project.

But was thinking along the lines of adding validations for users to be able to validate pyspark DataFrame against a SparkModel's schema.

Useful in validating extraction from upstream sources or performing downstream transformations (eg: does it cause nullability etc.)

Cheers Dan

[rjurney]

Sounds like a great idea!

[mitchelllisle]

Hi @danhosanee
Apologies for the delay in responding. I don't mind this idea. Did you have an initial idea about how you might implement this? Happy for you to suggest some approaches