Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for JSON datatype #399

Open
jlynchMicron opened this issue Jan 14, 2022 · 10 comments · May be fixed by #1130
Open

Add support for JSON datatype #399

jlynchMicron opened this issue Jan 14, 2022 · 10 comments · May be fixed by #1130
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@jlynchMicron
Copy link

Is your feature request related to a problem? Please describe.
Please add sqlalchemy_bigquery support for the JSON datatype, recently added to pre-GA support (link: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#json_type)

Describe alternatives you've considered
Ive considered just using the JSON data as a String and just doing JSON operations locally.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Jan 14, 2022
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Jan 15, 2022
@tswast tswast added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed triage me I really want to be triaged. labels Jan 18, 2022
@cullenwren95
Copy link

Bumping this, would be great functionality to add.

@jlynchMicron
Copy link
Author

Hi all, now that JSON is out of pre-GA and officially released (I think), could this ticket get bumped in priority for implementation?

@jlynchMicron
Copy link
Author

Since bigquery-sqlalchemy is just an extension of sqlalchemy, does that mean there is already inbuilt support for this datatype due to it being an already existing sqlachemy datatype? https://docs.sqlalchemy.org/en/14/core/type_basics.html#sqlalchemy.types.JSON

@jlynchMicron
Copy link
Author

Nevermind, looks like JSON is not in the sqlalchemy type map. https://github.com/googleapis/python-bigquery-sqlalchemy/blob/main/sqlalchemy_bigquery/_types.py#L32

@sachsbl
Copy link

sachsbl commented Feb 2, 2023

+1 would be great to have this

@sar009
Copy link

sar009 commented Apr 13, 2023

@tswast I would like to contribute here. is there a wiki or a guide on how to get started?

@lambrosopos
Copy link

does anyone know what the current status is for this?

@deepakmali
Copy link

Is this being implemented? Has someone found alternative to this ?
I am querying for a json column and expecting to json value, but getting string instead.

@mephmanx
Copy link

I need this as well

@pykenny
Copy link

pykenny commented Sep 21, 2023

For anyone who want to take a stab on this topic:

There's very little consensus on JSON accessing syntax among various SQL implementations and it's up until 2023 did ISO add specification of JSON column type.

If you're only using SQLAlchemy for managing table schema and perhaps dump full JSON content from queries, then simply adding JSON to type map should probably be fine. However when it comes to internal value accessing, you'll need to instruct SA what to do with the query.

  1. Internal value accessing (path querying)

    In Postgres, the implementation simply applies Postgres' arrow-like syntax with the key, and let Postgres itself to guess if it's accessing an array (integer key) or an object (string key), while in MySQL it has to identify key type to decide which syntax to use (square bracket subscription for arrays, and dot operator for objects). Also, some dialect-specific operators/syntax may exist (e.g. Postgres allows user to check if a JSONB contains some set of keys using ?&, with no equivalent functional call available) other than ones for accessing internal values.

  2. Type conversion

    Unlike STRUCT, JSON doesn't contain any internal schema information, so type conversion should be done to escape JSON type first when comparing accessed value to others applied to binary operators (such as comparison). While user can call CAST explicitly, as far as I know both MySQL and Postgres dialects calls CAST internally depends on the type on the other side.

For BigQuery's case:

  • Value/Path accessing: Using square bracket subscription for accessing both arrays and objects. (Using MySQL's syntax when using JSON_QUERY function to query value) and that's the only operator available for handling JSON type
  • Type conversion: Available BigQuery column types for casting primitive values in JSON: STRING, BOOL, INT64, FLOAT64

SandeepAkode added a commit to SandeepAkode/python-bigquery-sqlalchemy that referenced this issue Oct 30, 2024
@SandeepAkode SandeepAkode linked a pull request Oct 30, 2024 that will close this issue
SandeepAkode added a commit to SandeepAkode/python-bigquery-sqlalchemy that referenced this issue Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants