TST (string dtype): fix sql xfails with using_infer_string #60255

WillAyd · 2024-11-08T20:54:39Z

No description provided.

WillAyd · 2024-11-08T20:55:42Z

pandas/core/internals/construction.py

-                    arr = maybe_infer_to_datetimelike(arr)
-                    if dtype_backend != "numpy" and arr.dtype == np.dtype("O"):
+                    convert_to_nullable_dtype = dtype_backend != "numpy"
+                    arr = maybe_infer_to_datetimelike(arr, convert_to_nullable_dtype)


I was surprised by the fact that maybe_infer_to_datetimelike would change an object dtype to a string datatype; might be a cause of unexpected behavior elsewhere where this is used

Yeah that's a very confusing name for what it does (by accident?) in practice nowadays ..

(not necessarily for now, but long term I think we should directly use maybe_convert_objects here instead of maybe_infer_to_datimelike as a small wrapper around it)

Sounds good - I opened #60258 to keep track of this

WillAyd · 2024-11-08T20:57:00Z

pandas/io/sql.py

@@ -1422,6 +1425,10 @@ def _get_dtype(self, sqltype):
            return date
        elif isinstance(sqltype, Boolean):
            return bool
+        elif isinstance(sqltype, String):
+            if using_string_dtype():
+                return StringDtype(na_value=np.nan)


I hard-coded the nan value here for backwards compatability, but this might become a code smell in the future. Have we been trying to avoid this pattern @jorisvandenbossche or is it expected for now?

Yes, this is the way I also did it elsewhere in code paths that needs to construct the default string dtype (harcode the nan value, and the storage gets determined automatically based on whether pyarrow is installed / based on the pd.options)

But, is this change required? It doesn't seem that this is actually used in the one place where _get_dtype gets used?

It is required - I somehow did not commit the part that uses it in the caller. Stay tuned...

jorisvandenbossche

Thanks for looking into this!

jorisvandenbossche · 2024-11-08T21:52:05Z

pandas/core/internals/construction.py

-                    arr = maybe_infer_to_datetimelike(arr)
-                    if dtype_backend != "numpy" and arr.dtype == np.dtype("O"):
+                    convert_to_nullable_dtype = dtype_backend != "numpy"
+                    arr = maybe_infer_to_datetimelike(arr, convert_to_nullable_dtype)


Yeah that's a very confusing name for what it does (by accident?) in practice nowadays ..

jorisvandenbossche · 2024-11-08T21:53:55Z

pandas/io/sql.py

@@ -1422,6 +1425,10 @@ def _get_dtype(self, sqltype):
            return date
        elif isinstance(sqltype, Boolean):
            return bool
+        elif isinstance(sqltype, String):
+            if using_string_dtype():
+                return StringDtype(na_value=np.nan)


Yes, this is the way I also did it elsewhere in code paths that needs to construct the default string dtype (harcode the nan value, and the storage gets determined automatically based on whether pyarrow is installed / based on the pd.options)

jorisvandenbossche · 2024-11-08T21:56:53Z

pandas/io/sql.py

@@ -1422,6 +1425,10 @@ def _get_dtype(self, sqltype):
            return date
        elif isinstance(sqltype, Boolean):
            return bool
+        elif isinstance(sqltype, String):
+            if using_string_dtype():
+                return StringDtype(na_value=np.nan)


But, is this change required? It doesn't seem that this is actually used in the one place where _get_dtype gets used?

jorisvandenbossche · 2024-11-08T21:57:15Z

pandas/io/sql.py

@@ -2205,7 +2210,7 @@ def read_table(
        elif using_string_dtype():
            from pandas.io._util import arrow_string_types_mapper

-            arrow_string_types_mapper()
+            mapping = arrow_string_types_mapper()


jorisvandenbossche · 2024-11-08T22:03:22Z

pandas/core/internals/construction.py

-                    arr = maybe_infer_to_datetimelike(arr)
-                    if dtype_backend != "numpy" and arr.dtype == np.dtype("O"):
+                    convert_to_nullable_dtype = dtype_backend != "numpy"
+                    arr = maybe_infer_to_datetimelike(arr, convert_to_nullable_dtype)


(not necessarily for now, but long term I think we should directly use maybe_convert_objects here instead of maybe_infer_to_datimelike as a small wrapper around it)

WillAyd · 2024-11-09T17:01:38Z

I think the issue with the non-pyarrow future string build has to do with the tests running in parallel. @mroeschke maybe I am misunderstanding, but isn't each module supposed to run all of its tests on one worker? Wonder why this build in particular is the only one that seems to fail

mroeschke · 2024-11-09T18:52:07Z

but isn't each module supposed to run all of its tests on one worker?

I changed the execution method to worksteal a few months ago (test ran much faster that way).

I suspect that possibly not all the fixtures correctly create and clean up the tables they create, and since the ADBC and sqlalchemy connections reference the same table names on the same dbs, the importorskip("pyarrow")s added may have disrupted the table states on the db.

(In an ideal world, teach test should work with a unique table name which is cleaned up appropriately upon failure/success)

WillAyd · 2024-11-09T19:03:12Z

Ah OK interesting...I wonder how that hasn't affected the main CI jobs to date...

I suspect that possibly not all the fixtures correctly create and clean up the tables they create

They do as long as they only operate within a single thread, but with multiple threads there is nothing that prevents the tests from getting a race condition, since a lot of the test table names are hard-coded (although we probably could change them to a uuid)

I'm guessing the fix for now would be to add a pytestmarker for single_cpu to the test_sql.py module (?)

jorisvandenbossche

Looks good!

jorisvandenbossche · 2024-11-11T21:42:45Z

pandas/io/sql.py

+                elif (
+                    using_string_dtype()
+                    and is_string_dtype(col_type)
+                    and is_object_dtype(self.frame[col_name])


Is the check for object dtype of the column needed? (or the string dtype might already have been inferred when constructing the DataFrame?)

And for example a column of all missing values, that might get inferred as float by the DataFrame constructor? In that case maybe rather not is_string_dtype instead of is_object_dtype? (Or just pass a copy=False to astype)

It is needed, although admittedly the patterns used here are not very clear.

Without the is_object_dtype check, it is possible that a series that is constructed as a StringDtype using pd.NA as the na_value sentinel gets changed to using the np.nan sentinel, which fails a good deal of tests

As an alternative, we might be able to infer the na_value to use from the executing environment rather than hard-coding np.nan on L1436?

Maybe it would be clearer then to use if dtype_backend == "numpy" in the if clause? (instead of is_object_dtype) I.e. only do this cast to StringDtype in default backend mode.
That's also what the case just above and below actually use .. (so maybe those could be combined within a single if dtype_backend == "numpy" block)

(this also makes me wonder if we should call _harmonize_columns at all in case of a pyarrow dtype_backend, because at that point the self.frame already has all ArrowDtype columns. But I suppose this method handles some additional things like the parse_dates keyword?)

jorisvandenbossche · 2024-11-12T20:51:09Z

Hmm one more failing test now with test_insertion_method_on_conflict_do_nothing (not directly sure how that is caused by code change, because the wrong dtype is for the float column)

WillAyd commented Nov 8, 2024

View reviewed changes

WillAyd force-pushed the sql-string-xfail branch from 26e42d2 to a884333 Compare November 8, 2024 20:57

jorisvandenbossche reviewed Nov 8, 2024

View reviewed changes

WillAyd force-pushed the sql-string-xfail branch 6 times, most recently from 990bdc7 to 1abf86c Compare November 9, 2024 15:18

WillAyd requested a review from mroeschke as a code owner November 9, 2024 16:30

WillAyd added 2 commits November 9, 2024 14:03

TST (string dtype): fix sql xfails with using_infer_string

97f09f7

Add single_cpu mark

41af90a

WillAyd force-pushed the sql-string-xfail branch from 1f38f1d to 41af90a Compare November 9, 2024 19:05

jorisvandenbossche approved these changes Nov 11, 2024

View reviewed changes

Use dtype_backend instead of is_object_dtype

0dc9e64

jorisvandenbossche approved these changes Nov 12, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST (string dtype): fix sql xfails with using_infer_string #60255

TST (string dtype): fix sql xfails with using_infer_string #60255

WillAyd commented Nov 8, 2024

WillAyd Nov 8, 2024

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

WillAyd Nov 8, 2024

WillAyd Nov 8, 2024

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

WillAyd Nov 8, 2024

jorisvandenbossche left a comment

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

jorisvandenbossche Nov 8, 2024

WillAyd commented Nov 9, 2024

mroeschke commented Nov 9, 2024 •

edited

Loading

WillAyd commented Nov 9, 2024

jorisvandenbossche left a comment

jorisvandenbossche Nov 11, 2024

jorisvandenbossche Nov 11, 2024

WillAyd Nov 11, 2024

jorisvandenbossche Nov 12, 2024

jorisvandenbossche commented Nov 12, 2024

TST (string dtype): fix sql xfails with using_infer_string #60255

Are you sure you want to change the base?

TST (string dtype): fix sql xfails with using_infer_string #60255

Conversation

WillAyd commented Nov 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Nov 9, 2024

mroeschke commented Nov 9, 2024 • edited Loading

WillAyd commented Nov 9, 2024

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Nov 12, 2024

mroeschke commented Nov 9, 2024 •

edited

Loading