BUG: pd.api.types.infer_dtype on scalar input

[gnotisauton]

Context

I was trying to identify data types in columns with mixed data types:

df.map(pd.api.types.infer_dtype)

Pandas version checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandas.
• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.api.types.infer_dtype(1)
pd.api.types.infer_dtype(1.0)
pd.api.types.infer_dtype(True)

Issue Description

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[8], line 1
----> 1 pd.api.types.infer_dtype(1)

File lib.pyx:1605, in pandas._libs.lib.infer_dtype()

TypeError: 'int' object is not iterable

Expected Behavior

According to the documentation, pd.api.types.infer_dtype() should accept scalar input.

Installed Versions

INSTALLED VERSIONS
------------------
commit                : 0691c5cf90477d3503834d983f69350f250a6ff7
python                : 3.12.5
python-bits           : 64
OS                    : Windows
OS-release            : 11
Version               : 10.0.26100
machine               : AMD64
processor             : Intel64 Family 6 Model 154 Stepping 4, GenuineIntel
byteorder             : little
LC_ALL                : None
LANG                  : None
LOCALE                : English_United States.1252

pandas                : 2.2.3
numpy                 : 2.1.0
pytz                  : 2024.1
dateutil              : 2.9.0
pip                   : 24.2
Cython                : None
sphinx                : None
IPython               : 8.32.0
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
blosc                 : None
bottleneck            : None
dataframe-api-compat  : None
fastparquet           : None
fsspec                : None
html5lib              : None
hypothesis            : None
gcsfs                 : None
jinja2                : 3.1.4
lxml.etree            : None
matplotlib            : 3.9.2
numba                 : None
numexpr               : None
odfpy                 : None
openpyxl              : 3.1.5
pandas_gbq            : None
psycopg2              : None
pymysql               : None
pyarrow               : None
pyreadstat            : None
pytest                : None
python-calamine       : None
pyxlsb                : None
s3fs                  : None
scipy                 : 1.14.1
sqlalchemy            : 2.0.37
tables                : None
tabulate              : None
xarray                : None
xlrd                  : None
xlsxwriter            : None
zstandard             : 0.23.0
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None

[gnotisauton]

I've tried looking into the function definition of lib.pyx but couldn't find anything immediately obvious that pointed to a cause.

[gm-oo9]

take

[rhshadrach]

This goes back to da0523a#diff-40a8d0cc4a6796116f539b1e47d5f56dfc1d061e9563d2c927dbc8bd178df8f3. There the docstring added looks incorrect, it is clear the function never attempted to support scalars, although it is very easy to read:

if not isinstance(value, list):
    value = list(value)

as wrapping a scalar (but it does not). I think we should fix the documentation here.

[gm-oo9]

@rhshadrach Alright. I initially thought that replacing list(value) with [value] would allow the function to support scalars, which might work for scalar too. What do you think? If you prefer, we could update the tests and documentation to reflect scalar support.

[gnotisauton]

All right. Too bad, I was hoping I could use

df.map(pd.api.types.infer_dtype).unique()

or

df.map(pd.api.types.infer_dtype).value_counts()

to get a sense of the data type mix in dirty columns for a wide data set all at once. If vectorisation is not an option, I'll have to do it with a for loop.

[rhshadrach]

@gnotisauton - using map with a callable is not vectorized. pandas is just doing a for loop internally. In any case,

df.map(lambda x: pd.api.types.infer_dtype([x])).unique()

is a workaround.

@gm-oo9

Alright. I initially thought that replacing list(value) with [value] would allow the function to support scalars

No - that would break existing use cases when value is an iterable. We would need to infer whether the passed object is iterable (but not a string!) to determine whether we want to do list(value) or [value]. While this can be done, I don't think we should increase the scope of this function unless there is a compelling use case.

[gnotisauton]

using map with a callable is not vectorized. pandas is just doing a for loop internally. In any case,

df.map(lambda x: pd.api.types.infer_dtype([x])).unique()

is a workaround.

Just in case anyone finds this thread and wants to use this workaround: it doesn't work out of the box for imported data. Values in a mixed column are returned as strings, e.g.

df = pd.DataFrame({'a':[1.0, '1,0',10]})
df.map(lambda v: pd.api.types.infer_dtype([v]))

works (data types appropriate), but

# exporting and re-importing the above dataframe
df.to_csv(some_path, index=False)
df = pd.read_csv(some_path)
df.map(lambda v: pd.api.types.infer_dtype([v]))

does not (data types all 'string')

[rhshadrach]

it doesn't work out of the box for imported data. Values in a mixed column are returned as strings, e.g.

Not sure I understand. I get floating, string, integer for the example you posted. Is that not what is desired?

Edit: I think I see, your issue is when you read from csv, you get all strings. That is the correct answer - the data in that DataFrame is a string, and infer_dtype reports it as so. The question of whether it can be converted to another dtype is different, and I think not well-defined.