BUG: Get element from series of dtypes.value_counts() by index don't work most of times

[KizhiFox]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
df = pd.read_csv('https://stepik.org/media/attachments/course/4852/titanic.csv')
dt = df.dtypes.value_counts()
print(dt)
print(dt.index)
print(dt['float64'])
print(dt['object'])
print(dt['int64'])

Issue Description

I'm trying to get the number of columns of a particular data type. This code should work correctly, but most of times in crashed with KeyError on different keys. For example:

int64      5
object     5
float64    2
dtype: int64
Index([int64, object, float64], dtype='object')
2
5
Traceback (most recent call last):
  File "/home/kizhifox/coding/stepik/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3361, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 76, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'int64'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/kizhifox/coding/stepik/test.py", line 8, in <module>
    print(dt['int64'])
  File "/home/kizhifox/coding/stepik/venv/lib/python3.8/site-packages/pandas/core/series.py", line 942, in __getitem__
    return self._get_value(key)
  File "/home/kizhifox/coding/stepik/venv/lib/python3.8/site-packages/pandas/core/series.py", line 1051, in _get_value
    loc = self.index.get_loc(label)
  File "/home/kizhifox/coding/stepik/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3363, in get_loc
    raise KeyError(key) from err
KeyError: 'int64'

Process finished with exit code 1

I have also tried downloading the CSV file and using dt.get() to access the values, but that also didn't work.

Expected Behavior

Expected output (approximately every 10th launch is displayed):

int64      5
object     5
float64    2
dtype: int64
Index([int64, object, float64], dtype='object')
2
5
5

Installed Versions

INSTALLED VERSIONS

commit : 73c6825
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.11.0-34-generic
Version : #36~20.04.1-Ubuntu SMP Fri Aug 27 08:06:32 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : ru_RU.UTF-8
LOCALE : ru_RU.UTF-8

pandas : 1.3.3
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 20.0.2
setuptools : 44.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.27.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[asishm]

it's strange that some of those even work considering the values in the index are not strings but of type np.dtype.

In [34]: dt.index.tolist()
Out[34]: [dtype('int64'), dtype('O'), dtype('float64')]

In [35]: dt[np.dtype('float64')]
Out[35]: 2

In [36]: dt[np.dtype('int64')]
Out[36]: 5

In [37]: dt[np.dtype('object')]
Out[37]: 5

[KizhiFox]

@asishm dt[np.dtype('float64')] works, thank you. That was mistake in my code.

So, can it be considered a problem that dt['float64'] sometimes works, although it shouldn't? In this case, the above Pandas behavior is still a bug, albeit for the opposite reason.

I tried to calculate how often it works by running this program. It turned out that either it works every time, or never.

count = 0
for i in range(1000):
    try:
        import pandas as pd
        df = pd.read_csv('titanic.csv')
        dt = df.dtypes.value_counts()
        dt['float64']
        count += 1
    except:
        pass
print(count)

Then I made a bash script that runs the python program and counts how often it happens. In about 47.7% of cases, dt['float64'] worked without crashing.

#!/bin/bash
count=0
for i in {1..1000}
do
  ./venv/bin/python ./test.py 2> /dev/null
  count=$((count + $?))
done
echo $count

import pandas as pd
df = pd.read_csv('titanic.csv')
dt = df.dtypes.value_counts()
dt['float64']

[mroeschke]

Do you mind posting a more minimal example? Ideally the behavior should be reproducible without a CSV download so we can create a unit test if there's an issue: https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[asishm]

@mroeschke I can't get a consistent reproducer - but run this script a few times

import pandas as pd

df = pd.DataFrame([[1, 'a', 1.0]])
dt = df.dtypes.value_counts()
fail = []
dtype_list = ['float64', 'int64', 'object']
for i in dtype_list:
    try:
        dt[i]
    except KeyError:
        fail.append(i)
print(fail)

output on WSL

$ python inconsistent_index.py
['float64', 'int64', 'object']
$ python inconsistent_index.py
['float64', 'int64']
$ python inconsistent_index.py
[]
$ python inconsistent_index.py
['object']
$ python inconsistent_index.py
['int64', 'object']
$ python inconsistent_index.py
['float64']
$ python inconsistent_index.py
['float64', 'int64']
$ python inconsistent_index.py
['float64', 'object']
$ python inconsistent_index.py
['float64', 'object']
$ python inconsistent_index.py
[]
$ python inconsistent_index.py
['float64', 'int64', 'object']
$ python inconsistent_index.py
['float64', 'int64', 'object']