Covariance/contravariance on bound variables and subclasses

[karolyi]

Hey,

I'm kinda lost with trying to use a UserDict, from different angles. I've been fighting with this for two days now, to no avail. Typing is meta and hard, and I'm starting to think I don't have the mental fortitude for this complexity, to put it mildly. It might well be I'm missing something.

The original code was written for django, I've boiled it down to a minimal reproducible example. See the typing error in line 51:
Code sample in pyright playground

from abc import ABC, abstractmethod
from collections import UserDict
from typing import Sequence, TypeVar

_SeqT = TypeVar(name='_SeqT', bound=Sequence)


class CacheBase(ABC, UserDict[int, _SeqT]):
    'History cache base.'
    stored_type: type[_SeqT]

    @staticmethod
    @abstractmethod
    def cache_getter(idx: int) -> _SeqT: ...

    def __init__(self):
        if not hasattr(self.__class__, 'cache_getter'):
            raise NotImplementedError('No cache_getter set!')
        super().__init__()

    def __getitem__(self, idx: int) -> _SeqT:
        if idx in self.data:
            return self.data[idx]
        try:
            self.data[idx] = self.cache_getter(idx=idx)
            return self.data[idx]
        except KeyError:
            raise KeyError(idx)


class ListCache(CacheBase[list[str]]):
    stored_type = list[str]

    @staticmethod
    def cache_getter(idx: int) -> list[str]:
        return ['a', 'b']


class TupleCache(CacheBase[tuple[str, ...]]):
    stored_type = tuple[str, ...]

    @staticmethod
    def cache_getter(idx: int) -> tuple[str, ...]:
        return ('a', 'b')


class MixedCacheDict(UserDict[str, CacheBase[_SeqT]]):
    def __init__(self):
        super().__init__()
        self.data = {
            'list_cache': ListCache(),
            # 'tuple_cache': TupleCache()
        }

I've tried to play around with covariance/contravariance, in both cases the CacheBase definition started to fail on type checking. Also I couldn't find anything on this, but isn't it so that a bound TypeVar (e.g. _SeqT) is automatically covariant?

[erictraut]

Moving this to a discussion since it's a request for assistance and not a bug report.

[erictraut]

Yeah, it can take a while to wrap your head around variance and type variables. If you're using Python 3.12, I recommend switching to the new type variable syntax. It eliminates some of the most confusing parts about type variables including scoping and variance. For example, you would define CacheBase like this:

class CacheBase[T: Sequence](ABC, UserDict[int, T]): ...

For more details about this new syntax, see PEP 695.

Now, back to your code sample. You are attempting to assign a value of type dict[str, ListCache] to variable data which has a declared type of dict[str, CacheBase[_SeqT]]. Those types are not compatible, so pyright is correct to report this as a type error. The type parameters for dict are invariant, so their corresponding type arguments must match exactly. Since ListCache is not equivalent to CacheBase[_SeqT], this assignment violates the type rules.

I don't know enough about your use case to suggest an alternative approach.

I'm also not familiar with UserDict. It appears that it's largely there for historical (pre-Python 3) reasons to account for the fact that dict was not subclassable at some point in the distant past.

[karolyi]

Thanks for the elaborate response. I'm at 3.10 for now (looking to upgrade soon), so the new syntax is a no-go for now.

The general idea on MixedCacheDict was to either serve a previously stored value, or to use a function (hence staticmethod) to fetch, store and return a new one.

I looked into subclassing dict (MutableMapping, that is), but didn't manage to achieve better results, and also I'd probably need to call class methods (i.e. dict.__setitem__(self, key, value) and dict.__getitem__(self, key)) to retrieve/store values, what looks more ugly. Or I could just write the whole shebang what UserDict already does.

I still don't comprehend the type incompatibility, but will take a couple more reads on what you answered and see if I can figure out a working solution.

I could just have specified MixedCacheDict with

class MixedCacheDict(UserDict[str, TupleCache | ListCache]):

which probably would have worked, but in the original django code, I have about 20 different subclasses of CacheBase, hence my question.