Reduce type resolution complexity

[ezorita]

Problem

We are working on a medium-size project (our schema.prisma is < 400 lines) that generates a types.py file over 40.000 lines. This file is so big that Pylance refuses to process it, and hence we don't have type resolution while coding:

Code is too complex to analyze; reduce complexity by refactoring into subroutines or reducing conditional code paths

Suggested solution

I suggest splitting the types file into multiple files, e.g. one per Model.

Alternatives

Suggest Pylance settings that would allow processing such large file (I could not find any setting to fix it).

Additional context

[Pwuts]

The maintainer of Pyright (the underlying type checker that powers Pylance) just posted a detailed explanation of why Pylance and Pyright can't handle these files: microsoft/pyright#9709 (comment)

Highlights

The code within this file's global (module) scope is exceeding pyright's internal threshold for cyclomatic complexity and statement count. Beyond this threshold, pyright will refuse to perform any sort of analysis on the code flow graph.

This limit wouldn't be an issue if the file contained only class definitions because pyright can evaluate a class definition without performing any code flow analysis. However, the prisma module is defining _AgentGraphCompoundPrimaryKey using the "alternate functional syntax" rather than using a standard class statement and then it's aliasing it to another variable AgentGraphWhereUniqueInput. I'm not sure why this approach was chosen given that most of the TypedDict classes in this file are defined using a class statement. The alternative syntax and the type alias definition are syntactically indistinguishable from a normal variable assignment, so pyright needs to perform code flow analysis to determine the variables' types. That's why you see this problem specifically for AgentGraphWhereUniqueInput but not for other TypedDict classes that are defined using a class statement.

Example of the mentioned "alternative functional syntax"

The "alternative functional syntax" definition of _AgentGraphCompoundPrimaryKey:

_AgentGraphCompoundPrimaryKeyInner = TypedDict(
    '_AgentGraphCompoundPrimaryKeyInner',
    {
        'id': '_str',
        'version': '_int',
    },
    total=True
)

_AgentGraphCompoundPrimaryKey = TypedDict(
    '_AgentGraphCompoundPrimaryKey',
    {
        'graphVersionId': '_AgentGraphCompoundPrimaryKeyInner',
    },
    total=True
)

AgentGraphWhereUniqueInput = _AgentGraphCompoundPrimaryKey

You may want to report the issue to the maintainers of prisma. Since this file is auto-generated, I'm hoping that it's straightforward for them to break it up into a more manageable submodules. Alternately, if they could generate class statements for all of their TypedDicts rather than using the alternate functional syntax, that would mitigate the issue.

Offered solutions

• Define all TypedDict types with class statements rather than the functional syntax _SomeModelCompoundPrimaryKey = TypedDict(...)
• Break up types.py into multiple files

[ezorita]

Excellent point @Pwuts. I think the class statement definition is easier to handle. Updating the templates to render classes should be easier than breaking the code down into different files and handling the imports dynamically.

[ezorita]

I did a local test, updated the types.py template to generate all valid class name(TypeDict, total=...). Found the following issues:

• PyRight still complains the file is too complex.
• There are cases in which the dict key is a reserved python word (like not), that can only be resolved with functional syntax.

[ezorita]

The other possible solution is to split the types.py into multiple files. I bet this is possible but quite challenging. In a naive approach I created two templates:

• one for recursive types and filter type definitions
• the other to generate all types associated with a specific model

By modifying slightly the template generator I managed to generate one {model.name}_types.py file for each model.

Once each model lives in a different file, we need to resolve the dependencies among them (e.g. model A relies on types from model B if it has a foreign relation). Models that rely on each other would cause a circular import. To avoid this, we'd need to split files considering a second dimension that avoids circular imports (e.g. at least separate the *WithoutRelations* definitions in different files that are in turn imported).

[Pwuts]

Models that rely on each other would cause a circular import. To avoid this, we'd need to split files considering a second dimension that avoids circular imports

Or do the imports at a different level, e.g. inside the class that uses it?

In any case, it seems like we're hitting the limits of generating Python modules with jinja templates.

[RobertCraigie]

In any case, it seems like we're hitting the limits of generating Python modules with jinja templates.

yeah I've realised this was a problem so I started moving some of the type generation logic out of the templates directly, e.g.

prisma-client-py/src/prisma/generator/schema.py

Line 105 in b12ab3a

def where_unique(self) -> PrismaType:

it seems like we'd have to extend this to support imports as well and then move basically all of the types.py generation logic out to be able to split it up into multiple files...

PRs more than welcome! I don't think I'll personally have time to figure this out unfortunately

[Pwuts]

@RobertCraigie I'm happy to put in the work, but might need a bit of help getting started. Could you point me to e.g. the top 5 files that would need to be reworked and a general idea of what needs to happen?