Revert #674 when taking SPARK-31037 and SPARK-31124.
In #674, I re-did the changes from SPARK-31124 / apache/spark#27879 without taking the upstream commits. After the AQE framework was refactored in SPARK-31037 the diff was too large for what we needed.