Is the ability to synthesize a polymorphic function lost in 3.4.0?

[TomasMikula]

I'm generating a polymorphic function by a macro.

I cannot use a quote, because the number and kinds of type parameters are only known at macro expansion time.

In 3.3.2, I use a refinement of PolyFunction, something like this:

  val parents =
    List(TypeTree.of[Object], TypeTree.of[PolyFunction])

  val clsSym =
    Symbol.newClass(
      Symbol.spliceOwner,
      name = "$anon",
      parents = parents.map(_.tpe),
      decls = cls => List(
        Symbol.newMethod(
          cls,
          "apply",
          // helper to create the type of the apply method
          polyFunApplyMethodType(tParamNames, tParamBounds, vParamNames, vParamTypes, returnType),
        ),
      ),
      selfType = None,
    )

However, in 3.4.0 this stopped working: it gives the following error:

`PolyFunction` marker trait is reserved for compiler generated refinements

Does that mean that the ability to synthesize a polymorphic function is lost, or is there another way?

[nicolasstucki]

Does that mean that the ability to synthesize a polymorphic function is lost, or is there another way?

It should be allowed.

We tightened the checks in the compiler to not allow users to write new PolyPolyFunction { ... } explicitly. It seems that this accidentally identified the macro-generated version as source written. Though we do not want to support '{ new PolyPolyFunction { ... } }.

[nicolasstucki]

@TomasMikula could you provide a small/complete example in which you use this class?

[TomasMikula]

Here's a contrived example that shows my use of PolyFunction that works in 3.3.2 and doesn't work in 3.4.0.

Use

import scala.annotation.experimental

@main
@experimental
def main: Unit =
  val xs: List[String] =
    List("a", "b", "c")

  println(headOption(xs)) // headOption is a macro returning a polymorphic function

Failure in 3.4.0

[error] -- Error: /Users/tomas/tmp/poly-fun-synth/use.scala:9:10 -----------------------
[error]  9 |  println(headOption(xs))
[error]    |          ^^^^^^^^^^
[error]    |`PolyFunction` marker trait is reserved for compiler generated refinements
[error]    |----------------------------------------------------------------------------
[error]    |Inline stack trace
[error]    |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[error]    |This location contains code that was inlined from defns.scala:43
[error] 43 |    List(TypeTree.of[Object], TypeTree.of[PolyFunction])
[error]    |                                                       ^
[error]     ----------------------------------------------------------------------------
[error] one error found
[error] (Compile / compileIncremental) Compilation failed

Definitions

import scala.annotation.experimental
import scala.quoted.*

/** Shorthand for `quotes.reflect`. */
private transparent inline def qr(using Quotes): quotes.reflect.type =
  quotes.reflect

// Not my actual use, as here the number and kinds of type args are statically known.
// But it does show my use of PolyFunction that stopped working.
@experimental
transparent inline def headOption: [A] => List[A] => Option[A] =
  ${ headOptionImpl }

@experimental
private def headOptionImpl(using Quotes): Expr[[A] => List[A] => Option[A]] =
  import quotes.reflect.*

  polyFun(
    tParamNames  = List("A"),
    tParamBounds = _ => List(TypeBounds.empty),
    vParamNames  = List("as"),
    vParamTypes  = tparams => List(TypeRepr.of[List].appliedTo(tparams.head)),
    returnType   = tparams => TypeRepr.of[Option].appliedTo(tparams.head),
    body         = (targs, args) => {
      val List(targ) = targs
      val List(as) = args
      Select.unique(as, "headOption")
    },
  ).asExprOf[[A] => List[A] => Option[A]]

@experimental
private def polyFun(using Quotes)(
  tParamNames: List[String],
  tParamBounds: qr.PolyType => List[qr.TypeBounds],
  vParamNames: List[String],
  vParamTypes: List[qr.TypeRepr] => List[qr.TypeRepr],
  returnType: List[qr.TypeRepr] => qr.TypeRepr,
  body: (List[qr.TypeRepr], List[qr.Term]) => qr.Term,
): qr.Term = {
  import qr.*

  val parents =
    List(TypeTree.of[Object], TypeTree.of[PolyFunction])

  val clsSym =
    Symbol.newClass(
      Symbol.spliceOwner,
      name = "$anon",
      parents = parents.map(_.tpe),
      decls = cls => List(
        Symbol.newMethod(
          cls,
          "apply",
          polyFunApplyMethodType(tParamNames, tParamBounds, vParamNames, vParamTypes, returnType),
        ),
      ),
      selfType = None,
    )

  val applySym =
    clsSym.declaredMethod("apply").head

  val applyDef =
    DefDef(
      applySym,
      argss => {
        val List(targs, args) = argss
        val typeArgs: List[TypeRepr] = targs.map(_.asInstanceOf[TypeTree].tpe)
        val valArgs: List[Term] = args.map(_.asExpr.asTerm)
        Some(body(typeArgs, valArgs))
      },
    )

  val clsDef = ClassDef(clsSym, parents, List(applyDef))
  val newCls = Apply(Select(New(TypeIdent(clsSym)), clsSym.primaryConstructor), args = Nil)

  Block(
    List(clsDef),
    newCls,
  )
}

private def polyFunApplyMethodType(using Quotes)(
  tParamNames: List[String],
  tParamBounds: qr.PolyType => List[qr.TypeBounds],
  vParamNames: List[String],
  vParamTypes: List[qr.TypeRepr] => List[qr.TypeRepr],
  returnType: List[qr.TypeRepr] => qr.TypeRepr,
): qr.PolyType = {
  import qr.*

  extension (pt: PolyType)
    def params(n: Int): List[TypeRepr] =
      List.range(0, n).map(pt.param(_))

  val nTypeParams = tParamNames.length

  PolyType(tParamNames)(
    tParamBounds,
    pt => MethodType(
      vParamNames,
    )(
      mt => vParamTypes(pt.params(nTypeParams)),
      mt => returnType(pt.params(nTypeParams)),
    ),
  )
}

[sjrd]

It should be possible to directly use a Closure node, these days. Like for regular functions.

[TomasMikula]

I am yet unaware of Closure. Will explore, thanks for the suggestion.

[nicolasstucki]

@smarter is the anonymous class encoding still used in the compiler?

[TomasMikula]

@sjrd The Closure constructor takes meth: Term:

trait ClosureModule { this: Closure.type =>

      def apply(meth: Term, tpe: Option[TypeRepr]): Closure

What should be passed as the meth: Term parameter? A method definition (DefDef) is not a term.

[sjrd]

If defSym is the symbol of the DefDef, use Ident(defSym.termRef) or whatever the Quotes API calls these things.

[TomasMikula]

Thanks, makes sense. Here is an implementation of the polyFun from above using Closure that works in 3.4.0.

@experimental
private def polyFun(using Quotes)(
  tParamNames: List[String],
  tParamBounds: qr.PolyType => List[qr.TypeBounds],
  vParamNames: List[String],
  vParamTypes: List[qr.TypeRepr] => List[qr.TypeRepr],
  returnType: List[qr.TypeRepr] => qr.TypeRepr,
  body: (List[qr.TypeRepr], List[qr.Term]) => qr.Term,
): qr.Term = {
  import qr.*

  val methSym =
    Symbol.newMethod(
      Symbol.spliceOwner,
      name = "polyFunImpl",
      tpe = polyFunApplyMethodType(tParamNames, tParamBounds, vParamNames, vParamTypes, returnType)
    )

  val meth =
    DefDef(
      methSym,
      rhsFn = { case List(targTrees, argTrees) =>
        val targs = targTrees.map(_.asInstanceOf[TypeTree].tpe)
        val args  = argTrees.map(_.asInstanceOf[Term])
        Some(body(targs, args))
      },
    )

  Block(
    List(meth),
    Closure(Ident(methSym.termRef), tpe = None)
  )
}

It does not work in 3.3.2: the Closure.apply constructor fails with

MatchError: PolyType(...

[error] -- Error: /Users/tomas/tmp/poly-fun-synth/use.scala:9:10 -----------------------
[error]  9 |  println(headOption(xs))
[error]    |          ^^^^^^^^^^
[error]    |Exception occurred while executing macro expansion.
[error]    |scala.MatchError: PolyType(List(A), List(TypeBounds(TypeRef(ThisType(TypeRef(NoPrefix,module class scala)),class Nothing),TypeRef(ThisType(TypeRef(NoPrefix,module class scala)),class Any))), MethodType(List(as), List(AppliedType(TypeRef(TermRef(ThisType(TypeRef(NoPrefix,module class collection)),object immutable),List),List(TypeParamRef(A)))), AppliedType(TypeRef(TermRef(ThisType(TypeRef(NoPrefix,module class )),object scala),Option),List(TypeParamRef(A))))) (of class dotty.tools.dotc.core.Types$PolyType)
[error]    |	at dotty.tools.dotc.core.Types$Type.toFunctionType(Types.scala:1881)
[error]    |	at dotty.tools.dotc.typer.TypeAssigner.assignType(TypeAssigner.scala:404)
[error]    |	at dotty.tools.dotc.typer.TypeAssigner.assignType$(TypeAssigner.scala:16)
[error]    |	at dotty.tools.dotc.typer.Typer.assignType(Typer.scala:117)
[error]    |	at dotty.tools.dotc.ast.tpd$.Closure(tpd.scala:108)
[error]    |	at scala.quoted.runtime.impl.QuotesImpl.scala$quoted$runtime$impl$QuotesImpl$reflect$Closure$$$_$apply$$anonfun$21(QuotesImpl.scala:820)
[error]    |	at scala.quoted.runtime.impl.QuotesImpl$reflect$.scala$quoted$runtime$impl$QuotesImpl$reflect$$$withDefaultPos(QuotesImpl.scala:3002)
[error]    |	at scala.quoted.runtime.impl.QuotesImpl$reflect$Closure$.apply(QuotesImpl.scala:820)
[error]    |	at scala.quoted.runtime.impl.QuotesImpl$reflect$Closure$.apply(QuotesImpl.scala:819)
[error]    |	at defns$package$.polyFun(defns.scala:62)
[error]    |	at defns$package$.headOptionImpl(defns.scala:28)
[error]    |	at defns$package$.inline$headOptionImpl(defns.scala:15)
[error]    |
[error]    |----------------------------------------------------------------------------
[error]    |Inline stack trace
[error]    |- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[error]    |This location contains code that was inlined from defns.scala:12
[error] 12 |  ${ headOptionImpl }
[error]    |  ^^^^^^^^^^^^^^^^^^^
[error]     ----------------------------------------------------------------------------
  

but I can live with that.