feat: add APPLY operator representation

[ashvina]

This PR adds a new relation type for APPLY operation. APPLY is similar to JOIN
operator since it joins two table sources. However, in APPLY, the table source
on the right depends on the result of computation of the table source on left.
Since the right source cannot be computed without left, the right source is
represented by an Expression in Substrait and not a relation. The right source
will most frequently be a SubQuery.

Closes #357

[CLAassistant]

All committers have signed the CLA.

[jacques-n]

@ashvina , thanks for proposing this. It seems like we need an additional subquery type to support this pattern, no?

@hannes, I your team worked extensively on these patterns. Do you have a thought of how we should express this?

[jacques-n]

Following up here @ashvina , did you see my question about the need for an additional subquery type?

[ashvina]

Hi @jacques-n. Sorry, I missed your question earlier.

I agree, to properly support APPLY, additional subquery types will be needed. I have added examples queries in substrait-java/pull/106. In the examples, the correlated-right-subquery (FROM clause) produce a rel. A subquery_type for this rel is needed. I think the new type would be similar to subquery:in_predicate, as it returns a rel.

Although, I think extension of subquery expression is independent of this PR and is needed.

Assuming a new subquery type which produces a rel is defined, what do you think about specification of the APPLY?

Note, the APPLY is generally used with table-value-functions. IIUC, TVF spec in missing. Also, it is tricky to get java unit-test to work with TVF. Hence I have omitted TVF query examples. I believe the proposed APPLY spec should be able to support TVF whenever its expression spec is added.

[elharo]

Why lower case Apply here and APPLY for the next two?

[westonpace]

This has been on my "to do" list for a while but I finally got around to looking at it because some behavior like cross apply is being discussed in Arrow (apache/arrow#14682).

@jacques-n I am curious if this really is a subquery. @ashvina I am also uncertain about expressing the right side as an "expression". That being said, I am very much a novice here :)

Correlated subqueries traditionally return a single value (maybe a single row if multiple aggregates) for each input row:

• WHERE X IN ... (returns a single bool for each row)
• WHERE X < ANY(...) (returns a single bool for each row)
• SELECT X, (SUM(...) FROM ...) (returns a single int64 sum for each row)

However, the right side in a cross apply could return a complete table for each input row. Or it could (I think) return nothing at all.

That being said, I may be significantly misunderstanding any of these points and would appreciate any clarification.

[westonpace]

@ashvina is this still in progress?

[jacques-n]

No progress has been made in more than six months. Closing without prejudice.