dependency and how to use the jar

[rimmmmm]

Hi all,
I would like to know whether the new release is compatible with Spark 3 or not, and how to use the jar file of SparkCube
I tried to compile the code
There's a problem with the following dependency it's unavailable

		<groupId>com.swoop</groupId>
        <artifactId>spark-alchemy_2.11</artifactId>
		<version>0.3.28</version>

Failed to collect dependencies at com.swoop:spark-alchemy_2.11:jar:0.3.28: Failed to read artifact descriptor for com.swoop:spark-alchemy_2.11:jar:0.3.28: Could not transfer artifact com.swoop:spark-alchemy_2.11:pom:0.3.28 from/to swoop-inc (https://dl.bintray.com/swoop-inc/maven/): Access denied to: https://dl.bintray.com/swoop-inc/maven/com/swoop/spark-alchemy_2.11/0.3.28/spark-alchemy_2.11-0.3.28.pom , ReasonPhrase:Forbidden

If I change the dependency in pom.xml to
spark-alchemy-test_2.12
1.0.1
I have errors (please see below)

private implicit def cacheIdToTableIdent(cacheIdentifier: CacheIdentifier): TableIdentifier = {
[WARNING]                        ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:20: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.HyperLogLogInitSimpleAgg
[ERROR]                          ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/execution/PreCountDistinctTransformer.scala:44: not found: value HyperLogLogInitSimpleAgg
[ERROR]           HyperLogLogInitSimpleAgg(childExpr, relativeSD)
[ERROR]           ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:22: object spark is not a member of package com.swoop.alchemy
[ERROR] import com.swoop.alchemy.spark.expressions.hll.{HyperLogLogCardinality, HyperLogLogMerge}
[ERROR]                          ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:330: not found: value HyperLogLogCardinality
[ERROR]                   case _: CardinalityAfter => HyperLogLogCardinality(attrs.head)
[ERROR]                                               ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:417: not found: value HyperLogLogCardinality
[ERROR]                   Some(HyperLogLogCardinality(other))
[ERROR]                        ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:509: not found: value HyperLogLogMerge
[ERROR]       HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
[ERROR]       ^
[WARNING] one warning found
[ERROR] 6 errors found

thank you

[adrian-wang]

@rimmmmm you can try spark-alchemy_2.12 instead of spark-alchemy-test_2.12.

[rimmmmm]

Hi Adrian
thanks less errors (5 resolved) still 1 remains with spark-alchemy_2.12 1.1.0 or spark-alchemy_2.12 1.0.1 (check package mvn output below )
any idea how to resolve this ?
it shows a mismatch between alchemy jar and your code GenPlanFromCache.scala line 509

[WARNING]   private implicit def cacheIdToTableIdent(cacheIdentifier: CacheIdentifier): TableIdentifier = {
[WARNING]                        ^
[ERROR] /home/rym/Downloads/SparkCube-0.3.0/src/main/scala/com/alibaba/sparkcube/optimizer/GenPlanFromCache.scala:509: type mismatch;
 found   : Int
 required: com.swoop.alchemy.spark.expressions.hll.Implementation
[ERROR]       HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
[ERROR]                                         ^
[WARNING] one warning found
[ERROR] one error found

best

[rimmmmm]

when I comment approx count Agg fct in GenPlanFromCache.scala , the build succeeds
how to use the jar ? should I put it in spark folder and start spark ?
/*
case "pre_approx_count_distinct" if promo =>
assert(hll.isDefined)
val hllpp = hll.get /hllpp.mutableAggBufferOffset/
HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
case "pre_approx_count_distinct" if !promo =>
assert(hll.isDefined)
val hllpp = hll.get
HyperLogLogPlusPlus(
args.head, hllpp.relativeSD, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset)
// case "pre_approx_count_distinct" if promo => PreApproxCountDistinct(args.head)
// already handled pre_approx_count_distinct outside
*/

[adrian-wang]

when I comment approx count Agg fct in GenPlanFromCache.scala , the build succeeds how to use the jar ? should I put it in spark folder and start spark ? /* case "pre_approx_count_distinct" if promo => assert(hll.isDefined) val hllpp = hll.get /hllpp.mutableAggBufferOffset/ HyperLogLogMerge(args.head, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset) case "pre_approx_count_distinct" if !promo => assert(hll.isDefined) val hllpp = hll.get HyperLogLogPlusPlus( args.head, hllpp.relativeSD, hllpp.mutableAggBufferOffset, hllpp.inputAggBufferOffset) // case "pre_approx_count_distinct" if promo => PreApproxCountDistinct(args.head) // already handled pre_approx_count_distinct outside */

Please refer to https://github.com/alibaba/SparkCube#use-with-apache-spark