Hive Pulsar Integration

[sijie]

Is your feature request related to a problem? Please describe.

Hive is a data warehouse software that built on top of Hadoop. It would be great that Pulsar topics can be registered as external tables in Hive, so people can use Hive:

1. dump data from Pulsar to Hive
2. dump data from Hive to Pulsar
3. query Pulsar directory

Describe the solution you'd like

Implement a Hive Pulsar Storage Handler

Describe alternatives you've considered

N/A

Additional context

I am not sure if Pulsar main repo is the right place for hosting this integration. Alternatively, it can go to Hive repo, or any other hosted repo. I am open for the discussions for where to host this integration.

[jerrypeng]

@sijie it would be great if we can host this integration in the Hive repo as it would give Pulsar more visibility, though I am open to developing it first in the Apache Pulsar repo or another repo under Apache Pulsar and then pushing it out to Hive.

[sijie]

@jerrypeng will start with a sub module of pulsar, and get the work done first and then talk to Hive team to push the work there.

[ambition119]

Issue #3399 also involve support Apache Hive, it is similar to kafka-handler in Hive. So it's Hive team or Pulsar team that decides to do this work?

[sijie]

@ambition119 - ah I see. I am working with @VinceShieh on this feature. I forgot that @VinceShieh already created an issue for that. I will probably mark that issue as a duplicate of this one.

[tiankangkan]

We need this feature ...