Skip to content

Commit 9473118

Browse files
committed
update README and Config Type class in configuration
1 parent b2802cc commit 9473118

File tree

2 files changed

+15
-2
lines changed

2 files changed

+15
-2
lines changed

README.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Pulsar AWS S3 sink
22

3-
Pulsar AWS S3 sink
3+
Pulsar AWS S3 sink receives JSON messages over Pulsar topics with the same schema and saves them as Parquet format on AWS S3.
44

55
## Operations
66
### Deployment
@@ -11,8 +11,17 @@ GET `admin/v2/functions/connectors` displays the nar is loaded successfully as
1111
{"name":"aws-s3","description":"AWS S3 sink","sinkClass":"com.kesque.pulsar.sink.s3.AWSS3Sink"}
1212
```
1313

14+
`EFFECTIVELY_ONCE` processing guarantees is required since it implements message [cumulative acknowledgement](https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarSource.java#L129)
15+
16+
Create a sink by uploading a nar file.
17+
```
18+
$ bin/pulsar-admin sinks create --archive ./connectors/pulsar-io-s3-1.0.nar --inputs aws-s3-input-topic --name aws-s3-test --sink-config-file ./connectors/pulsar-s3-io.yaml --processing-guarantees EFFECTIVELY_ONCE --subs-position Earliest --sub-name auniquename
19+
"Created successfully"
1420
```
15-
$ bin/pulsar-admin sinks create --archive ./connectors/pulsar-io-s3-1.0.nar --inputs aws-s3-input-topic --name aws-s3-test --sink-config-file ./connectors/pulsar-s3-io.yaml --processing-guarantees EFFECTIVELY_ONCE --subs-position Earliest
21+
22+
Create a sink from a preloaded nar file.
23+
```
24+
$ bin/pulsar-admin sinks create --sink-type aws-s3 --inputs aws-s3-input-topic --name aws-s3-test --sink-config-file ./connectors/pulsar-s3-io.yaml --processing-guarantees EFFECTIVELY_ONCE --subs-position Earliest --subs-name auniquename
1625
"Created successfully"
1726
1827
$ bin/pulsar-admin sinks list
@@ -24,6 +33,9 @@ $ bin/pulsar-admin sinks delete --name aws-s3-test
2433
"Deleted successfully"
2534
```
2635

36+
### Topic schema registry
37+
It is mandatory a schema is enforced over the input topics. The Sink would have fatal error to create parquet format when it receives messages with different schemas.
38+
2739
## Build
2840
The command to build a nar file.
2941
```

s3/src/main/resources/META-INF/services/pulsar-io.yaml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,4 @@
2020
name: aws-s3
2121
description: AWS S3 sink
2222
sinkClass: com.kesque.pulsar.sink.s3.AWSS3Sink
23+
sinkConfigClass: com.kesque.pulsar.sink.s3.AWSS3Config

0 commit comments

Comments
 (0)