Skip to content

Commit 0ffcae5

Browse files
Added connector hub support. Fixes #3 (#4)
1 parent ec26594 commit 0ffcae5

File tree

3 files changed

+85
-13
lines changed

3 files changed

+85
-13
lines changed

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,5 @@
2020

2121
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
2222
hs_err_pid*
23+
target
24+
.okhttpcache

README.md

Lines changed: 44 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,66 @@
1+
12
# Introduction
23

4+
5+
36
# Transformations
47

8+
59
## Archive
610

711
The Archive transformation is used to help preserve all of the data for a message when archived to S3.
812

13+
14+
15+
16+
### Note
17+
18+
This transform works by copying the key, value, topic, and timestamp to new record where this is all contained in the value of the message. This will allow connectors like Confluent's S3 connector to properly archive the record.
19+
20+
921
### Configuration
1022

11-
| Name | Type | Importance | Default Value | Validator | Documentation|
12-
| ---- | ---- | ---------- | ------------- | --------- | -------------|
1323

24+
#### Examples
25+
26+
##### Standalone Example
1427

15-
#### Standalone Example
28+
This configuration is used typically along with [standalone mode](http://docs.confluent.io/current/connect/concepts.html#standalone-workers).
1629

1730
```properties
18-
transforms=Archive
19-
transforms.Archive.type=com.github.jcustenborder.kafka.connect.archive.Archive
20-
# The following values must be configured.
31+
name=Connector1
32+
connector.class=org.apache.kafka.some.SourceConnector
33+
tasks.max=1
34+
transforms=tran
35+
transforms.tran.type=com.github.jcustenborder.kafka.connect.archive.Archive
2136
```
2237

23-
#### Distributed Example
38+
##### Distributed Example
39+
40+
This configuration is used typically along with [distributed mode](http://docs.confluent.io/current/connect/concepts.html#distributed-workers).
41+
Write the following json to `connector.json`, configure all of the required values, and use the command below to
42+
post the configuration to one the distributed connect worker(s).
2443

2544
```json
2645
{
27-
"name": "connector1",
28-
"config": {
29-
"connector.class": "com.github.jcustenborder.kafka.connect.archive.Archive",
30-
"transforms": "Archive",
31-
"transforms.Archive.type": "com.github.jcustenborder.kafka.connect.archive.Archive",
32-
}
46+
"name" : "Connector1",
47+
"connector.class" : "org.apache.kafka.some.SourceConnector",
48+
"transforms" : "tran",
49+
"transforms.tran.type" : "com.github.jcustenborder.kafka.connect.archive.Archive"
3350
}
3451
```
3552

53+
Use curl to post the configuration to one of the Kafka Connect Workers. Change `http://localhost:8083/` the the endpoint of
54+
one of your Kafka Connect worker(s).
55+
56+
Create a new instance.
57+
```bash
58+
curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors
59+
```
60+
61+
Update an existing instance.
62+
```bash
63+
curl -s -X PUT -H 'Content-Type: application/json' --data @connector.json http://localhost:8083/connectors/TestSinkConnector1/config
64+
```
65+
66+

pom.xml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@
2929
<name>kafka-connect-transform-archive</name>
3030
<url>https://github.com/jcustenborder/kafka-connect-transform-archive</url>
3131
<inceptionYear>2017</inceptionYear>
32+
<description>Kafka Connect transformation used to help with archiving data to s3. This transformation copies the timestamp, topic, key, and value to a struct in the value of the record.</description>
3233
<licenses>
3334
<license>
3435
<name>Apache License 2.0</name>
@@ -74,4 +75,42 @@
7475
<scope>test</scope>
7576
</dependency>
7677
</dependencies>
78+
<build>
79+
<plugins>
80+
<plugin>
81+
<groupId>io.confluent</groupId>
82+
<artifactId>kafka-connect-maven-plugin</artifactId>
83+
<version>0.9.0</version>
84+
<executions>
85+
<execution>
86+
<goals>
87+
<goal>kafka-connect</goal>
88+
</goals>
89+
<configuration>
90+
<ownerUsername>jcustenborder</ownerUsername>
91+
<confluentControlCenterIntegration>true
92+
</confluentControlCenterIntegration>
93+
<documentationUrl>
94+
https://jcustenborder.github.io/kafka-connect-documentation/
95+
</documentationUrl>
96+
<ownerName>Jeremy Custenborder</ownerName>
97+
<dockerNamespace>jcustenborder</dockerNamespace>
98+
<dockerName>kafka-connect-docker</dockerName>
99+
<pluginTypes>
100+
<pluginType>transform</pluginType>
101+
</pluginTypes>
102+
<tags>
103+
<tag>Archive</tag>
104+
<tag>S3</tag>
105+
</tags>
106+
<title>Archive Transformation</title>
107+
<supportUrl>${pom.issueManagement.url}</supportUrl>
108+
<supportSummary>Support provided through community involvement.
109+
</supportSummary>
110+
</configuration>
111+
</execution>
112+
</executions>
113+
</plugin>
114+
</plugins>
115+
</build>
77116
</project>

0 commit comments

Comments
 (0)