Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

ismailsimsek · 2025-01-09T11:09:16Z

resolves #10844
resolves #11914

Copied over kafka-connect-transforms code. no code changes made
applied code formatting
and updated build.gradle accordingly

cc: @bryanck could you please take look at this when you have chance?

ismailsimsek · 2025-01-09T11:44:25Z

kafka-connect/build.gradle

+  dependencies {
+    implementation project(path: ':iceberg-bundled-guava', configuration: 'shadow')
+    implementation libs.bson
+//    implementation libs.slf4j //TODO DISABLED! do we need this??


commented out this. not sure if its needed?

This is a direct dependency so should not be commented out

kafka-connect/build.gradle

ismailsimsek · 2025-01-10T12:02:04Z

@jbonofre @bryanck its ready for review

(cherry picked from commit 639b0d5b41b827d984aae04efe594315ec2b2b91)

(cherry picked from commit 63cde8e7f6c12392c7741922d5e6ad807051f24a)

(cherry picked from commit b9cd15de938e57bf178e1ccb443e481fde881224)

(cherry picked from commit c17c6f734dac96975959e6165416396e4058332c)

(cherry picked from commit 03dcf40b484f40f62c551b9ddf5cefea93a3440a)

(cherry picked from commit d0adaf9f961ceb89aaa408c03874788b3cf2c422)

(cherry picked from commit 5812322e595cee663d920aedaed21998fffa9bdf)

(cherry picked from commit bf82d607dc2b5e816c8b6f59bcbdc48281154e98)

(cherry picked from commit 89f533b2e689cbd1935c3bd1b82eea5e9dc0cd07)

(cherry picked from commit 92e4d984fe41c20faf68b1c36e6fd20759e0a19f)

* smt-nested-json-as-map - parse json objects into Maps rather than Structs prior to handing to the iceberg connector, for users with unstructured json data. (cherry picked from commit 303435aa794d8df1728f83ca5179e896b17ca4ff)

* option-to-inject-kafka-metadata - SMT to add Kafka metadata (topic, partition, offset, timestamp) to Struct and Map types (cherry picked from commit 423b4a8b0f2e42f2dd7de315631e944c285dcb09)

* matf-non-flattening-mongodb-debezium-smt - adds debezium mongo SMT for converting BSON before/after into typed Struct before/after (cherry picked from commit 21d741e53ce77547edbb5838f1b2b49db619be0c)

danielcweeks · 2025-01-15T18:50:27Z

gradle/libs.versions.toml

@@ -100,6 +101,7 @@ avro-avro = { module = "org.apache.avro:avro", version.ref = "avro" }
 awssdk-bom = { module = "software.amazon.awssdk:bom", version.ref = "awssdk-bom" }
 awssdk-s3accessgrants = { module = "software.amazon.s3.accessgrants:aws-s3-accessgrants-java-plugin", version.ref = "awssdk-s3accessgrants" }
 azuresdk-bom = { module = "com.azure:azure-sdk-bom", version.ref = "azuresdk-bom" }
+bson = { module = "org.mongodb:bson", version.ref = "bson-ver"}


We need to verify this addition is captured in LICENSE/NOTICE files.

added section for mongodb bson libs to LICENSE

I think we can remove it until we add the transform library to the runtime. When we do that, then the new license plugin can add what is needed.

danielcweeks · 2025-01-15T18:50:52Z

gradle/libs.versions.toml

@@ -159,6 +161,7 @@ jaxb-runtime = { module = "org.glassfish.jaxb:jaxb-runtime", version.ref = "jaxb
 kafka-clients = { module = "org.apache.kafka:kafka-clients", version.ref = "kafka" }
 kafka-connect-api = { module = "org.apache.kafka:connect-api", version.ref = "kafka" }
 kafka-connect-json = { module = "org.apache.kafka:connect-json", version.ref = "kafka" }
+kafka-connect-transforms = { module = "org.apache.kafka:connect-transforms", version.ref = "kafka" }


We need to verify this addition is captured in LICENSE/NOTICE files (if separate from the other kafka dependencies)

added section for kafka libs to LICENSE

You can remove this too for now, until we add the transforms to the runtime.

bryanck · 2025-01-16T17:51:14Z

Thanks @ismailsimsek for porting this over!

ismailsimsek · 2025-01-16T18:00:19Z

LICENSE

+
+--------------------------------------------------------------------------------
+
+This product includes software developed at The Apache Software Foundation.
+
+Apache Kafka (kafka-clients, kafka-connect-api, kafka-connect-json, kafka-connect-transforms)
+
+Copyright: 1999-2022 The Apache Software Foundation.
+Home page: https://kafka.apache.org/
+License: https://www.apache.org/licenses/LICENSE-2.0
+
+--------------------------------------------------------------------------------
+
+This product includes software developed by MongoDB, Inc.
+
+* MongoDB BSON (bson)
+
+Copyright: 2008-present MongoDB, Inc.
+Home page: https://www.mongodb.com/
+License: https://github.com/mongodb/mongo-java-driver/blob/main/LICENSE.txt


@danielcweeks is this look ok?

You shouldn't modify the root LICENSE

Given this isn't yet being included in the runtime, we don't necessarily need to add anything to the runtime LICENSE until we add it to that.

@bryanck should it be the NOTICE file instead? bson is using implementation scope? is it then included with the distribution?

reverted

…onnect project

github-actions bot added INFRA docs build KAFKACONNECT labels Jan 9, 2025

ismailsimsek commented Jan 9, 2025

View reviewed changes

kafka-connect/build.gradle Show resolved Hide resolved

Fokko requested a review from bryanck January 15, 2025 14:04

bryanck and others added 13 commits January 15, 2025 17:13

Add SMT project with copy value SMT (#50)

18fed10

(cherry picked from commit 639b0d5b41b827d984aae04efe594315ec2b2b91)

Add SMT to tranform DMS messages for CDC (#56)

801067b

(cherry picked from commit 63cde8e7f6c12392c7741922d5e6ad807051f24a)

Update dependencies and SMT readme (#58)

7c51578

(cherry picked from commit b9cd15de938e57bf178e1ccb443e481fde881224)

Style updates to match Iceberg project (#99)

09ecf98

(cherry picked from commit c17c6f734dac96975959e6165416396e4058332c)

Update to Iceberg 1.4.0-rc.2 (#101)

bf4b049

(cherry picked from commit 03dcf40b484f40f62c551b9ddf5cefea93a3440a)

Add experimental Debezium SMT (#104)

2bbd007

(cherry picked from commit d0adaf9f961ceb89aaa408c03874788b3cf2c422)

Add key to Debezium CDC metadata (#107)

932fe3c

(cherry picked from commit 5812322e595cee663d920aedaed21998fffa9bdf)

Config to set target in Debezium SMT (#111)

ba451e4

(cherry picked from commit bf82d607dc2b5e816c8b6f59bcbdc48281154e98)

Place CDC metadata in struct/map in SMTs (#116)

245c9c4

(cherry picked from commit 89f533b2e689cbd1935c3bd1b82eea5e9dc0cd07)

Update readme for transforms (#158)

cb72c08

(cherry picked from commit 92e4d984fe41c20faf68b1c36e6fd20759e0a19f)

SMT for json parsing (apache#214)

d17d51b

* smt-nested-json-as-map - parse json objects into Maps rather than Structs prior to handing to the iceberg connector, for users with unstructured json data. (cherry picked from commit 303435aa794d8df1728f83ca5179e896b17ca4ff)

Option to inject kafka metadata (apache#205)

3df104d

* option-to-inject-kafka-metadata - SMT to add Kafka metadata (topic, partition, offset, timestamp) to Struct and Map types (cherry picked from commit 423b4a8b0f2e42f2dd7de315631e944c285dcb09)

Non-flattening MongoDb Debezium SMT (apache#204)

6a0b394

* matf-non-flattening-mongodb-debezium-smt - adds debezium mongo SMT for converting BSON before/after into typed Struct before/after (cherry picked from commit 21d741e53ce77547edbb5838f1b2b49db619be0c)

ismailsimsek force-pushed the kafka-smt-copy branch 2 times, most recently from 2e2c727 to 9e64c5b Compare January 15, 2025 16:45

danielcweeks reviewed Jan 15, 2025

View reviewed changes

ismailsimsek force-pushed the kafka-smt-copy branch from d24d97d to 80edb19 Compare January 16, 2025 17:48

bryanck approved these changes Jan 16, 2025

View reviewed changes

ismailsimsek force-pushed the kafka-smt-copy branch 2 times, most recently from 823cc83 to 34445e6 Compare January 16, 2025 17:56

ismailsimsek commented Jan 16, 2025

View reviewed changes

Kafka Connect: Add SMTs for Debezium and AWS DMS from tabular kafka c…

cb36925

…onnect project

ismailsimsek force-pushed the kafka-smt-copy branch from 34445e6 to cb36925 Compare January 16, 2025 18:19

braislchao mentioned this pull request Jan 17, 2025

Config for deciding whether to use Iceberg Time type #11174

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

ismailsimsek commented Jan 9, 2025 •

edited

Loading

ismailsimsek Jan 9, 2025

bryanck Jan 17, 2025

ismailsimsek commented Jan 10, 2025

danielcweeks Jan 15, 2025

ismailsimsek Jan 16, 2025

bryanck Jan 17, 2025

danielcweeks Jan 15, 2025

ismailsimsek Jan 16, 2025

bryanck Jan 17, 2025

bryanck commented Jan 16, 2025

ismailsimsek Jan 16, 2025

bryanck Jan 16, 2025

bryanck Jan 16, 2025

ismailsimsek Jan 16, 2025 •

edited

Loading

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

Are you sure you want to change the base?

Kafka Connect: Add SMTs for Debezium and AWS DMS #11936

Conversation

ismailsimsek commented Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ismailsimsek commented Jan 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryanck commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ismailsimsek Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

ismailsimsek commented Jan 9, 2025 •

edited

Loading

ismailsimsek Jan 16, 2025 •

edited

Loading