New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding support Mariadb events, such as GTID and annotate rows event(sql) #45

Merged

osheroff merged 3 commits into osheroff:master from wingerx:master

Aug 27, 2022

wingerx commented Jul 13, 2021 •

edited

Loading

Change EventType to use eventNumber not ordinal().
ANNOTATE_ROWS(160), MARIADB_GTID(162) MARIADB_GTID_LIST(163)
Add MariaDB event data implements, not only types.
Now MariaDB can support query(sql) event and GTID events

@osheroff pls check this PR, ths.


          Adding support Mariadb events, such as GTID and annotate rows event(sql)

3598de7

wingerx mentioned this pull request

mariadb-support: MariaDB event types #25

Closed

Owner

osheroff commented Jul 14, 2021

generally speaking I'd prefer inheritance over a bunch of if statements. Can we try to make MariaDBBinaryLogConnector class that inherits from the base class and changes functionality that way?

I'd also like to see quite a few more tests around mariaDB functionality, if possible -- especially ensuring that we test all the various deserializers.

Author

wingerx commented Jul 15, 2021

generally speaking I'd prefer inheritance over a bunch of if statements. Can we try to make MariaDBBinaryLogConnector class that inherits from the base class and changes functionality that way?

I'd also like to see quite a few more tests around mariaDB functionality, if possible -- especially ensuring that we test all the various deserializers.

ok, i will try to refact.

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java Outdated

               import com.github.shyiko.mysql.binlog.event.deserialization.EventDeserializer.EventDataWrapper;
-              import com.github.shyiko.mysql.binlog.event.deserialization.GtidEventDataDeserializer;
-              import com.github.shyiko.mysql.binlog.event.deserialization.QueryEventDataDeserializer;
-              import com.github.shyiko.mysql.binlog.event.deserialization.RotateEventDataDeserializer;

Owner

osheroff Jul 15, 2021

please leave the import style alone. everyone's editor has an opinion about how imports should be done and they're always changing back and forth. I personally do not care even a little bit but I'm tired of it changing all the time.

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java Outdated

                       }
                   }
+                  protected GtidSet newGtidSet(String gtidSet) {

Owner

osheroff Jul 15, 2021

rename: buildGtidSet

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java Outdated

-                                          gtidSet = new GtidSet(fetchGtidPurged());
-                                      }
-                                  }
+                                  setupGtidToPurgedIfNeeded();

Owner

osheroff Jul 15, 2021

rename: setupGtidSet

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java Outdated

                           ensureEventDataDeserializer(EventType.ROTATE, RotateEventDataDeserializer.class);
                           synchronized (gtidSetAccessLock) {
                               if (gtidSet != null) {
-                                  ensureEventDataDeserializer(EventType.GTID, GtidEventDataDeserializer.class);
-                                  ensureEventDataDeserializer(EventType.QUERY, QueryEventDataDeserializer.class);
+                                  ensureGtidEventDataDeserializerIfNeeded();

Owner

osheroff Jul 15, 2021

rename: ensureGtidEventDataDeserializer

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/BinaryLogClient.java Outdated

    
                              QueryEventData queryEventData = (QueryEventData) EventDataWrapper.internal(event.getData());

                              String sql = queryEventData.getSql();

                          case ANNOTATE_ROWS:

                              String sql = getQuerySql(event);

Owner

osheroff Jul 15, 2021

if you look at the implementation of getQuerySql, you'll see that this is false sharing -- in the case of MariaDB's QUERY event it bounces right back to the mysql-compatible getQuerySql.

Therefore -- keep the implementation of the QUERY case the same, just make a different impl for maria.

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/MariadbBinaryLogClient.java

+                  @Override
+                  protected void setupGtidToPurgedIfNeeded() throws IOException {
+                      //Mariadb ignore

Owner

osheroff Jul 15, 2021

does maria-db not support the gtid_purged variable?

Author

wingerx Jul 16, 2021

yeah, u can find this in https://mariadb.com/kb/en/system-variable-differences-between-mariadb-100-and-mysql-56/

osheroff reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/MariadbBinaryLogClient.java Outdated

+                  @Override
+                  protected void ensureGtidEventDataDeserializerIfNeeded() {
+                      if (isUseSendAnnotateRowsEvent()) {
+                          ensureEventDataDeserializer(EventType.ANNOTATE_ROWS, AnnotateRowsEventDataDeserializer.class);

Owner

osheroff Jul 15, 2021

i don't think we need all the isUseSendAnnotate stuff. It's not going to hurt anything to have an unsed data deserializer in there.

Owner

osheroff commented Jul 15, 2021

needs some documentation about setting maria-db specific flags and such

wingerx added 2 commits

July 16, 2021 13:56


          refact to support mariadb types

1f27791


          add some doc informations about mariadb

486a447

wingerx force-pushed the master branch from 8812d83 to 486a447 Compare

July 16, 2021 05:57

Collaborator

gunnarmorling commented Jul 17, 2021

Hey @jpechane, @Naros, check this one out, it will probably be relevant for the MariaDB exploration within Debezium.

Great effort, @wingerx!

jpechane commented Jul 21, 2021

@wingerx Great effort, thanks! Does the code incorporate old PRs?

Thanks

Author

wingerx commented Jul 23, 2021

No, i think they are similar only in Mariadb

xiaoyu19950911 reviewed

View reviewed changes

src/main/java/com/github/shyiko/mysql/binlog/MariadbBinaryLogClient.java

+                              dumpBinaryLogCommand = new DumpBinaryLogCommand(serverId, "", 0L, isUseSendAnnotateRowsEvent());
+                          } else {
+                              dumpBinaryLogCommand = new DumpBinaryLogCommand(serverId, getBinlogFilename(), getBinlogPosition());

xiaoyu19950911 Aug 6, 2021

There should be some different implementations of Mariadb,look at https://github.com/shyiko/mysql-binlog-connector-java/pull/135/commits/36fca51464ccb8eb900b2addc23ac7e6c1c623ff

scottmf commented Nov 14, 2021

hi, I haven't seen any activity on this in a while. Is this is a dead PR or is there a chance of it moving forward? I, like many others, am eagerly awaiting GTID support for mariadb :)

arman1371 commented Dec 4, 2021

Hi
Thanks for your effort.
Is there any activity here? Can we use this development branch and hope this will be merged soon?

ivapiv commented Aug 10, 2022 •

edited

Loading

Hi @osheroff, could you maybe share any updates regarding the status in this PR? If you could let us know what is missing in order the changes to make it on master, would be happy to contribute and help to move forward with it.

Owner

osheroff commented Aug 13, 2022

trying to get this going here #79

osheroff merged commit 8123277 into osheroff:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet