Skip to content

Commit

Permalink
Add motivation part to the documentation.
Browse files Browse the repository at this point in the history
  • Loading branch information
rtar committed Jan 31, 2024
1 parent f9a9a31 commit 062aa74
Showing 1 changed file with 45 additions and 0 deletions.
45 changes: 45 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,39 @@

Scala wrapper for [kafka-clients v3.4.0](https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients/3.4.0)

## Motivation

Kafka provides an official Java client out of the box, which could be used from
Scala code without any additional modifications.

The main disadvantage of using an official client directly is that it implies
a very specific threading model to the application. I.e. the consumer is not
thread safe and also expects a rebalance listener to do the operations in the
same thread.

This makes wrapping a client with [Cats Effect](https://typelevel.org/cats-effect/)
classes a bit more complicated than just calling `IO { consumer.poll() }` unless
this is the only call, which is expected to be used.

Skafka does exactly that: a very thin wrapper over official Kafka client to
provide a ready-made Cats Effect API and handle some corner cases concerning
[ConsumerRebalanceListener](https://kafka.apache.org/34/javadoc/org/apache/kafka/clients/consumer/ConsumerRebalanceListener.html) calls.

Comparing to more full-featured libraries such as
[FS2 Kafka](https://fd4s.github.io/fs2-kafka), it might be a little bit more
reliable, because there is little code/logic to do hide the accidenital bugs in.

To summarize:
1. If it suits your goals (i.e. you only ever need to do `consumer.poll()`
without acting on rebalance etc.) then using an official Kafka client directly,
optionally, wrapping all the calls with `cats.effect.IO` is a totally fine idea.
2. If more complicated integration to Cats Effect is required, i.e.
_ConsumerRebalanceListener_ is going to be used then consider using _Skafka_.
3. If streaming with [FS2](https://fs2.io) is required or any other features
the library provides then _FS2 Kafka_ could be a good choice. Note, that it is
less trivial then _Skafka_ and may contain more bugs on top of the official
Kafka client.

## Key features

1. It provides null-less Scala apis for [Producer](skafka/src/main/scala/com/evolutiongaming/skafka/producer/Producer.scala) & [Consumer](skafka/src/main/scala/com/evolutiongaming/skafka/consumer/Consumer.scala)
Expand Down Expand Up @@ -49,3 +82,15 @@ addSbtPlugin("com.evolution" % "sbt-artifactory-plugin" % "0.0.2")

libraryDependencies += "com.evolutiongaming" %% "skafka" % "15.0.0"
```

## Notes

While _Skafka_ provides an ability to use `ConsumerRebalanceListener`
functionality, not all of the method calls are supported.

See the following PRs for more details:
https://github.com/evolution-gaming/skafka/pull/150
https://github.com/evolution-gaming/skafka/pull/122

To our latest knowledge neither `FS2 Kafka` supports all of the
methods / functionality.

0 comments on commit 062aa74

Please sign in to comment.