[compat] [server] [client] [test] Global RT DIV: Chunking Support #1385

KaiSernLim · 2024-12-11T10:06:43Z

Summary

Continuation from #1257.

This PR mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics, as the size of the DIV message can surpass the ~1MB Kafka message limit. The existing chunking mechanism is reused, including the CHUNK and CHUNKED_VALUE_MANIFEST values in the message's schemaId:

Every DIV message has GLOBAL_RT_DIV for the header byte in its KafkaKey. The corresponding KafkaMessageEnvelope has a Put payload utilizing the putValue field containing the GlobalRtDiv data, and which has the following schemaId:

If the DIV message is non-chunked, the KME schemaId is set to the current protocol version of GLOBAL_RT_DIV.
If the DIV message is chunk message, the KME schemaId is set to CHUNK.
If the DIV message is a chunk manifest message, the KME schemaId is set to CHUNKED_VALUE_MANIFEST. The schemaId of the ChunkedValueManifest will be the current protocol version of GLOBAL_RT_DIV.

ChunkAssembler is adapted, on the receiver side, to buffer, assemble, and deserialize DIV messages (chunked/non-chunked).

Changes

Added a new MessageType called GlobalRtDiv, which reuses the Put message type format and objects. When the Venice server encounters a message with KafkaKey containing the GlobalRtDiv header byte, it will know to process this message differently from a regular Put.
- The only indication of this new GlobalRtDiv message type is the header byte in KafkaKey. Otherwise, it's identical to a regular Put.
- KafkaMessageEnvelope.avsc will not be updated to avoid the unnecessary risk of incompatible avro formats when upgrading the cluster.
- The risk in not creating a dedicated GlobalRtDiv message type in KME is that the GlobalRtDiv objects will be processed as user records and stored in the storage engine, which seems to be much less scary than a cluster upgrade issue.
Changed bufferAndAssembleRecord() in ChunkAssembler to use a deserialization Function instead of a RecordDeserializer. This is because RecordDeserializer is on a separate inheritance path to the InternalAvroSpecificSerializer.
GlobalRtDiv messages should not be processed if they originate from remote VT and RT, because those are invalid scenarios. These two conditions are checked.

Minor Changes

Renamed resetUpstreamOffsetMap() to mergeUpstreamOffsets() in OffsetRecord.
Updated toString() in KafkaKey, which incorrectly assumed all messages would be ControlMessage, Put, or Delete. This misses Update messages and the new GlobalRtDiv message that is being added.
Added various helper methods (buildPutPayload() and buildManifestPayload()) in VeniceWriter for creating the Put payloads and when chunking is involved.

Testing

Unit Tests
1. testGlobalRtDivChunking() in VeniceWriterUnitTest
2. testShouldProcessRecordForGlobalRtDivMessage() in StoreIngestionTaskTest
3. testProcessGlobalRtDivMessage() in StoreIngestionTaskTest
Integration Test
1. testChunkedDiv() in TestGlobalRtDiv

Does this PR introduce any user-facing changes?

No. You can skip the rest of this section.

…ssages, and renamed `GlobalDivState` avro object to `GlobalRtDivState`. 🌯

…ation . 🥓

2. Split `ChunkAssembler` for RT DIV into its own object. 🍐 3. `GlobalRtDiv` serializer is per-message to be safe, because it doesn't seem to be thread-safe. 🍋‍🟩 4. Fixed spotbugs. 🌶️ 5. Fixed `divChunkAssembler` for the SIT unit test. 🫨

internal/venice-common/src/main/java/com/linkedin/venice/writer/VeniceWriter.java

internal/venice-common/src/test/java/com/linkedin/venice/writer/VeniceWriterUnitTest.java

…will have `GLOBAL_RT_DIV` as the `MessageType`. 🏯

…ject` in the `sendMessageFunction`. 🪀

sixpluszero · 2024-12-16T19:03:21Z

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java

@@ -1479,7 +1482,7 @@ protected void updateOffsetMetadataInOffsetRecord(PartitionConsumptionState part
        upstreamTopic = versionTopic;
      }
      if (upstreamTopic.isRealTime()) {
-        offsetRecord.resetUpstreamOffsetMap(partitionConsumptionState.getLatestProcessedUpstreamRTOffsetMap());
+        offsetRecord.mergeUpstreamOffsets(partitionConsumptionState.getLatestProcessedUpstreamRTOffsetMap());


Why do you do this change? Personally I think this makes less explicit about the offset map we are tracking

Because I feel that reset() is not the right word for it. It's not clearing the state or replacing the map entirely. It's updating the existing map with the values of the new map and retaining the values missing from the new map.

I can rename it to updateUpstreamOffsets().

sixpluszero · 2024-12-16T19:17:39Z

internal/venice-common/src/main/java/com/linkedin/venice/kafka/protocol/enums/MessageType.java

@@ -21,7 +21,8 @@
 */
 public enum MessageType implements VeniceEnumValue {
  PUT(0, Constants.PUT_KEY_HEADER_BYTE), DELETE(1, Constants.PUT_KEY_HEADER_BYTE),
-  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE);
+  CONTROL_MESSAGE(2, Constants.CONTROL_MESSAGE_KEY_HEADER_BYTE), UPDATE(3, Constants.UPDATE_KEY_HEADER_BYTE),
+  GLOBAL_RT_DIV(4, Constants.GLOBAL_RT_DIV_KEY_HEADER_BYTE);


I wonder if it is ok to not add it into KME. I am thinking for example, Da Vinci and CDC, if the code is not upgraded, how can it be handling the global rt div? Or should we completely mute them from processing RT div message?
Will this controlled by store config or just server config?

In my opinion, Davinci and CDC don't need to process the RT div messages. Only the next leader has to read and process it, right. They can simply ignore them.

It's controlled by server config.

I mean, with this code change, we will eventually enable the feature to produce RT DIV snapshot to VT topic. This will be processed by follower, which also applicable to DVC. I know DVC won't be leader, but will the existing code handle unknown new msg type gracefully?

You're right. The reader of the msg type (to be able to ignore it) has to be deployed before enabling this feature. i don't know if there is other way to walk around it, feel free to suggest alternatives.

I didn't look very deep, but I think that the existing code will throw an exception when encountering an unknown message type.

Handling an unknown message type will be implemented in a future PR. Since we control the feature flag, we can choose not to enable the feature flag until all server instances support the new message type so there won't be any messages with this message type floating around.

We need to deploy the changes to all VT consumers before enabling this feature, which includes:

Server.

DaVinci.

ETL.

CDC.

And I think it is better to treat the new KME type as a Control Message, and I would like to learn why if we can't do that.

gaojieliu

I would like to call out another item we didn't discuss in the review meeting:

Cleanup the Global RT DIV messages from VT.

This new type of message can be large and with chunking support, we are leaking the data chunks as chunk id is unique.
I was thinking some strategy to let follower to send out a Kafka delete or Kafka message with empty value for the previous key after consuming a new Global RT DIV message, and if we don't do the cleanup, the size of Kafka topic might grow a lot depending on the sending frequency.
Maybe we don't need to implement such cleanup in the MVP, but I think eventually, we need some way to clean them up from the version topics.

gaojieliu · 2024-12-17T17:46:00Z

...ient/src/main/java/com/linkedin/davinci/kafka/consumer/LeaderFollowerStoreIngestionTask.java

@@ -1904,6 +1906,10 @@ protected boolean shouldProcessRecord(PubSubMessage<KafkaKey, KafkaMessageEnvelo
              }
            }
          }
+          // Global RT DIV messages should be completely ignored when leader is consuming from remote version topic
+          if (record.getKey().isGlobalRtDiv()) {


Is it possible to have Global RT DIV messages in the remote version topic?
There are a few cases for remote version topic consumption:

NR for the venice push job.

Streaming Reprocessing job.

Data recovery for batch store.
None of the use cases in theory would encounter Global RT DIV messages.
Today, there is no support for hybrid store data recovery and it is being done via re-push.
So can you explain the scenarios in your mind?

gaojieliu · 2024-12-17T17:50:04Z