Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add PulsarAvroMessageDecoder #14397

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

chunxiaozheng
Copy link

@chunxiaozheng chunxiaozheng commented Nov 6, 2024

With this PR, decoder can directly obtain the schema through the pulsar admin API, instead of setting the avro schema in the table config.
In addition, the following configurations need to be added in the table config

"stream.pulsar.decoder.prop.admin.servers": "http://localhost:8081",
"stream.pulsar.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.PulsarAvroMessageDecoder"

@codecov-commenter
Copy link

codecov-commenter commented Nov 6, 2024

Codecov Report

Attention: Patch coverage is 0% with 46 lines in your changes missing coverage. Please review.

Project coverage is 55.43%. Comparing base (59551e4) to head (2543e39).
Report is 1293 commits behind head on master.

Files with missing lines Patch % Lines
...gin/inputformat/avro/PulsarAvroMessageDecoder.java 0.00% 42 Missing ⚠️
...gin/inputformat/avro/SimpleAvroMessageDecoder.java 0.00% 4 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (59551e4) and HEAD (2543e39). Click for more details.

HEAD has 44 uploads less than BASE
Flag BASE (59551e4) HEAD (2543e39)
integration 7 0
integration2 3 0
temurin 12 3
java-21 7 2
skip-bytebuffers-true 3 1
skip-bytebuffers-false 7 2
unittests 5 3
java-11 5 1
unittests2 3 0
integration1 2 0
custom-integration1 2 0
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #14397      +/-   ##
============================================
- Coverage     61.75%   55.43%   -6.32%     
- Complexity      207      791     +584     
============================================
  Files          2436     2095     -341     
  Lines        133233   110372   -22861     
  Branches      20636    17502    -3134     
============================================
- Hits          82274    61188   -21086     
+ Misses        44911    44321     -590     
+ Partials       6048     4863    -1185     
Flag Coverage Δ
custom-integration1 ?
integration ?
integration1 ?
integration2 ?
java-11 55.40% <0.00%> (-6.31%) ⬇️
java-21 55.29% <0.00%> (-6.33%) ⬇️
skip-bytebuffers-false 55.43% <0.00%> (-6.32%) ⬇️
skip-bytebuffers-true 55.24% <0.00%> (+27.52%) ⬆️
temurin 55.43% <0.00%> (-6.32%) ⬇️
unittests 55.43% <0.00%> (-6.31%) ⬇️
unittests1 55.43% <0.00%> (+8.54%) ⬆️
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Jackie-Jiang Jackie-Jiang added documentation Configuration Config changes (addition/deletion/change in behavior) labels Nov 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Configuration Config changes (addition/deletion/change in behavior) documentation ingestion plugins
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants