Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Validate importer to ingestion takes <2s with Kafka #29

Open
mgoelswirlds opened this issue Aug 26, 2022 · 2 comments
Open

Validate importer to ingestion takes <2s with Kafka #29

mgoelswirlds opened this issue Aug 26, 2022 · 2 comments
Assignees
Labels
BigDataPOC Mirror Node POC

Comments

@mgoelswirlds
Copy link
Collaborator

Import a 20k+ record file data through the new importer into Pinot.
Verify the roundtrip time <2s

@mgoelswirlds mgoelswirlds added the BigDataPOC Mirror Node POC label Aug 26, 2022
@mgoelswirlds mgoelswirlds moved this to 🏃‍♀ Sprint backlog in Mirror Node Aug 29, 2022
@mgoelswirlds mgoelswirlds self-assigned this Aug 29, 2022
@steven-sheehy steven-sheehy moved this from 🏃‍♀ Sprint backlog to 👷 In progress in Mirror Node Aug 29, 2022
@mgoelswirlds
Copy link
Collaborator Author

mgoelswirlds commented Aug 31, 2022

Tested this with a records file from Performance environment containing 24667 records.
For the entire file ingestion took about 2.6 seconds.
Testing just one transaction took 665 ms

@mgoelswirlds mgoelswirlds changed the title Validate importer to ingestion takes <2s Validate importer to ingestion takes <2s with Kafka Sep 1, 2022
@mgoelswirlds mgoelswirlds moved this from 👷 In progress to ✅ Done in Mirror Node Sep 1, 2022
@mgoelswirlds
Copy link
Collaborator Author

So After further Investigation the new importer is taking most of the time and actual Kafka->Pinot->Rest Query takes about 290ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BigDataPOC Mirror Node POC
Projects
None yet
Development

No branches or pull requests

1 participant