Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upsert optimizations #2

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Upsert optimizations #2

wants to merge 3 commits into from

Conversation

DosticJelena
Copy link

@DosticJelena DosticJelena commented Sep 20, 2023

Optimizations include the following changes:

  • Modified the upsert function to use asynchronous requests for each batch.
  • Increased the batch size from 100 to 200, based on experimental results.
  • Implemented asynchronous version for the add method (async_add) in the Pinecone vector store.

NOTE: Please do not merge this pull request.

Copy link

@jamescalam jamescalam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DosticJelena looks good to me!

@DosticJelena DosticJelena force-pushed the async_optimizations branch 3 times, most recently from 4c8b403 to 8f5fc90 Compare October 2, 2023 08:17
DosticJelena pushed a commit that referenced this pull request Dec 28, 2023
* Grammatical_error in docs/api_reference/query.rst

* Grammatical error in docs/end_to_end_tutorials/dev_practices/e2e_evaluation.md

* Changed word from unstructed to unstructured

* Corrected the spelling of performing

* Update evaluating.md

Corrected the spelling of "guidelines"

* Update evaluation.md

Changed " things are working will together" to " things are working well together"

* Update usage_pattern.md (#2)

* Update usage_pattern.md

Corrected the spellings of measure.

* Update basic_strategies.md

* Update (run-llama#8)

* Introduce Google Generative Language Semantic Retriever (run-llama#9440)

* Add a Gemini models for text and multi-modal. (run-llama#9452)

* Add a Gemini models for text and multi-modal.

Includes each of {text, mm} x {completion, chat} x {stream, one-shot}
x {sync, async} for Gemini's Developer API.

And two notebooks - regular text & multi-modal.

* add gemini into doc and rename MM example file

* Add version constraints to pip install

And remove noisy output.

* Enrich Gemini Example

* cr

* Use gemini-pro instead of ultra

* cr

* cr

* cr

---------

Co-authored-by: Haotian Zhang <[email protected]>

* [version] bump to v0.9.15 (run-llama#9485)

* use persist dir in faiss (run-llama#9486)

* Gemini Embedding (run-llama#9476)

* Gemini Embedding

* cr

* Merge Gemini Same Role Chat Message (run-llama#9498)

* Fix Gemini Same Role Chat Msg

* cr

* cr

* cr

* cr

* Teeny stray link causing weirdness (run-llama#9501)

* Fixing CHANGELOG for wrong PR ID (run-llama#9505)

* Overhauling indexing to emphasize VectorStoreIndex and SEO

* Expanding use-cases docs: Q&A and chatbots

* Fix small bug in string method of NodeWithScore (run-llama#9494)

Update schema.py

Avoid exception in __str__ method of NodeWithScore when the optional member variable self.score is None.

* Added missing `default=None` to `LLM.system_prompt` (run-llama#9504)

* Adding default=None to system_prompt to fix type errors

* Added CHANGELOG entry

* Update PDFReader (run-llama#9508)

Add to PDFReader so that a user can specify if they want the PDF read in as one whole Document, or each page as a Document.

* Updating Spelling Mistakes (run-llama#9513)

* Grammatical_error in docs/api_reference/query.rst

* Grammatical error in docs/end_to_end_tutorials/dev_practices/e2e_evaluation.md

* Changed word from unstructed to unstructured

* Corrected the spelling of performing

---------

Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>

* FIxing Grammatical issues in docs (run-llama#9514)

* Fixing Docs (run-llama#9515)

* Allowing `LOCALAI_DEFAULTS` to work with `__ror__` (run-llama#9502)

---------

Co-authored-by: Lawrence Tsang <[email protected]>
Co-authored-by: Mark McDonald <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: Logan <[email protected]>
Co-authored-by: Laurie Voss <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: Max <[email protected]>
Co-authored-by: Alex McGraw <[email protected]>
Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>

* Update train_questions.txt (run-llama#5)

removed a from a developing

* updated basic_strategies.md and building_rag_from_scratch.md files (run-llama#6)

* Update basic_strategies.md

* Update building_rag_from_scratch.md

* Update deprecated_terms.md , changed "THe" to "The"

* Update graph_stores.md

* Update guidance.md

* Update vector_stores.md

* This PR addresses grammatical and clarity enhancements. (run-llama#7)

* Update structured_outputs.md

 Grammatical Improvements

* Update query_transformations.md

Changed followup to follow-up.

* Update structured_outputs.md

---------

Co-authored-by: Shorthills AI <[email protected]>

* made a grammatical correction (run-llama#10)

* Update train_questions.txt

removed a from a developing

* Update agents.md

removed a from a more information

* Update gatsby_license.txt (run-llama#11)

* Update gatsby_license.txt

grammar and tense correction: "read, understand, agree to and accept all the terms" to "read, understood, and agreed to accept all the terms", "do not agree" to "disagree", "agreement for keeping" to "agreement to keep", "unlink or detach or remove" to "unlink, detach or remove", "statements concerning tax" to "statements concerning the tax"

* Update concepts.md

grammar and tense correction: "or retrieved data" to "or retrieve data", "data which is" to "data that is", "class, are responsible" to "class, is responsible"," that allow you to ask question" to "that allows you to ask questions".

* Update discover_llamaindex.md (run-llama#15)

* Update llms.md (run-llama#9)

Corrected spellings of 'results'.

* Update llms.md (run-llama#13)

Corrected spelling of results.

* Update CHANGELOG.md (run-llama#14)

fixed the spelling mistake of  the evaluation

* Update deprecated_terms.md

Links Urls in Deprecated_terms is not working.
1. Configuring settings in the Service Context Link updated.
2. Parsing Documents into Nodes Link updated.
3. LLMs in LlamaIndex Link updated.
4. Setting LLMs in the ServiceContext updated

* Update deprecated_terms.md

Links Updated in Deprecated_terms mentioned below:
1. Index Usage Pattern link updated.
2. Vector Store Guide link updated
3. Vector Store Integrations link updated.
4. Index Usage Pattern link updated.
5. Vector Store Guide link updated.
6. Vector Store Integrations link updated.

* Update deprecated_terms.md

GPTVectorStoreIndex "Index Usage Pattern" Link updated.

---------

Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>
Co-authored-by: ManpreetShorthillsAI <[email protected]>
Co-authored-by: SakshiShorthillsAI <[email protected]>
Co-authored-by: Lawrence Tsang <[email protected]>
Co-authored-by: Mark McDonald <[email protected]>
Co-authored-by: Logan <[email protected]>
Co-authored-by: Laurie Voss <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: Max <[email protected]>
Co-authored-by: Alex McGraw <[email protected]>
Co-authored-by: Aayush <[email protected]>
Co-authored-by: Saransh Sharma <[email protected]>
Co-authored-by: AbhishekYadavShorthillsAI <[email protected]>
Co-authored-by: AryamanJaiswalShorthillsAI <[email protected]>
Co-authored-by: Sanskar Tanwar <[email protected]>
Co-authored-by: ShreeramKumawatShorthillsAI <[email protected]>
Co-authored-by: Vishal <[email protected]>
Co-authored-by: AashutoshPathakShorthillsAI <[email protected]>
Co-authored-by: Rajneesh Singh <[email protected]>
DosticJelena pushed a commit that referenced this pull request Dec 28, 2023
* Update CHANGELOG.md

* Update agents.md

* Updated understanding.md (run-llama#51)

* Grammatical_error in docs/api_reference/query.rst

* Grammatical error in docs/end_to_end_tutorials/dev_practices/e2e_evaluation.md

* Changed word from unstructed to unstructured

* Corrected the spelling of performing

* Update evaluating.md

Corrected the spelling of "guidelines"

* Update evaluation.md

Changed " things are working will together" to " things are working well together"

* Update usage_pattern.md (#2)

* Update usage_pattern.md

Corrected the spellings of measure.

* Update basic_strategies.md

* Update (run-llama#8)

* Introduce Google Generative Language Semantic Retriever (run-llama#9440)

* Add a Gemini models for text and multi-modal. (run-llama#9452)

* Add a Gemini models for text and multi-modal.

Includes each of {text, mm} x {completion, chat} x {stream, one-shot}
x {sync, async} for Gemini's Developer API.

And two notebooks - regular text & multi-modal.

* add gemini into doc and rename MM example file

* Add version constraints to pip install

And remove noisy output.

* Enrich Gemini Example

* cr

* Use gemini-pro instead of ultra

* cr

* cr

* cr

---------

Co-authored-by: Haotian Zhang <[email protected]>

* [version] bump to v0.9.15 (run-llama#9485)

* use persist dir in faiss (run-llama#9486)

* Gemini Embedding (run-llama#9476)

* Gemini Embedding

* cr

* Merge Gemini Same Role Chat Message (run-llama#9498)

* Fix Gemini Same Role Chat Msg

* cr

* cr

* cr

* cr

* Teeny stray link causing weirdness (run-llama#9501)

* Fixing CHANGELOG for wrong PR ID (run-llama#9505)

* Overhauling indexing to emphasize VectorStoreIndex and SEO

* Expanding use-cases docs: Q&A and chatbots

* Fix small bug in string method of NodeWithScore (run-llama#9494)

Update schema.py

Avoid exception in __str__ method of NodeWithScore when the optional member variable self.score is None.

* Added missing `default=None` to `LLM.system_prompt` (run-llama#9504)

* Adding default=None to system_prompt to fix type errors

* Added CHANGELOG entry

* Update PDFReader (run-llama#9508)

Add to PDFReader so that a user can specify if they want the PDF read in as one whole Document, or each page as a Document.

* Updating Spelling Mistakes (run-llama#9513)

* Grammatical_error in docs/api_reference/query.rst

* Grammatical error in docs/end_to_end_tutorials/dev_practices/e2e_evaluation.md

* Changed word from unstructed to unstructured

* Corrected the spelling of performing

---------

Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>

* FIxing Grammatical issues in docs (run-llama#9514)

* Fixing Docs (run-llama#9515)

* Allowing `LOCALAI_DEFAULTS` to work with `__ror__` (run-llama#9502)

---------

Co-authored-by: Lawrence Tsang <[email protected]>
Co-authored-by: Mark McDonald <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: Logan <[email protected]>
Co-authored-by: Laurie Voss <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: Max <[email protected]>
Co-authored-by: Alex McGraw <[email protected]>
Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>

* Update train_questions.txt (run-llama#5)

removed a from a developing

* updated basic_strategies.md and building_rag_from_scratch.md files (run-llama#6)

* Update basic_strategies.md

* Update building_rag_from_scratch.md

* Update deprecated_terms.md , changed "THe" to "The"

* Update graph_stores.md

* Update guidance.md

* Update vector_stores.md

* This PR addresses grammatical and clarity enhancements. (run-llama#7)

* Update structured_outputs.md

 Grammatical Improvements

* Update query_transformations.md

Changed followup to follow-up.

* Update structured_outputs.md

---------

Co-authored-by: Shorthills AI <[email protected]>

* made a grammatical correction (run-llama#10)

* Update train_questions.txt

removed a from a developing

* Update agents.md

removed a from a more information

* Update gatsby_license.txt (run-llama#11)

* Update gatsby_license.txt

grammar and tense correction: "read, understand, agree to and accept all the terms" to "read, understood, and agreed to accept all the terms", "do not agree" to "disagree", "agreement for keeping" to "agreement to keep", "unlink or detach or remove" to "unlink, detach or remove", "statements concerning tax" to "statements concerning the tax"

* Update concepts.md

grammar and tense correction: "or retrieved data" to "or retrieve data", "data which is" to "data that is", "class, are responsible" to "class, is responsible"," that allow you to ask question" to "that allows you to ask questions".

* Update discover_llamaindex.md (run-llama#15)

* Update llms.md (run-llama#9)

Corrected spellings of 'results'.

* Update llms.md (run-llama#13)

Corrected spelling of results.

* Update CHANGELOG.md (run-llama#14)

fixed the spelling mistake of  the evaluation

* Update deprecated_terms.md

Links Urls in Deprecated_terms is not working.
1. Configuring settings in the Service Context Link updated.
2. Parsing Documents into Nodes Link updated.
3. LLMs in LlamaIndex Link updated.
4. Setting LLMs in the ServiceContext updated

* Update deprecated_terms.md

Links Updated in Deprecated_terms mentioned below:
1. Index Usage Pattern link updated.
2. Vector Store Guide link updated
3. Vector Store Integrations link updated.
4. Index Usage Pattern link updated.
5. Vector Store Guide link updated.
6. Vector Store Integrations link updated.

* Update deprecated_terms.md

GPTVectorStoreIndex "Index Usage Pattern" Link updated.

* Update chatbots.md

Improve clarity and consistency in the Chatbots section of the documentation:
1. Corrected the term 'LLM's' to 'LLMs' for consistency.
2. Modified 'Here's some relevant resources' to 'Here are some relevant resources' for grammatical correctness.

* Update chatbots.md with spelling : chatbots

Previous : chat  bots
Corrected : chatbots

* Update multimodal.md

removed an extra word : "out" for consistency and clarity

* Update chatbots.md

* Update agents.md

* Update embeddings.md

* Update understanding.md

* Update extraction.md

---------

Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: Shorthills AI <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>
Co-authored-by: ManpreetShorthillsAI <[email protected]>
Co-authored-by: SakshiShorthillsAI <[email protected]>
Co-authored-by: Lawrence Tsang <[email protected]>
Co-authored-by: Mark McDonald <[email protected]>
Co-authored-by: Logan <[email protected]>
Co-authored-by: Laurie Voss <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: Max <[email protected]>
Co-authored-by: Alex McGraw <[email protected]>
Co-authored-by: Aayush <[email protected]>
Co-authored-by: Saransh Sharma <[email protected]>
Co-authored-by: AbhishekYadavShorthillsAI <[email protected]>
Co-authored-by: AryamanJaiswalShorthillsAI <[email protected]>
Co-authored-by: Sanskar Tanwar <[email protected]>
Co-authored-by: ShreeramKumawatShorthillsAI <[email protected]>
Co-authored-by: Vishal <[email protected]>
Co-authored-by: AashutoshPathakShorthillsAI <[email protected]>
Co-authored-by: Rajneesh Singh <[email protected]>
Co-authored-by: VagishaMisraShorthillsAI <[email protected]>
Co-authored-by: PankajKumarShorthillsAI <[email protected]>
Co-authored-by: Adarsh Shrivastav <[email protected]>
Co-authored-by: Puneet Dhiman <[email protected]>
Co-authored-by: AmitSinghShorthillsAI <[email protected]>

* Update embeddings.md

* Update embeddings.md

* Update embeddings.md

* Update embeddings.md

---------

Co-authored-by: ChetnaGuptaShorthillsAI <[email protected]>
Co-authored-by: Puneet Dhiman <[email protected]>
Co-authored-by: AdityaKalraShorthillsAI <[email protected]>
Co-authored-by: KamalSharmaShorthillsAI <[email protected]>
Co-authored-by: Haotian Zhang <[email protected]>
Co-authored-by: UpneetShorthillsAI <[email protected]>
Co-authored-by: ManpreetShorthillsAI <[email protected]>
Co-authored-by: SakshiShorthillsAI <[email protected]>
Co-authored-by: Lawrence Tsang <[email protected]>
Co-authored-by: Mark McDonald <[email protected]>
Co-authored-by: Logan <[email protected]>
Co-authored-by: Laurie Voss <[email protected]>
Co-authored-by: James Braza <[email protected]>
Co-authored-by: Max <[email protected]>
Co-authored-by: Alex McGraw <[email protected]>
Co-authored-by: Aayush <[email protected]>
Co-authored-by: Saransh Sharma <[email protected]>
Co-authored-by: AbhishekYadavShorthillsAI <[email protected]>
Co-authored-by: AryamanJaiswalShorthillsAI <[email protected]>
Co-authored-by: Sanskar Tanwar <[email protected]>
Co-authored-by: ShreeramKumawatShorthillsAI <[email protected]>
Co-authored-by: Vishal <[email protected]>
Co-authored-by: AashutoshPathakShorthillsAI <[email protected]>
Co-authored-by: Rajneesh Singh <[email protected]>
Co-authored-by: VagishaMisraShorthillsAI <[email protected]>
Co-authored-by: PankajKumarShorthillsAI <[email protected]>
Co-authored-by: Adarsh Shrivastav <[email protected]>
Co-authored-by: AmitSinghShorthillsAI <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants