Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Retrievers #1138

Merged
merged 55 commits into from
Jan 16, 2025
Merged

Refactor Retrievers #1138

merged 55 commits into from
Jan 16, 2025

Conversation

letonghan
Copy link
Collaborator

@letonghan letonghan commented Jan 14, 2025

Description

Refactor retrievers into E-RAG style.
The folder structure after refactored is as below:

  • retrievers
    • deployment
      • docker_compose
      • kubernetes
    • src
      • integrations
        • redis.py
        • milvus.py
        • ...
      • Dockerfile
      • opea_retrievers_microservice.py
      • README.md
      • requirements.txt
      • set_env.sh

Issues

#1004

Type of change

List the type of change like below. Please delete options that are not relevant.

  • Breaking change (fix or feature that would break existing design and interface)

Dependencies

In requirements.txt

Tests

Local tested

Sorry, something went wrong.

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Signed-off-by: letonghan <[email protected]>
pre-commit-ci bot and others added 4 commits January 14, 2025 01:59
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
Signed-off-by: letonghan <[email protected]>
@letonghan
Copy link
Collaborator Author

Related PR to fix GenAIExamples issues: opea-project/GenAIExamples#1387

letonghan and others added 4 commits January 14, 2025 10:49
Signed-off-by: letonghan <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
letonghan and others added 13 commits January 14, 2025 11:15
Signed-off-by: letonghan <[email protected]>
Signed-off-by: letonghan <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
…/GenAIComps into refactor_retrievers
Signed-off-by: letonghan <[email protected]>
update
Signed-off-by: letonghan <[email protected]>
Signed-off-by: letonghan <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
for more information, see https://pre-commit.ci
@rbrugaro
Copy link
Collaborator

@letonghan looks like this refactor completely deleted the neo4j llama-index retriever comps/retrievers/neo4j/llama_index/retriever_community_answers_neo4j.py that is needed for GraphRAG example. This retriever is different from the langchain one in functionality. This retriever is specific for GraphRAG following the microsoft Graphrag pipeline (it retrieves community summaries from neo4j graphDB and generates partial query answers that are later consolidated by final LLM in final response)

@letonghan
Copy link
Collaborator Author

@letonghan looks like this refactor completely deleted the neo4j llama-index retriever comps/retrievers/neo4j/llama_index/retriever_community_answers_neo4j.py that is needed for GraphRAG example. This retriever is different from the langchain one in functionality. This retriever is specific for GraphRAG following the microsoft Graphrag pipeline (it retrieves community summaries from neo4j graphDB and generates partial query answers that are later consolidated by final LLM in final response)

Hi @rbrugaro . Thanks for your comment. Since the GraphRAG example depends on llama_index version of neo4j retrievers, I will refactor it and make sure it doesn't impact the examples.

letonghan and others added 16 commits January 15, 2025 15:05
…o refactor_retrievers
for more information, see https://pre-commit.ci
…/GenAIComps into refactor_retrievers
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
Signed-off-by: letonghan <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
Signed-off-by: letonghan <[email protected]>
…o refactor_retrievers
for more information, see https://pre-commit.ci
Signed-off-by: letonghan <[email protected]>
@rbrugaro
Copy link
Collaborator

@letonghan neo4j yaml is still missing in docker_compose folder

@rbrugaro
Copy link
Collaborator

@letonghan the retriever_neo4j test passed but seems the telemetry traces endpoint 4318 didn't connect: https://github.com/opea-project/GenAIComps/actions/runs/12788252024/job/35649181305?pr=1138#step:4:4037
I also see some neo4j warnings that will be resolved with my upcoming PR that is on hold until this refactor is complete (is related to a package version)

@joshuayao joshuayao linked an issue Jan 16, 2025 that may be closed by this pull request
letonghan and others added 3 commits January 16, 2025 10:56
…o refactor_retrievers
for more information, see https://pre-commit.ci
@letonghan
Copy link
Collaborator Author

@letonghan neo4j yaml is still missing in docker_compose folder

Thanks @rbrugaro, the missing docker compose yaml files are added now.

@chensuyue
Copy link
Collaborator

comps/retrievers/vdms/langchain/Dockerfile will be implement in another PR.

@chensuyue chensuyue merged commit c49db2b into main Jan 16, 2025
24 of 25 checks passed
@chensuyue chensuyue deleted the refactor_retrievers branch January 16, 2025 06:12
chensuyue pushed a commit to opea-project/GenAIExamples that referenced this pull request Jan 16, 2025
Delete redundant retrievers docker image in docker_images_list.md.
Refactor Retrievers related Examples READMEs.
Change all of the comps/retrievers/xxx/xxx/Dockerfile path into comps/retrievers/src/Dockerfile.

Fix the Examples CI issues of PR opea-project/GenAIComps#1138.
Signed-off-by: letonghan <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] ERAG refactor - retrievers
4 participants