Citation consistency in Citation Processing (initial ranking vs post validation/re-ranking)

[joachim-danswer]

Purpose: Alignment & renaming of objects for initial (displayed) ranking and r…e-ranking/validation citations

Comments:

• renamed post-reranking/validation citation information consistently to final_... (example: doc_id_to_rank_map -> final_doc_id_to_rank_map)
• changed and renamed objects containing initial ranking information (now: display_...) consistent with final rankings (final_...). Specifically, {} to [] for displayed_search_results
• for CitationInfo, changed citation_num from 'x-th citation in response stream' to the initial position of the doc [NOTE: test implications]
• changed tests: onyx/backend/tests/unit/onyx/chat/stream_processing/test_citation_processing.py onyx/backend/tests/unit/onyx/chat/stream_processing/test_citation_substitution.py

Description

[Provide a brief description of the changes in this PR]

How Has This Been Tested?

• Tested using the unit tests, including the the test_citation_substitution
• Testing using simple sample document where citation displayed on first position is not valid.

Accepted Risk (provide if relevant)

N/A

Related Issue(s) (provide if relevant)

• Mismatch issue of Documents shown and Citation number in text fix #3421

Mental Checklist:

• All of the automated tests pass
• All PR comments are addressed and marked resolved
• If there are migrations, they have been rebased to latest main
• If there are new dependencies, they are added to the requirements
• If there are new environment variables, they are added to all of the deployment methods
• If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
• Docker images build and basic functionalities work
• Author has done a final read through of the PR right before merge

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 18, 2024 10:14pm

[yuhongsun96]

Don't see any issues but the tests are failing

[yuhongsun96]

lgtm