OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

FuryMartin · 2024-09-23T00:41:07Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs

Which issue(s) this PR fixes:

Fixes #96

FuryMartin · 2024-09-23T00:47:51Z

Changes

Ianvs

Core

add joint inference const
- core/common/constant.py
fix deprecated interface of pandas, see details at fix: deprecate outdated interface of pandas #135
- core/storymanager/rank/rank.py
fix delimiter error while reading existed csv when new hyper-paramteters are added
- core/storymanager/visualization/visualization.py
add Joint Inference Paradigm in :
- core/testcasecontroller/algorithm/algorithm.py
- core/testcasecontroller/algorithm/paradigm/base.py
- core/testcasecontroller/algorithm/paradigm/joint_inference/joint_inference.py
- core/testcasecontroller/algorithm/paradigm/joint_inference/__init__.py
fix Hard Example Mining module initilization process, see details at Fix incorrect preloading process of the Hard Example Mining module. #141
- core/testcasecontroller/algorithm/module/module.py
align dataset interface with @IcyFeather , see details at: OSPP: Implementation of Domain-Specific Large Model Benchmarking Based on KubeEdge-Ianvs #144

Denpendencies

fix missing dependencies.
- requirements.txt

Example

add cloud-edge-collaborative-inference-for-llm example
- examples/cloud-edge-collaborative-inference-for-llm/

Sedna

See details at FuryMartin/Sedna

support llm in Joint Inference
- lib/sedna/core/joint_inference/joint_inference.py
fix version conflict of dependencies
- lib/requirements.txt
fix TypeError Exception when model_url is None
- lib/sedna/core/base.py
align dataset interface with @IcyFeather
- lib/sedna/datasources/__init__.py

FuryMartin · 2024-09-26T15:37:00Z

CI will install sedna-0.4.1-py3-none-any.whl under examples/resources/third_party/ to build environment.

However, since this PR updates Sedna, the old version installed in the CI environment lacks the new interfaces, leading to unexpected-keyword-arg and no-name-in-module errors.

Run pylint '/home/runner/work/ianvs/ianvs/core'
  
************* Module core.testcasecontroller.algorithm.paradigm.base
core/testcasecontroller/algorithm/paradigm/base.py:130:19: E1123: Unexpected keyword argument 'cloud' in constructor call (unexpected-keyword-arg)
core/testcasecontroller/algorithm/paradigm/base.py:130:19: E1123: Unexpected keyword argument 'LCReporter_enable' in constructor call (unexpected-keyword-arg)
************* Module core.testenvmanager.dataset.dataset
core/testenvmanager/dataset/dataset.py:21:0: E0611: No name 'JsonlDataParse' in module 'sedna.datasources' (no-name-in-module)
core/testenvmanager/dataset/dataset.py:21:0: E0611: No name 'JSONDataInfoParse' in module 'sedna.datasources' (no-name-in-module)

-----------------------------------
Your code has been rated at 9.8/10
Error: The operation was canceled.

It's a little tricky, I'm not sure how to solve it.

@hsj576 @MooreZheng

I have written a detailed analysis of this issue in #134, and further discussions will take place there.

MooreZheng

Reminder: there are CI issues that remain to be resolved, see https://github.com/kubeedge/ianvs/actions/runs/11201880744?pr=149

hsj576

The sedna installation originally used by the community should have a backup in third_party-bk.

FuryMartin · 2024-10-28T10:28:52Z

Reminder: there are CI issues that remain to be resolved, see https://github.com/kubeedge/ianvs/actions/runs/11201880744?pr=149

As discussed in #152 , this has been temporarily fixed by introducing sedna-0.6.0.1-py3-none-any.whl. This package is constructed from my sedna fork FuryMartin/sedna.

The sedna installation originally used by the community should have a backup in third_party-bk.

I have move the sedna-0.4.1-py3-none-any.whl to third_party-bk

I have completed all the comment completion and document organization work.
Please review the code. @hsj576 @MooreZheng

hsj576 · 2024-10-28T13:07:00Z

/lgtm

MooreZheng

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/common/constant.py
core/testcasecontroller/algorithm/algorithm.py
core/testcasecontroller/algorithm/module/module.py
core/testcasecontroller/algorithm/paradigm/init.py
core/testcasecontroller/algorithm/paradigm/base.py
core/testenvmanager/dataset/dataset.py
This pull request contains 12 commits, which might make maintenance difficult, considering the number of contributors, pull requests, and their commits in KubeEdge Ianvs recently. In the final stage, @FuryMartin can squash the commits into one using rebase techniques.

FuryMartin · 2024-10-29T16:27:05Z

This branch has conflicts that must be resolved. Use the web editor or the to resolve conflicts.

squash the commits into one using rebase techniques.

Thanks for the review. I'll fix them tomorrow.

MooreZheng

Current conflicts:

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/common/constant.py
core/storymanager/rank/rank.py
core/testcasecontroller/algorithm/algorithm.py
core/testcasecontroller/algorithm/module/module.py
core/testcasecontroller/algorithm/paradigm/init.py
core/testcasecontroller/algorithm/paradigm/base.py
core/testcasecontroller/metrics/metrics.py
core/testenvmanager/dataset/dataset.py

FuryMartin · 2024-10-30T13:10:33Z

Current conflicts:

This branch has conflicts that must be resolved Use the web editor or the to resolve conflicts. Conflicting files core/common/constant.py core/storymanager/rank/rank.py core/testcasecontroller/algorithm/algorithm.py core/testcasecontroller/algorithm/module/module.py core/testcasecontroller/algorithm/paradigm/init.py core/testcasecontroller/algorithm/paradigm/base.py core/testcasecontroller/metrics/metrics.py core/testenvmanager/dataset/dataset.py

Thanks for the review. These conflicts are caused by the revert #160.I believe they will be fixed automatically by merging #161 at first.

Update at 2024-10-30 21:18

#161 has been closed. @Yoda-wu is trying to create a new PR #162 contains the commits of #143 and #161.

I advise @MooreZheng to first merge #162 , and then proceed with merging #144 . (Because I might have some conflicts with #144 as well)

I will be on call any time tomorrow.

Thanks!

MooreZheng · 2024-10-30T13:53:41Z

Thanks for the review. These conflicts are caused by the revert #160.I believe they will be fixed automatically by merging #161 at first.

Update at 2024-10-30 21:18

#161 has been closed. @Yoda-wu is trying to create a new PR #162 contains the commits of #143 and #161.

I advise @MooreZheng to first merge #162 , and then proceed with merging #144 . (Because I might have some conflicts with #144 as well)

I will be on call any time tomorrow.

Thanks!

Sure. Will try following that path.

hsj576 · 2024-10-30T14:10:35Z

/lgtm

MooreZheng

After merging PR 162 and PR 144, the following new conflicts do come to fix

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/common/constant.py
core/testenvmanager/dataset/dataset.py

FuryMartin · 2024-10-30T14:32:00Z

After merging PR 162 and PR 144, the following new conflicts do come to fix

Fixed.

FuryMartin · 2024-10-30T14:52:49Z

Hi @MooreZheng , if possible, please merge #159 first, too.

Reason

#144 fix #152's pylint error by globally disable two style checks in dataset.py

ianvs/core/testenvmanager/dataset/dataset.py

Lines 20 to 21 in 27519de

    
           # pylint: disable=no-name-in-module 
        
           # pylint: disable=too-many-instance-attributes

As we discussed before, such global-disable-thing is not a good manner.

#162, #144, #159 all used such method to pass the CI check.

I have fixed it in my PR by including a new sedna package, so it's better to leave me the last to merge.

BTW, #154 is not related to this issue, it can be merged at anytime.

MooreZheng · 2024-10-30T15:00:42Z

Hi @MooreZheng , if possible, please merge #159 first, too.

FYI I will on my flight tomorrow morning... Have talked to the #159 owner @safe-b on WeChat and left comments on his PR. @FuryMartin might want to lend him a hand to speed up the overall process.

MooreZheng

Other OSPP projects have been merged, so this shall be the last conflict

This branch has conflicts that must be resolved
Use the web editor or the to resolve conflicts.
Conflicting files
core/testenvmanager/dataset/dataset.py

…example Signed-off-by: Yu Fan <[email protected]> add: sedna 0.6.0.1 and move 0.4.1 to third_party-bk Signed-off-by: Yu Fan <[email protected]>

FuryMartin · 2024-10-31T13:00:32Z

Other OSPP projects have been merged, so this shall be the last conflict

Solved.

MooreZheng · 2024-10-31T14:51:59Z

/lgtm

MooreZheng · 2024-10-31T14:52:31Z

/approve

MooreZheng

All concerns are fixed. Well done! @FuryMartin

kubeedge-bot · 2024-10-31T14:53:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: MooreZheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [MooreZheng]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubeedge-bot added the kind/feature Categorizes issue or PR as related to a new feature. label Sep 23, 2024

kubeedge-bot requested review from jaypume and MooreZheng September 23, 2024 00:41

kubeedge-bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 23, 2024

FuryMartin force-pushed the code branch from 48c2aa6 to 58fcfb4 Compare September 23, 2024 00:43

MooreZheng requested review from hsj576 and removed request for jaypume September 26, 2024 09:39

MooreZheng assigned hsj576 Sep 26, 2024

FuryMartin force-pushed the code branch from ed05960 to fdb9902 Compare September 26, 2024 14:46

FuryMartin mentioned this pull request Sep 27, 2024

Pylint Errors Due to Compatibility Issues with Updated Sedna in CI Environment #152

Open

MooreZheng requested changes Oct 14, 2024

View reviewed changes

kubeedge-bot assigned MooreZheng Oct 14, 2024

FuryMartin force-pushed the code branch from 72714ab to c708cbe Compare October 24, 2024 05:10

hsj576 suggested changes Oct 26, 2024

View reviewed changes

MooreZheng assigned FuryMartin Oct 28, 2024

FuryMartin force-pushed the code branch 2 times, most recently from 520d514 to 5646e27 Compare October 28, 2024 10:22

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 28, 2024

FuryMartin requested a review from MooreZheng October 28, 2024 14:25

MooreZheng requested changes Oct 29, 2024

View reviewed changes

kubeedge-bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2024

kubeedge-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 29, 2024

FuryMartin force-pushed the code branch from 5646e27 to b13ad4f Compare October 30, 2024 10:01

kubeedge-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2024

FuryMartin force-pushed the code branch from b13ad4f to 62b598c Compare October 30, 2024 10:10

MooreZheng requested changes Oct 30, 2024

View reviewed changes

kubeedge-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2024

kubeedge-bot added lgtm Indicates that a PR is ready to be merged. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 30, 2024

MooreZheng requested changes Oct 30, 2024

View reviewed changes

kubeedge-bot removed the lgtm Indicates that a PR is ready to be merged. label Oct 30, 2024

FuryMartin force-pushed the code branch from 62b598c to 0b72480 Compare October 30, 2024 14:30

FuryMartin force-pushed the code branch 2 times, most recently from 3005581 to 0c6427c Compare October 30, 2024 14:42

FuryMartin mentioned this pull request Oct 30, 2024

Fix incorrect preloading process of the Hard Example Mining module. #141

Closed

MooreZheng requested changes Oct 31, 2024

View reviewed changes

add: Joint Inference paradigm and cloud-edge collaborative inference …

fb95a9c

…example Signed-off-by: Yu Fan <[email protected]> add: sedna 0.6.0.1 and move 0.4.1 to third_party-bk Signed-off-by: Yu Fan <[email protected]>

FuryMartin force-pushed the code branch from 0c6427c to fb95a9c Compare October 31, 2024 11:10

FuryMartin requested a review from MooreZheng October 31, 2024 11:13

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 31, 2024

kubeedge-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 31, 2024

MooreZheng approved these changes Oct 31, 2024

View reviewed changes

kubeedge-bot merged commit 852973a into kubeedge:main Oct 31, 2024
13 checks passed

MooreZheng mentioned this pull request Dec 6, 2024

Misalignment of the Hard Example Mining interface with Sedna #136

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

FuryMartin commented Sep 23, 2024

FuryMartin commented Sep 23, 2024 •

edited

Loading

FuryMartin commented Sep 26, 2024 •

edited

Loading

MooreZheng left a comment

hsj576 left a comment

FuryMartin commented Oct 28, 2024

hsj576 commented Oct 28, 2024

MooreZheng left a comment

FuryMartin commented Oct 29, 2024

MooreZheng left a comment

FuryMartin commented Oct 30, 2024 •

edited

Loading

MooreZheng commented Oct 30, 2024

Update at 2024-10-30 21:18

hsj576 commented Oct 30, 2024

MooreZheng left a comment

FuryMartin commented Oct 30, 2024

FuryMartin commented Oct 30, 2024 •

edited

Loading

MooreZheng commented Oct 30, 2024

MooreZheng left a comment •

edited

Loading

FuryMartin commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng left a comment

kubeedge-bot commented Oct 31, 2024

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

OSPP: Cloud-edge collaborative inference for LLM based on KubeEdge-Ianvs #149

Conversation

FuryMartin commented Sep 23, 2024

FuryMartin commented Sep 23, 2024 • edited Loading

Changes

Ianvs

Core

Denpendencies

Example

Sedna

FuryMartin commented Sep 26, 2024 • edited Loading

MooreZheng left a comment

Choose a reason for hiding this comment

hsj576 left a comment

Choose a reason for hiding this comment

FuryMartin commented Oct 28, 2024

hsj576 commented Oct 28, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

FuryMartin commented Oct 29, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

FuryMartin commented Oct 30, 2024 • edited Loading

Update at 2024-10-30 21:18

MooreZheng commented Oct 30, 2024

Update at 2024-10-30 21:18

hsj576 commented Oct 30, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

FuryMartin commented Oct 30, 2024

FuryMartin commented Oct 30, 2024 • edited Loading

Reason

MooreZheng commented Oct 30, 2024

MooreZheng left a comment • edited Loading

Choose a reason for hiding this comment

FuryMartin commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng commented Oct 31, 2024

MooreZheng left a comment

Choose a reason for hiding this comment

kubeedge-bot commented Oct 31, 2024

FuryMartin commented Sep 23, 2024 •

edited

Loading

FuryMartin commented Sep 26, 2024 •

edited

Loading

FuryMartin commented Oct 30, 2024 •

edited

Loading

FuryMartin commented Oct 30, 2024 •

edited

Loading

MooreZheng left a comment •

edited

Loading