Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WIP: Add more tests #301

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

WIP: Add more tests #301

wants to merge 10 commits into from

Conversation

kulcsaradam
Copy link
Contributor

This is just a demonstration of expanding the testing system of Walrus.
Add tests:

  • regression: random tests
  • malformed: incorrect syntax tests
  • standalone: feature tests (with heavy wasi utilization)
  • proposal_features: features that are only just proposed in some way (for example anyref and array type)

This will be changed to fit the right licenses, it is just a demonstration.

@zherczeg
Copy link
Collaborator

zherczeg commented Nov 8, 2024

Thank you! We can discuss what kind of tests we want to run. For example, malformed tests are testing wabt, do we need them?

@kulcsaradam kulcsaradam force-pushed the more_testing branch 3 times, most recently from e6a4980 to d024da4 Compare November 13, 2024 11:27
@clover2123
Copy link
Collaborator

Thanks for your great contribution!
Please consider the followings:

  • What about merging tools/run-tests-other.py into tools/run-tests.py file?
  • IMO we don't need to run tests for proposed-features

@kulcsaradam
Copy link
Contributor Author

Initially we just wanted the files to be seperated so it's easier to keep track of "other" tests but I will merge it if that is prefered.
Also if proposed-feature tests are not really needed then we can remove them, it was just in case if we implemented any of them then they could be tested instantly.

@kulcsaradam kulcsaradam force-pushed the more_testing branch 3 times, most recently from bfbedec to d149148 Compare December 4, 2024 14:31
matetokodi and others added 10 commits December 11, 2024 08:55
@matetokodi @kulcsaradam
WIP: add regression and some unit tests from WebAssembly Micro Runtim…
22059eb 
…e (WAMR)

Signed-off-by: Máté Tokodi [email protected]
@matetokodi @kulcsaradam
WIP: add regression tests from WebAssembly Micro Runtime (WAMR)
36cc226 
Signed-off-by: Máté Tokodi [email protected]
@kulcsaradam
WIP: Add test from WebAssembly Micro Runtime (WAMR)
7afd88c 
Signed-off-by: Ádám László Kulcsár <[email protected]>
@kulcsaradam
WIP: Merge branches
819116d 
Signed-off-by: Ádám Kulcsár <[email protected]>
@kulcsaradam
WIP: Modify test system structure
9dbd2e4 
Move wamr tests to 'tests/other' directory and restructure the wamr test runner into a standalone file.

Signed-off-by: Ádám Kulcsár <[email protected]>
@kulcsaradam
Delete proposal tests and merge run-tests.py and run-tests-other.py
432315b 
Signed-off-by: Ádám László Kulcsár <[email protected]>
@kulcsaradam
WIP: Add coremark as a program to test and a program builder script
ad1abe9 
The section 'target_features' needs to be implemented to work with wasi-clang
compilation.
Add build_programs.py.

Signed-off-by: Ádám László Kulcsár <[email protected]>
@matetokodi @kulcsaradam
WIP: add wamr_compilert test and dhrystone
2ec7428 
Signed-off-by: Máté Tokodi [email protected]
@kulcsaradam
WIP: Update coremark build scirpt and move executables to a different
64db275 
folder

Signed-off-by: Ádám László Kulcsár <[email protected]>
@kulcsaradam
Add c-ray to tests
8bb7f74 
Signed-off-by: Ádám László Kulcsár <[email protected]>
@kulcsaradam
Copy link
Contributor Author

@clover2123 We have been trying out things in this patch and started to wonder how should the tests be added.

Just to clarify all of these tests have been collected from wamr but most of the complicated programs are open source and we can compile them if we would like to.

But the important thing is:
Should we add tests separately or together as one big patch? (like this)
Which tests should we add? Currently, we are thinking about:

  • Coremark
  • Dhrystone
  • c-ray
    • the problem is that it's more of a demo, only benchmark we can do is measuring the render time
  • regression tests from wamr
    • these come handily because walrus still fails on some of them and they are actual errors most of the time and not Insufficient impport errors
  • unit tests from wamr
    • some of these test optional features that we have not implemented yet

We would not add tests that are still failing though.

Lastly, there are still some tests that we are looking at. Should we contiue looking into them and search for more or should we rather focus on merging these into walrus?

Also, should we maybe introduce a new CI run for more complicated tests that take more time?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ksh8281 ksh8281 Awaiting requested review from ksh8281 ksh8281 will be requested when the pull request is marked ready for review ksh8281 is a code owner

@clover2123 clover2123 Awaiting requested review from clover2123 clover2123 will be requested when the pull request is marked ready for review clover2123 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kulcsaradam @zherczeg @clover2123 @matetokodi