Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Project revamp and docker integration #251

Merged
merged 30 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
744c649
added zipfile functionality, seperated adding input files from runnin…
Triver1 Apr 27, 2024
3de42d8
temp commit
Triver1 May 2, 2024
9b9151e
temp commit
Triver1 May 2, 2024
7f946da
Revamped Tests, not using files anymore but now using text for the te…
Triver1 May 5, 2024
355aa0f
Added test_finished functionality
Triver1 May 5, 2024
7184232
Merge branch 'development' into docker_integration_test_revamp
Triver1 May 5, 2024
1b19348
Fixed copy test to work with revamp
Triver1 May 5, 2024
25c4a1a
Delete backend/app/d:\test.zip
Aqua-sc May 6, 2024
6021861
small changes to checks and deleting
Aqua-sc May 6, 2024
e8ef87e
small update to checks
Aqua-sc May 6, 2024
1c840ca
PR feedback commit
Triver1 May 7, 2024
8f04b8d
added docer-test-state to database
Triver1 May 7, 2024
b65e433
small fixes
Aqua-sc May 7, 2024
ad2ee51
Updates too checks
Aqua-sc May 7, 2024
7a54c9c
PR feedback commit
Triver1 May 8, 2024
e306fc3
Made image uninstall happen threaded. Changed validTemplate so at lea…
Triver1 May 8, 2024
6aa5957
fixed artifacts not wokring
Triver1 May 8, 2024
adff602
simpele docker testing seems to work now
Aqua-sc May 9, 2024
133e658
Merge branch 'project_revamp_docker_integration' of https://github.co…
Aqua-sc May 9, 2024
7121013
Fixed template test and simple test 😎
Triver1 May 9, 2024
b0377a9
added database to script
Triver1 May 9, 2024
d669531
restructured testfeedback responses
Aqua-sc May 9, 2024
fd00a86
Added route to download artifacts
Aqua-sc May 9, 2024
5e4d1ee
fix zo structuretemplate gets saved correctly
Aqua-sc May 9, 2024
c35e8cb
fixed template validation
Triver1 May 9, 2024
59c24ca
added route to fetch all test related files
Aqua-sc May 9, 2024
b672fda
final fixes
Aqua-sc May 9, 2024
cc22cdd
tests should succeed now
Aqua-sc May 10, 2024
5e1b86a
maybe tests work now
Aqua-sc May 10, 2024
1630a15
hopefully files are excluded properly right now
Aqua-sc May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added backend/app/d:\test.zip
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
import com.ugent.pidgeon.model.json.GroupJson;
import com.ugent.pidgeon.model.json.LastGroupSubmissionJson;
import com.ugent.pidgeon.model.json.SubmissionJson;
import com.ugent.pidgeon.model.submissionTesting.DockerOutput;
import com.ugent.pidgeon.model.submissionTesting.DockerSubmissionTestModel;
import com.ugent.pidgeon.model.submissionTesting.DockerTestOutput;
import com.ugent.pidgeon.model.submissionTesting.SubmissionTemplateModel;
import com.ugent.pidgeon.postgre.models.*;
import com.ugent.pidgeon.postgre.models.types.UserRole;
Expand All @@ -30,7 +33,7 @@
import java.util.zip.ZipFile;

@RestController
public class SubmissionController {
public class SubmissionController {

@Autowired
private GroupRepository groupRepository;
Expand Down Expand Up @@ -58,20 +61,52 @@ public class SubmissionController {


private SubmissionTemplateModel.SubmissionResult runStructureTest(ZipFile file, TestEntity testEntity) throws IOException {
// Get the test file from the server
FileEntity testfileEntity = fileRepository.findById(testEntity.getStructureTestId()).orElse(null);
if (testfileEntity == null) {
// There is no structure test for this project
if(testEntity.getStructureTemplate() == null){
return null;
}
String testfile = Filehandler.getStructureTestString(Path.of(testfileEntity.getPath()));
String structureTemplateString = testEntity.getStructureTemplate();

// Parse the file
SubmissionTemplateModel model = new SubmissionTemplateModel();
model.parseSubmissionTemplate(testfile);

model.parseSubmissionTemplate(structureTemplateString);
return model.checkSubmission(file);
}

private DockerOutput runDockerTest(ZipFile file, TestEntity testEntity, Path outputPath) throws IOException {

// Get the test file from the server
String testScript = testEntity.getDockerTestScript();
String testTemplate = testEntity.getDockerTestTemplate();
String image = testEntity.getDockerImage();

// The first script must always be null, otherwise there is nothing to run on the container
if(testScript == null){
return null;
}

// Init container and add input files
DockerSubmissionTestModel model = new DockerSubmissionTestModel(image);
model.addZipInputFiles(file);

// Copy artifacts to the destination
List<File> artifacts = model.getArtifacts();

// filehandler copy zips
Filehandler.copyFilesAsZip(artifacts, outputPath);

// cleanup docker
model.cleanUp();

if(testTemplate == null){
// This docker test is configured in the simple mode (store test console logs)
return model.runSubmission(testScript);
}else{
// This docker test is configured in the template mode (store json with feedback)
return model.runSubmissionWithTemplate(testScript, testTemplate);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ksnap nie helemaal hoe de artifacts gekopieërd worden voordat het script gelopen wordt

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad dit is een logic error. Het kopieeren gebeurd 5 lines erboven maar mijn order of execution was verkeerd.

}
}

/**
* Function to get a submission by its ID
*
Expand Down Expand Up @@ -170,7 +205,6 @@ public ResponseEntity<?> submitFile(@RequestParam("file") MultipartFile file, @P

long groupId = checkResult.getData();

//TODO: execute the docker tests onces these are implemented
try {
//Save the file entry in the database to get the id
FileEntity fileEntity = new FileEntity("", "", userId);
Expand Down Expand Up @@ -203,24 +237,39 @@ public ResponseEntity<?> submitFile(@RequestParam("file") MultipartFile file, @P

// Run structure tests
TestEntity testEntity = testRepository.findByProjectId(projectid).orElse(null);
SubmissionTemplateModel.SubmissionResult testresult;
SubmissionTemplateModel.SubmissionResult structureTestResult;
DockerOutput dockerOutput;
if (testEntity == null) {
Logger.getLogger("SubmissionController").info("no test");
testresult = new SubmissionTemplateModel.SubmissionResult(true, "No structure requirements for this project.");
Logger.getLogger("SubmissionController").info("no tests");
submission.setStructureFeedback("No specific structure requested for this project.");
submission.setStructureAccepted(true);
} else {
testresult = runStructureTest(new ZipFile(savedFile), testEntity);
}
if (testresult == null) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("Error while running tests: test files not found");

// Check file structure
structureTestResult = runStructureTest(new ZipFile(savedFile), testEntity);
if (structureTestResult == null) {
submission.setStructureFeedback(
"No specific structure requested for this project.");
submission.setStructureAccepted(true);
} else {
submission.setStructureAccepted(structureTestResult.passed);
submission.setStructureFeedback(structureTestResult.feedback);
}
// Check if docker tests succeed
dockerOutput = runDockerTest(new ZipFile(savedFile), testEntity, Filehandler.getSubmissionPath(projectid, groupId, submission.getId()));
if (dockerOutput == null) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body("Error while running docker tests.");
}
// Representation of dockerOutput, this will be a json(easily displayable in frontend) if it is a template test
// or a string if it is a simple test
submission.setDockerFeedback(dockerOutput.toString());
submission.setDockerAccepted(dockerOutput.isAllowed());
}
submission.setTestFinished(true);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dit is niet helemaal hoe ik het in gedachten had. Dit heeft nu vrij weinig nut want op het moment dat de frontend de response krijgt is de volledige dockertest dus al gerund en zal dit veld dus altijd true zijn. Tis eerder de bedoeling dat er een reponse wordt teruggegeven waar dit dus default op false staat en dat dan in de achtergrond de dockertesten gerunt worden & dit veld geupdate wordt. Op deze manier kan de user al feedback krijgen over de structure test terwijl de docker testen aan het runnen zijn.

submissionRepository.save(submissionEntity);
// Update the submission with the test resultsetAccepted
submission.setStructureAccepted(testresult.passed);
// Update the dataabse
submission = submissionRepository.save(submission);

// Update the submission with the test feedbackfiles
submission.setDockerFeedback("TEMP DOCKER FEEDBACK");
submission.setStructureFeedback(testresult.feedback);
submissionRepository.save(submission);

return ResponseEntity.ok(entityToJsonConverter.getSubmissionJson(submissionEntity));
Expand Down
Loading
Loading