Validate Split-Plot analysis

[qigongFDA]

Input file 12 x 7 x 3
Raw file: https://github.com/DIDSR/iMRMC/tree/gh-pages/000_iRoeMetz/inputFiles/RMconfigurations. (12)
Number of readers: 6,7,8,9,10,11,12 (7)
Study Design: fully crossed, Split-plot ng: 2, 3 (3)

Command line execution:
Use command line run the software

1. function (inputFile)
2. Run the function called by the " Perform Simulation Experiments" button
3. Run the export button for simulation data
4. Do the same procedure fir numerical calculations.

[qigongFDA]

Create a new class validateFunction in roemetz.core package to do the validation. validateFunction could call "simulation", "numerical", and "export" buttons functions to do the analysis and export. In order to call these functions from other class, we move out them from button listener.

Create a variable doValidation in Roemetz.java class to avoid display windows during validation.

Previous, both simulation and numerical are working in parallel. This will make a problem in validation. In validation, program will try do export result before parallel analysis done. Thus, we have to figure out a method to wait analysis done before export result.

1. Use while loop and Thread.sleep() function to do it. But this might consume CUP.
2. Delete parallel part and let program run in sequential. This will slow down the software when we do big analysis.
3. Call export result function after (parallel) process result function. This might make code hard to understand, and the analysis and export must be the last analysis.

[brandon-gallas]

Good comments here.

From: Qi Gong [mailto:[email protected]]
Sent: Wednesday, January 13, 2016 5:57 PM
To: DIDSR/iMRMC
Subject: Re: [iMRMC] Validate Split-Plot analysis (#81)

Create a new class validateFunction in roemetz.core package to do the validation. validateFunction could call "simulation", "numerical", and "export" buttons functions to do the analysis and export. In order to call these functions from other class, we move out them from button listener.

Create a a variable doValidation in Roemetz.java class to avoid display windows during validation.

Previous, both simulation and numerical are working in parallel. This will make a problem in validation. In validation, program will try do export result before parallel analysis done. Thus, we have to figure out a method to wait analysis done before export result.

1. Use while loop and Thread.sleep() function to do it. But this might consume CUP.
2. Delete parallel part and let program run in sequential. This will slow down the software when we do big analysis.
3. Call export result function after (parallel) process result function. This might make code hard to understand, and the analysis and export must be the last analysis.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/81#issuecomment-171464156.

[qigongFDA]

1. parallel problem
   We use the second and the third method together to solve the problem.
   Numerical: we delete parallel part let program run in sequential (Numerical part doesn't have big calculation, sequential will not affect speed)
   Simulation: This part has big calculation, we need to keep parallel processing. Thus, we call export result function after (parallel) process result function. And if end user do validation, Software will do Simulation after Numerical and automatically exit after export simulation( this could guarantee simulation is the last analysis)
2. Run in command line
   In windows, we could use ( java -jar iRoeMetz.jar "input irm file path and name" ) to run the software.
   We create a bat file to run analysis all file one by one (Can't use Start to run in parallel in windows)
   252 10 trails input files analysis should take about 7- 8 min
3. doValidation Variable
   It could control whether display all GUI windows and automatically export simulation
4. input file (irm)
   Create a new project to generate different #readers and #split-plot input file base on files in GITHUB.
   If input file doesn't have study design information, software will automatically set them to 1-yes-yes-yes
5. Export result
   Export iRoeMetz parameter setting for Simulation and Numerical. The parameter is in the first part of the export file
6. Save validation result
   In validation, software will create a folder "output" in input file directory (if "output" folder is not exist), and save output files in it. Output file name's formation is "input file name + NumericalOutput/SimulationOutput
7. Not fully crossed study
   Solve the bug for not fully crossed study analysis: Don't do Hillis calculation for not fully crossed study.

[brandon-gallas]

Can we set the validation to use the MLE variance estimate?
I would like to characterize that too.

[brandon-gallas]

Here's the resources from our other project that you should use for this project.

https://github.com/DIDSR/mitoticCountSimulation/blob/gh-pages/000_resources/20160227agreementSPIE/SPIEMI2016agreementGallasProc.pdf

https://github.com/DIDSR/mitoticCountSimulation/tree/bdg/src/000output/SPIEMI2016

Thanks!

[qigongFDA]

Export reuslt in “(U-statistics, Not MLE) / (MLE, Not U-statistics)” format.

Enable MLE analysis in iRoemetz simulation analysis. But still have problem with MC variance and Numerical analysis.

[qigongFDA]

Next steps for validation.

1. Move total var from table part to statpanel and export it.
2. import / export RNG, # of trails and MLE from / to .irm file and set default to 123456/10/unMLE
3. Build code to generate simulationOutputSummary.csv, NumericalOutputSummary.csv and simulationConfig.csv files. They are summary of all studies. Each row records one study's information.
   a. simulationConfig.csv records each study's setting.
   b. simulationOutputSummary.csv, NumericalOutputSummary.csv record results. And they should have the same order of columns.
   We should run this code after finishing all studies.
   We might need to rewrite simulation and numerical result files. That could help us to extract data

[qigongFDA]

MLE should affect MC variance.

We might need add new variable varAUC_A and varAUC_B in DBRecord to record readers's AUC variance in each modality.

simulationOutputSummary.csv should has 12 colums:
1-6:
mcMean of ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB,
7-12
mcVar of ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB,

numericalOutputSummary.csv should has 6 colums:
ACU_A, AUC_B, AUC_AminusB, varAUC_A, varAUC_B, varAUC_AminusB,

[qigongFDA]

Copy the comment from issue 91
We could use swarm command to run iRoeMetz validation parallel on linux. (i.e. swarm doValidation.txt)

There are two things we need to pay attention:

1. We should run swarm in perceus but not login to one of the node.
2. The txt file should in following format: export DISPLAY=":0.0"; java -jar iRoeMetz.jar /raidb/qigong/imrmc/001.irm As our iRoeMetz has gui (we just don't display it in validation), and some node might not connect to a screen, we need to set a number to DISPLAY. Otherwise, system will give us this error. "No X11 DISPLAY variable was set, but this program performed an operation which requires it."

[qigongFDA]

In validation we should avoid software display any warning. Because warning needs user to click "OK" button. It will stop validation.

[qigongFDA]

In validation, we are not only export mc simulation and numerical results. We also generate trialResultArray to save all trails simulation results, and export it to .csv file.

[qigongFDA]

We should generate several plot by mc simulation and numerical output. The plot could help us analysis how configures affect mcmean, mcvar, RSE and reject.

[brandon-gallas]

You are close to resolving this issue. You should start to document how to run iRoeMetz in parallel. When ready, I will try and run iRoeMetz in parallel following the documentation (no one-on-one help from you). Then we can work together to improve the documentation and ultimately close this issue.

[qigongFDA]

Here is the draft of manual for validation.
Validation_manual.txt

[qigongFDA]

Update simulation and numerical output file. Now, both of them export all analysis results.

[qigongFDA]

Could we close this issue?

[qigongFDA]

Add sample validation input file in iMRMC/validation/

[brandon-gallas]

Move validation tools to the iMRMC repository and document.

[qigongFDA]

README.MD and READMEexample1.md files show how to run validation in Linux.

Do we need to put it in iMRMC or iRoeMetz manual? Both of them have description about command line running. I don't think we should put the validation instruction in the manual, as it is not useful for outside users.