Treat a study that compares a single reader against multiple readers

[GoogleCodeExporter]

Please describe the desired enhancement

I am a student from ****** ******. I downloaded your iMRMC software and would 
like to use it to analyze a dataset. My dataset includes 10 readers, 1 modality 
and the ratings for about 100 cases. The first reader is CAD and the rest nine 
readers are radiologists. My study is about the comparison between CAD and the 
average performance of all radiologists. 

Could you please tell me if it is possible to perform such kind of analysis 
using the iMRMC software?

Thank you very much!

Best regards,

****** ******

Original issue reported on code.google.com by Brandon.Gallas on 10 Feb 2014 at 3:20

[GoogleCodeExporter]

Original comment by Brandon.Gallas on 27 Feb 2014 at 8:24

• Changed state: AssignTo_BDG

[marta-pinto]

Hi! I downloaded the iMRMC 4.0.3 software and I am performing the same type of analysis between a CAD system and radiologist readers. The problem is that for the readers I have a scale of 1-100 and for the CAD system I have a scale from 1-10 (with decimal values as scores, such as 0.41, 1,23, etc.).
Therefore, I end up with a ROC curve for readers with multiple points but for the CAD system I have only 10 points in the ROC curve, even though I am using decimal values that should give me a bigger set of point in the curve. Am I doing something wrong? Thank you very much for your attention.

[brandon-gallas]

iMRMC analyzes numeric data. It will see two numbers as distinct if they are different, no matter the decimal place, 5.01 > 5.001. You may get 10 points (not counting horizontal and vertical segments) on the ROC curve if your data bunches into groups when sorted by score (10 signal-present in a row, 15 signal-absent in a row, etc.). It's hard for me to say more without seeing the data. Try plotting the data. Let x be the score and y be 1 for signal-present and 2 for signal absent.

[marta-pinto]

I am attaching a picture with the two ROC curves I get (one for a reader and another for the CAD system) and also some examples of the scores I used. Could you see if this helps to know why I get approximately 10 points in the AI ROC curve and more points in the readers ROC curve? Thank you very much for the fast answer!

[brandon-gallas]

This does not help.
Try plotting the data as symbols.
Let x be the scores and y be the truth.
I've attached a sample figure below.

[marta-pinto]

So, like this?

[brandon-gallas]

OK. It's not entirely clear, but it looks like your CAD algorithm yields about 19 scores for signal-present cases (or there are only 19 signal-present cases). It looks like your algorithm probably outputs a lot of ties. In your previous image, you can see that 10.0 appears twice and 9.99 appears twice. There must be a lot of those because the ROC curve jumps from (0,0) to approximately (0.13, 0.81). There are no operating points in between because there are no signal-absent scores in between.
I don't think you are doing anything wrong. Your data has ties.

[marta-pinto]

Ok, thank you for helping me understand!
I will check how I can change the CAD algorithm part.