Merge pull request #11 from DIDSR/LOA

Loa

README.md

@@ -1,4 +1,4 @@
-<h3>README for iMRMC and iRoeMetz</h3>
+# README for iMRMC and iRoeMetz
 
 Here is a high-level overview of the iMRMC software:
 
@@ -7,39 +7,31 @@ Here is a high-level overview of the iMRMC software:
 
 Examples of specific MRMC analyses and other FAQs are available in the [iMRMC repository's Wiki](https://github.com/DIDSR/iMRMC/wiki).
 
-Visit the <a href="https://github.com/DIDSR/iMRMC/releases" rel="nofollow">release Page</a> to download the software or access the repository [HERE](https://github.com/DIDSR/iMRMC).
+The R package is available from [CRAN](https://cran.r-project.org/web/packages/iMRMC/index.html). You can also visit the <a href="https://github.com/DIDSR/iMRMC/releases" rel="nofollow">release Page</a> to download the R package (source tar.gz file and binary zip file), the zip and tar.gz files of the entire repository, or the iMRMC java app (jar file).
 
-The documentation for both packages can be found <a href="http://didsr.github.io/iMRMC/" rel="nofollow">here.</a>
+Documentation of the java app can be found here: http://didsr.github.io/iMRMC/.
 
-Here is a summary of updates to [iMRMC](https://github.com/DIDSR/iMRMC/blob/master/UPDATES_iMRMC.md) and a summary of updates to [iRoeMetz](https://github.com/DIDSR/iMRMC/blob/master/UPDATES_iRoeMetz.md).
+Here are summaries of updates to the now frozen java apps:  [iMRMC](https://github.com/DIDSR/iMRMC/blob/master/UPDATES_iMRMC.md) and [iRoeMetz](https://github.com/DIDSR/iMRMC/blob/master/UPDATES_iRoeMetz.md).
 
 Please check out a statement of the licenses related to our software [here](LICENSE.md).
 
-<h5>iMRMC</h5>
+## iMRMC
 
 Desc: The primary objective of the iMRMC application is to assist investigators with analyzing and sizing multi-reader multi-case (MRMC) reader studies that compare the difference in the area under Receiver Operating Characteristic curves (AUCs) from two modalities. The core elements of this application include the ability to perform MRMC variance analysis and the ability to size an MRMC trial.
 
  * The core iMRMC application is a stand-alone, precompiled, license-free Java applications and the source code. It can be used in GUI mode or on the command line.
  * There is also an R package that utilizes the core Java application. Examples for using the programs can be found in the R help files.
 
 Here are some ways to cite our work:
+* FDA/CDRH, “iMRMC: Software for the Statistical Analysis of multi-reader multi-case studies,” *RST Catalog*, 2022, https://doi.org/10.5281/ZENODO.6628838.
 * Gallas, Brandon D., Andriy Bandos, Frank Samuelson, and Robert F. Wagner. “A Framework for Random-Effects ROC Analysis: Biases with the Bootstrap and Other Variance Estimators.” Commun Stat A-Theory 38, no. 15 (2009): 2586–2603. https://doi.org/10.1080/03610920802610084.
 * Gallas, Brandon D. “One-Shot Estimate of MRMC Variance: AUC.” Acad Radiol 13, no. 3 (2006): 353–62. https://doi.org/10.1016/j.acra.2005.11.030.
 
-To cite the java code, please use this (with the appropriate version number):
-* Gallas, Brandon D., "IMRMC-java vX.XX: Application for Analyzing and Sizing MRMC Reader Studies." Silver Spring, MD, 2017. https://github.com/DIDSR/iMRMC/releases.
 
-To cite the R package, please use this (version number depends on whether you downloaded the code from GitHub or CRAN):
-* Gallas, Brandon D., "IMRMC-R vX.XX: Application for Analyzing and Sizing MRMC Reader Studies." Silver Spring, MD, 2017. https://cran.r-project.org/web/packages/iMRMC/index.html.
-
-<h5>iRoeMetz</h5>
+## iRoeMetz
 
 Desc: The iRoeMetz application can be used to simulate the reader scores for MRMC experiments via Monte Carlo methods given variance components of the ROC scores. The application also estimates the variance components of AUC and can calculate the variance components directly with numerical integration. The simulated experiments can be saved and used in MRMC variance analysis programs, such as iMRMC. The iRoeMetz application is a stand-alone, precompiled, license-free Java applications and the source code.
 
-<h5>R package iMRMC</h5>
-
-Desc: The R package iMRMC executes the iMRMC program which writes the results to the local files system, it reads the analysis results from the local file system, packs the analysis results into a list object, deletes the data and analysis results from the local file system, and returns the list object.
-
 Here is one way to cite our work:
 * Gallas, Brandon D., and Stephen L. Hillis. “Generalized Roe and Metz ROC Model: Analytic Link between Simulated Decision Scores and Empirical AUC Variances and Covariances.” J Med Img 1, no. 3 (2014): 031006. https://doi.org/doi:10.1117/1.JMI.1.3.031006.
 

Rpackage/iMRMC/DESCRIPTION

@@ -1,7 +1,7 @@
 Package: iMRMC
 Type: Package
 Title: Multi-Reader, Multi-Case Analysis Methods (ROC, Agreement, and Other Metrics)
-Version: 2.0.0
+Version: 2.1.0
 Author: Brandon Gallas
 Maintainer: Brandon Gallas <[email protected]>
 Description: This software does Multi-Reader, Multi-Case (MRMC) analyses of data from imaging studies where clinicians (readers) evaluate patient images (cases). What does this mean? ... Many imaging studies are designed so that every reader reads every case in all modalities, a fully-crossed study. In this case, the data is cross-correlated, and we consider the readers and cases to be cross-correlated random effects. An MRMC analysis accounts for the variability and correlations from the readers and cases when estimating variances, confidence intervals, and p-values. The functions in this package can treat arbitrary study designs and studies with missing data, not just fully-crossed study designs. An overview of this software, including references presenting details on the methods, can be found here: <https://www.fda.gov/medical-devices/science-and-research-medical-devices/imrmc-software-do-multi-reader-multi-case-statistical-analysis-reader-studies>.

Rpackage/iMRMC/NEWS.md

@@ -11,6 +11,12 @@ A big thank you to all that have worked on this software throughout the years. W
 *	Yousun Ko          •	Rowhan Pathare  
 * Si Wen             •	Adam Wunderlich   
 
+# iMRMC 2.1.0
+*08/2024*
+- Add functions and tests to do MRMC analysis of limits of agreement `laWRWM`, `laWRBM`, `laBRWM`, `laBRBM`
+- Add function and test to simulate MRMC agreement data: `sim.NormalIG.Hierarchical`
+- Add scripts to demonstrate new simulation and analysis of MRMC agreement data in the `Rpackage/iMRMC/inst/extra/limitsOfAgreement` folder
+
 # iMRMC 2.0.0
 *04/2024*
 - Add R only functions to analyze MRMC studies, notably `doIMRMC()`  

Rpackage/iMRMC/R/limitsOfAgreement.R

@@ -1,32 +1,46 @@
+# limitsOfAgreement ##########
 #' @title MRMC Analysis of Limits of Agreement using ANOVA
 #' @description
-#' These four functions calculate four types of Limits of Agreement using ANOVA: Within-Reader Between-Modality(WRBM), 
-#' Between-Reader Within-Modality(BRWM), Within-Reader Within-Modality(WRWM) and Between-Reader Between-Modality(BRBM). 
+#' These four functions calculate four types of Limits of Agreement using ANOVA: 
+#' Within-Reader Within-Modality(WRWM), Between-Reader Within-Modality(BRWM),
+#' Within-Reader Between-Modality(WRBM), and Between-Reader Between-Modality(BRBM). 
 #' The 95\% confidence interval of the mean difference is also provided. If the study is fully crossed, the ANOVA 
-#' method are realized either by applying stats::aov or by matrix multiplication. Otherwise, the SS in ANOVA is 
-#' computed as the residual sum of squares of linear models. See more details below about the model structure.
+#' methods are realized either by applying \code{stats::aov} or by matrix multiplication. Otherwise, the SS in ANOVA are
+#' computed as residual sums of squares of linear models. See details below about the model structure
+#' and these references.
+#' 
+#' \itemize{
+#'   \item S. Wen and B. D. Gallas,
+#'     “Three-Way Mixed Effect ANOVA to Estimate MRMC Limits of Agreement,”
+#'     \emph{Statistics in Biopharmaceutical Research}, \strong{14}, pp. 532–541, 2022,
+#'     \url{https://www.doi.org/10.1080/19466315.2022.2063169}.
+#'   \item S. Wen and B. D. Gallas,
+#'     “Expanding to Arbitrary Study Designs: ANOVA to Estimate Limits of Agreement for MRMC Studies,”
+#'     \emph{arXiv}, 2023, \url{https://www.doi.org/10.48550/ARXIV.2312.16097}.
+#' }
 #'
 #' @details
-#' Suppose the score from reader j for case k under modality i is\eqn{X_{ijk}}, then the difference score from the
-#' same reader for the same cases under two different modalities is \eqn{Y_{jk} = X_{1jk} - X_{2jk}}.
+#' Suppose the score from a reader j for case k under modality \eqn{i} is\eqn{X_{ijk}}, then the difference score from the
+#' same reader for the same case under two different modalities is \eqn{Y_{jk} = X_{1jk} - X_{2jk}}.
 #' \itemize{
 #'   \item\code{laWRBM} use two-way random effect ANOVA to analyze the difference scores \eqn{Y_{jk}}. The model
 #'   is \eqn{Y_{jk}=\mu + R_j + C_k + \epsilon_{jk}}, where \eqn{R_j} and \eqn{C_k} are random effects for readers
-#'   and cases. The variance of mean and individual observation is expressed as the linear combination of the MS
+#'   and cases. The variances of mean and individual observations are expressed as linear combinations of the MS
 #'   given by ANOVA.
-#'   \item\code{laBRWM} use two-way random effect ANOVA to analyze the scores \eqn{X_{ijk}} for a single modality i. 
+#'   \item\code{laBRWM} use two-way random effect ANOVA to analyze the scores \eqn{X_{jk}} for a single modality. 
 #'   The model is \eqn{X_{jk}=\mu + R_j + C_k + \epsilon_{jk}}, where \eqn{R_j} and \eqn{C_k} are random effects 
-#'   for readers and cases. The variance of mean and individual observation is expressed as the linear combination 
+#'   for readers and cases. The variances of mean and individual observations are expressed as linear combinations 
 #'   of the MS given by ANOVA.
 #'   \item\code{laWRWM} use two-way random effect ANOVA to analyze the difference scores \eqn{Y_{jk}} from the same 
-#'   reader for the same cases under the same modality with different replicates \eqn{Y_{jk} = X_{ijk1} - X_{ijk2}}. 
+#'   reader for the same cases under the same modality with different replicates \eqn{Y_{jk} = X_{jk1} - X_{jk2}}. 
 #'   The model is \eqn{Y_{jk}=\mu + R_j + C_k + \epsilon_{jk}}, where \eqn{R_j} and \eqn{C_k} are random effects for 
-#'   readers and cases. The variance of mean and individual observation is expressed as the linear combination of 
+#'   readers and cases. The variances of mean and individual observations are expressed as  linear combinations of 
 #'   the MS given by ANOVA.
 #'   \item\code{laBRBM} use three-way mixed effect ANOVA to analyze the scores \eqn{X_{ijk}}. The model is given by
 #'   \eqn{X_{ijk}=\mu + R_j + C_k + m_i + RC_{jk} + mR_{ij} + mC_{ik} + \epsilon_{ijk}}, where \eqn{R_j} and
-#'   \eqn{C_k} are random effects for readers and cases and \eqn{m_i} is a fixed effect for modality. The variance
-#'   of mean and individual observation is expressed as the linear combination of the MS given by ANOVA.
+#'   \eqn{C_k} are random effects for readers and cases, \eqn{m_i} is a fixed effect for modality, and the other terms
+#'   are interaction terms. The variances of mean and individual observations are expressed as linear combinations
+#'   of the MS given by ANOVA.
 #' }
 #'
 #' @name limitsOfAgreement
@@ -41,36 +55,43 @@
 #'     The caseID is treated as a random effect.}
 #'   \item{modalityID}{The factor corresponding to the different modalities in the study.
 #'     The modalityID is treated as a fixed effect.}
-#'   \item{score}{The score given by the reader to the case for the modality indicated.}
+#'   \item{score}{The number (observation) given by the reader to the case for the modality indicated.}
 #' }
 #'
 #' @param modalitiesToCompare
-#' The factors identifying the modalities to compare. It should be at length 2. Default
-#' \code{modalitiesToCompare = c("testA","testB")}
+#' The factors identifying the modalities to compare. It should be length 2.
+#' Default = \code{c("testA","testB")}
 #' 
 #' @param replicatesToCompare
-#' The factors identifying the replicates to compare for laWRWM. It should be at length 2. Default
-#' \code{modalitiesToCompare = c("testA","testB")}
+#' The factors identifying the replicates to compare for \code{laWRWM}. It should be length 2.
+#' Default = \code{c("testA","testB")}
 #' 
 #' @param modality
-#' The factor identifying the modality for laBRWM. It should be at length 1. Default
-#' \code{modality = c("testA")}
+#' The factor identifying the modality for laBRWM. It should be length 1.
+#' Default = \code{modality = c("testA")}
 #'
 #' @param keyColumns
-#' Identify the factors corresponding to the readerID, caseID, modalityID, and score
-#' (or alternative random and fixed effects). Default \code{keyColumns = c("readerID", "caseID",
-#' "modalityID", "score")}
+#' Identify the factors corresponding to the readerID (random effect), caseID (random effect),
+#' modalityID (fixed effect), and score (observation).
+#' Default = \code{c("readerID", "caseID", "modalityID", "score")}
 #'
 #' @param if.aov
-#' Boolean value to determine whether using aov function to do ANOVA for the fully cross study only. 
-#' Default \code{if.aov = TRUE}
+#' Boolean value to determine whether to use the `stats::aov` function or to
+#' calculate the ANOVA statistics explicitly. `stats::aov` is only appropriate
+#' for fully-crossed study only. This flag permits head-to-head comparisons of
+#' the output from `stats::aov` and the explicit calculations.
+#' Default = \code{TRUE}
 #' 
 #' @param is.sparseQR 
-#' Boolean value to determine whether using sparseQR function to do QR decomposition when the study is not fully crossed. 
-#' Default \code{is.sparseQR = TRUE}
+#' Boolean value to determine whether the `base::qr` function assumes the input
+#' data is sparse or not. 
+#' Default = \code{TRUE}
 #' 
 #' @param type
-#' Identify which type of SS is computed in ANOVA. The possible values are c(1,2,3) 
+#' Identify how SS are computed in ANOVA for unbalanced study designs.
+#' The possible values are c(1,2,3), corresponding to the approaches
+#' introduced in the SAS package(Langsrud2003_Stat-Comput_v13p163).
+#' 
 #' Default \code{type= 1}
 #' 
 #' @param reader.first
@@ -79,29 +100,36 @@
 #' 
 #'
 #' @return
-#' A list of two dataframes 
-#' The first dataframe is \code{limits.of.agreement} with one row. Each column is as following:
+#' A list of two dataframes.
+#' 
+#' The first dataframe is \code{limits.of.agreement}. It has one row. Each column is as follows:
 #' \describe{
-#'   \item{meanDiff}{The mean of difference score.}
-#'   \item{var.MeanDiff}{The variance of mean difference score}
-#'   \item{var.1obs}{The variance of a single WRBM/BRBM difference score}
+#'   \item{meanDiff}{The mean difference score.}
+#'   \item{var.MeanDiff}{The variance of the mean difference score.}
+#'   \item{var.1obs}{The variance of the difference score.}
 #'   \item{ci95meanDiff.bot}{Lower bound of 95\% CI for the mean difference score. \code{meanDiff+
 #'   1.96*sqrt(var.MeanDiff)}}
 #'   \item{ci95meanDiff.top}{Upper bound of 95\% CI for the mean difference score. \code{meanDiff-
 #'   1.96*sqrt(var.MeanDiff)}}
-#'   \item{la.bot}{Lower bound of WRBM/BRBM Limits of Agreement. \code{meanDiff+1.96*sqrt(var.1obs)}}
-#'   \item{la.top}{Upper bound of WRBM/BRBM Limits of Agreement. \code{meanDiff-1.96*sqrt(var.1obs)}}
+#'   \item{la.bot}{Lower Limit of Agreement for the difference score. \code{meanDiff+1.96*sqrt(var.1obs)}}
+#'   \item{la.top}{Upper Limit of Agreement for the difference score. \code{meanDiff-1.96*sqrt(var.1obs)}}
 #' }
-#' The second dataframe is \code{two.way.ANOVA} or \code{three.way.ANOVA} showing the degree of freedom, 
-#' sum of squares, and variance component estimates for different sources of variation
+#' 
+#' The second dataframe is \code{two.way.ANOVA} or \code{three.way.ANOVA} shows the degrees of freedom, 
+#' sums of squares, and estimates of variance components for each source of variation
 #'
 #'
 #' @importFrom stats aov var qnorm model.matrix
 #' @importFrom Matrix Matrix tcrossprod
 #' @export
 #'
 #' @examples
+#' # Initialize the simulation configuration parameters
 #' config <- sim.NormalIG.Hierarchical.config(modalityID = c("testA", "testB"))
+#' 
+#' # Initizlize the seed and stream of the random number generator
+#' init.lecuyerRNG()
+#' 
 #' # Simulate an MRMC ROC data set
 #' dFrame <- sim.NormalIG.Hierarchical(config)
 #'