diff --git a/R/ci_for_2indep_prop.qmd b/R/ci_for_2indep_prop.qmd new file mode 100644 index 000000000..8825f2570 --- /dev/null +++ b/R/ci_for_2indep_prop.qmd @@ -0,0 +1,179 @@ +--- +title: "Confidence Intervals for Independent Proportions in R" +--- + +## Introduction + +\[See separate page for general introductory information on confidence intervals for proportions.\] + +\[Note: information about cicalc package will be added to this page soon.\] + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{r} +#| echo: false +#| include: false +library(tidyverse) +library(cardx) +library(DescTools) +adcibc2 <- read_csv("../data/adcibc.csv") + +adcibc <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 0, 1) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +act <- filter(adcibc, trt == "ACT") |> + select(respn) +act2 <- act$respn +``` + +The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders). + +```{r} +#| echo: false +adcibc |> + group_by(trt, resp) |> + tally() +``` + +## Packages + +**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for. + +See [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations. + +If calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector). + +Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. + +Methods included are \[TBC\] methods for 2 independent samples. + +**The {ratesci} package** is ... \[TBC\] + +**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail. + +**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios. + +## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples + +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions. + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx} + +For more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html). + +#### Example code + +`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied. + +Although this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method). + +Both the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables. + +The prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI. + +```{r} +indat1 <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 1, 0) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +# To get the comparison the correct way around Placebo must be 1, and Active 0 + +indat <- select(indat1, trtn, respn) + +cardx::ard_stats_prop_test( + data = indat, + by = trtn, + variables = respn, + conf.level = 0.95, + correct = FALSE +) +cardx::ard_stats_prop_test( + data = indat, + by = trtn, + variables = respn, + conf.level = 0.95, + correct = TRUE +) +``` + +### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools} + +For more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks. + +#### Example code + +With 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI. + +```{r} +count_dat <- indat |> + count(trtn, respn) +count_dat + +# BinomDiffCI requires +# x1 = successes in active, n1 = total subjects in active, +# x2 = successes in placebo, n2 = total subjects in placebo + +DescTools::BinomDiffCI( + x1 = 36, + n1 = 154, + x2 = 12, + n2 = 77, + conf.level = 0.95, + sides = c("two.sided"), + method = c( + "wald", + "waldcc", + "score", + "scorecc", + "ac", + "mn", + "mee", + "blj", + "ha", + "hal", + "jp" + ) +) +``` + +## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples + +\[TBC\] + +### + +## Methods for Calculating Confidence Intervals for Odds Ratio from 2 independent samples + +\[TBC\] + +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with hypothesis tests + +\[TBC\] - cf. chi-squared tests + +## References + +1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) +2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf) +3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. +4. diff --git a/R/ci_for_paired_prop.qmd b/R/ci_for_paired_prop.qmd new file mode 100644 index 000000000..6b409645f --- /dev/null +++ b/R/ci_for_paired_prop.qmd @@ -0,0 +1,134 @@ +--- +title: "Confidence Intervals for Paired Proportions in R" +--- + +## Introduction + +\[See separate page for general introductory information on confidence intervals for proportions.\] + +\[Note: information about cicalc package will be added to this page soon.\] + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{r} +#| echo: false +#| include: false +library(tidyverse) +library(cardx) +library(DescTools) +adcibc2 <- read_csv("../data/adcibc.csv") + +adcibc <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 0, 1) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +act <- filter(adcibc, trt == "ACT") |> + select(respn) +act2 <- act$respn +``` + +The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders). + +```{r} +#| echo: false +adcibc |> + group_by(trt, resp) |> + tally() +``` + +## Packages + +**The {ratesci} package** is ... \[TBC\] + +**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs). + +## Methods for Calculating Confidence Intervals for Proportion Difference from matched pairs using {ExactCIdiff} and {ratesci} + +For more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses. + +To date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above. + +**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\ +NOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf). + +Using a cross over study as our example, a 2 x 2 table can be formed as follows: + ++-----------------------+---------------+---------------+---------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+===============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+---------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+---------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+---------------+ + +: The proportions of subjects responding on each treatment are: + +Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ + +Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ + +Suppose : + ++-----------------------+---------------+---------------+------------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+==================+ +| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | ++-----------------------+---------------+---------------+------------------+ +| Active Response = No | t = 6 | u = 5 | t+u = 11 | ++-----------------------+---------------+---------------+------------------+ +| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | ++-----------------------+---------------+---------------+------------------+ + +Active: $\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\hat p_2= (r+t)/n$ = 26/46 =0.565 + +Difference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below + +-0.00339 to 0.38065 + +```{r} +#| eval: false +# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95) + +CI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI +CI +``` + +## Methods for Calculating Confidence Intervals for Relative Risk from matched pairs using {ratesci} + +\[TBC\] + +## Methods for Calculating Confidence Intervals for Conditional Odds Ratio from matched pairs using {ratesci} + +\[TBC\] + +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with Hypothesis Tests + +\[TBC\] - cf. McNemar test + +## + +## + +## References + +1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) +2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf) +3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. +4. diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index e015f8a45..8822671eb 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -1,24 +1,12 @@ --- -title: "Confidence Intervals for Proportions in R" +title: "Confidence Intervals for a Proportion in R" --- ## Introduction -The methods to use for calculating a confidence interval (CI) for a proportion depend on the type of proportion you have. +\[See separate page for general introductory information on confidence intervals for proportions.\] -- 1 sample proportion (1 proportion calculated from 1 group of subjects) - -- 2 sample proportions and you want a CI for the difference in the 2 proportions. - - - If the 2 samples come from 2 independent samples (different subjects in each of the 2 groups) - - - If the 2 samples are matched (i.e. the same subject has 2 results, one on each group \[paired data\]). - -The method selected is also dependent on whether your proportion is close to 0 or 1 (or near to the 0.5 midpoint), and your sample size. - -For more information about these methods, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^1^. - -Note: information about cicalc package will be added to this page soon. +\[Note: information about cicalc package will be added to this page soon.\] ## Data used @@ -67,6 +55,8 @@ If calculating the CI for a difference in proportions, the package requires both Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. +**The {ratesci} package** is ... \[TBC\] - note, current development version at has new features in the rateci() function (including more of the CI methods described below) compared to the CRAN release. + **The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. **The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. @@ -111,7 +101,7 @@ cardx::proportion_ci_clopper_pearson(act2, conf.level = 0.95) |> as_tibble() ``` -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) +### Normal Approximation Method (Also known as the Wald Method) In large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\alpha)\%$ confidence interval for population proportion can be calculated using normal approximation as follows: @@ -191,150 +181,13 @@ cardx::proportion_ci_jeffreys(act2, conf.level = 0.95) |> as_tibble() ``` -## Methods for Calculating Confidence Intervals for a matched pair proportion using {ExactCIdiff} - -For more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses. - -To date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above. - -**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\ -NOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf). - -Using a cross over study as our example, a 2 x 2 table can be formed as follows: - -+-----------------------+---------------+---------------+---------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+===============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+---------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+---------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+---------------+ - -: The proportions of subjects responding on each treatment are: - -Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ - -Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ - -Suppose : - -+-----------------------+---------------+---------------+------------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+==================+ -| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | -+-----------------------+---------------+---------------+------------------+ -| Active Response = No | t = 6 | u = 5 | t+u = 11 | -+-----------------------+---------------+---------------+------------------+ -| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | -+-----------------------+---------------+---------------+------------------+ - -Active: $\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\hat p_2= (r+t)/n$ = 26/46 =0.565 - -Difference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below - --0.00339 to 0.38065 - -```{r} -#| eval: false -# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95) - -CI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI -CI -``` - -## Methods for Calculating Confidence Intervals for 2 independent samples proportion - -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions. - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx} - -For more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html). - -#### Example code - -`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied. - -Although this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method). - -Both the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables. - -The prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI. - -```{r} -indat1 <- adcibc2 |> - select(AVAL, TRTP) |> - mutate( - resp = if_else(AVAL > 4, "Yes", "No"), - respn = if_else(AVAL > 4, 1, 0), - trt = if_else(TRTP == "Placebo", "PBO", "ACT"), - trtn = if_else(TRTP == "Placebo", 1, 0) - ) |> - select(trt, trtn, resp, respn) - -# cardx package required a vector with 0 and 1s for a single proportion CI -# To get the comparison the correct way around Placebo must be 1, and Active 0 - -indat <- select(indat1, trtn, respn) - -cardx::ard_stats_prop_test( - data = indat, - by = trtn, - variables = respn, - conf.level = 0.95, - correct = FALSE -) -cardx::ard_stats_prop_test( - data = indat, - by = trtn, - variables = respn, - conf.level = 0.95, - correct = TRUE -) -``` - -### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools} - -For more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks. +## Continuity Adjusted Methods -#### Example code +\[TBC\] -With 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI. +## Consistency with hypothesis tests -```{r} -count_dat <- indat |> - count(trtn, respn) -count_dat - -# BinomDiffCI requires -# x1 = successes in active, n1 = total subjects in active, -# x2 = successes in placebo, n2 = total subjects in placebo - -DescTools::BinomDiffCI( - x1 = 36, - n1 = 154, - x2 = 12, - n2 = 77, - conf.level = 0.95, - sides = c("two.sided"), - method = c( - "wald", - "waldcc", - "score", - "scorecc", - "ac", - "mn", - "mee", - "blj", - "ha", - "hal", - "jp" - ) -) -``` +\[TBC\] ## References diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd new file mode 100644 index 000000000..368378dc1 --- /dev/null +++ b/SAS/ci_for_2indep_prop.qmd @@ -0,0 +1,295 @@ +--- +title: "Confidence Intervals for Independent Proportions in SAS" +bibliography: references.bib +csl: nature.csl +execute: + eval: false +--- + +## Introduction + +This page covers confidence intervals for comparisons of two independent proportions in SAS, including the contrast parameters for risk difference (RD) $\theta_{RD} = p_1 - p_2$, relative risk (RR) $\theta_{RR} = p_1 / p_2$, and odds ratio (OR) $\theta_{OR} = p_1(1-p_2) / (p_2(1-p_1))$. + +See the [summary page](../method_summary/ci_for_prop_intro.html) for general introductory information on confidence intervals for proportions, including the principles underlying the most common methods. + +Caution is required if there are no responders (or all responders) in both groups, which might happen in a subgroup analysis for example. PROC FREQ (as of v9.4) does not output any confidence intervals in this case, when valid CIs can (and should) be reported for the RD contrast, since the dataset provides an estimate of zero for RD (and the confidence in the estimate is proportional to the sample size). Similarly, if $\hat p_1 = \hat p_2 = 1$ then an estimate and confidence interval can be obtained for RR, but not from PROC FREQ. + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{sas} +proc import datafile = 'data/adcibc.csv' + out = adcibc + dbms = csv + replace; + getnames = yes; + guessingrows = max; +run; + +data adcibc2 (keep=trt resp) ; + set adcibc; + if aval gt 4 then resp="Yes"; + else resp="No"; + if trtp="Placebo" then trt="PBO"; + else trt="Act"; +run; + +* Sort to ensure that the outcome of interest ("Yes" in this example) is first; +* when using default COLUMN=1 option in the TABLES statement; +proc sort data=adcibc2; +by trt descending resp; +run; +``` + +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558, giving a risk difference of 0.0779, relative risk 1.50, and odds ratio 1.6525. + +```{sas} +proc freq data=adcibc2; + table trt*resp/ nopct nocol; +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") +``` + +## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples + +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf) describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast, and/or when group sizes are imbalanced). + +SAS PROC FREQ is able to calculate CIs for RD using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. + +The SCAS method is not available in PROC FREQ, but can be produced using a SAS macro (`%SCORECI`) which can be downloaded from . + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) + +The difference between two independent sample proportions is calculated as: $\hat \theta_{RD} = \hat p_1 - \hat p_2 = x_1 / n_1 - x_2 / n_2$ + +The Wald CI for $\theta_{RD}$ is calculated using: + +$\hat \theta_{RD} \pm z_{\alpha/2} \times SE(\hat \theta_{RD})$, + +where $SE (\hat \theta_{RD}) = \sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ + +With continuity correction, the equation becomes + +$\hat \theta_{RD} \pm (CC + z_{\alpha/2} \times SE(\hat \theta_{RD}))$, + +where $CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ + +### Newcombe Method (Also known as the Hybrid Score method, Square-and-Add, or the Method of Variance Estimates Recovery (MOVER) ) + +Derive the confidence intervals for the separate proportions in each group, $p_1$ and $p_2$, using the Wilson Score Method equations as described [here](ci_for_prop.html). + +Let $l_1$ = Lower CI for sample 1, and $u_1$ be the upper CI for sample 1. + +Let $l_2$ = Lower CI for sample 2, and $u_2$ be the upper CI for sample 2. + +Let D = $\hat p_1 - \hat p_2$ (the difference between the observed proportions) + +The CI for $\theta_{RD}$ the difference between two proportions is: $$ D - sqrt((\hat p_1 - l_1)^2+(u_2 - \hat p_2)^2) \quad, \quad D + sqrt((\hat p_2 - l_2)^2 + (u_1 - \hat p_1)^2 ) $$ + +Note that earlier versions of SAS PROC FREQ (before implementation of the MN method) allowed the option CL=WILSON or CL=SCORE to produce this method, but it is not really a score method. As of SAS/STAT 15.4, the CL=WILSON option (undocumented) still gives the Newcombe interval, but CL=SCORE gives the Miettinen-Nurminen score method. + +### Miettinen-Nurminen, Mee and other Asymptotic Score Methods + +These truly are score methods, as they are based on an extension of the score methodology applied to two independent proportions, using the contrast function $S(\theta) = \hat p_1 - \hat p_2 - \theta$, which for any given value of $\theta$ has expectation zero, and variance $\{\tilde p_1 (1 - \tilde p_1)/n_1 + \tilde p_2 (1 - \tilde p_2)/n_2\} \times N/(N-1)$ where $N = n_1 + n_2$. + +The Mee variant of the method omits the $N/(N-1)$ variance bias correction factor (sometimes referred to as an $'N-1'$ correction). + +Gart and Nam derived a similar method, arrived at from a different underlying 'efficient score' methodology, so the formulae look different but are essentially equivalent to the Mee interval. They added a correction for skewness to improve one-sided coverage. The skewness correction was applied to the Miettinen-Nurminen formula for all contrast parameters by Laud, to give the SCAS method[@laud2017]. The SCAS method is available for SAS via the `%SCORECI` macro. + +### Agresti-Caffo Method + +Similar to the Agresti-Coull method for a single proportion, the Agresti-Caffo interval is designed to be an easily taught method to enable the CI to be calculated by hand. The formula involves simply adding one success and one failure to each sample, and then using the Wald formula. + +### 'Exact' Methods + +To obtain 'exact' confidence intervals, you need to add an `EXACT RISKDIFF` statement to the PROC FREQ. + +The current default method produced by PROC FREQ (`METHOD=SCORE`) is the Chan-Zhang variant. This is undoubtedly an improvement on the extremely over-conservative Santner-Snell method (obtained with the `RISKDIFF(METHOD=NOSCORE)` option in the EXACT statement), but is more computationally intensive, and can result in a Warning in the SAS log about long computation times. + +The third alternative 'exact' method by Agresti & Min (`RISKDIFF(METHOD=SCORE2)`) is less conservative, but is only guaranteed to achieve strictly conservative two-sided coverage, so is not appropriate for use in one-sided hypothesis testing. + +## Methods for Calculating Confidence Intervals for Relative Risk or Odds Ratio from 2 independent samples + +SAS PROC FREQ is able to calculate CIs for RR or OR using the following methods: Miettinen-Nurminen (SCORE), Likelihood Ratio (LR), Wald, Haldane Modified Wald. There is also a modified version of the MN method which omits the 'N-1' variance bias correction, using the `CL=(SCORE(CORRECT=NO))` option - note this is NOT referring to a 'continuity correction' (and also note that no indication is given in the output that this option has been applied). For OR, an additional `CL=MIDP` option is also available. + +For conservative coverage, the 'Exact' methods are provided. For RR this includes the 3 variants described above for RD, while only one version is given for OR. Run times for the `METHOD=SCORE` version for RR may be lengthy (but the faster alternative Santner-Snell version produces a fairly uninterpretable interval of \[0.03, 23698\] for the example dataset used in this article). Continuity adjustments are not implemented for any method for RR or OR. + +The SCAS method (which addresses asymmetric one-sided coverage of the MN Score method which is particularly pronounced for RR, and also adds a further bias correction for OR[@laud2018]) is not available in PROC FREQ, but is given by the SAS macro `%SCORECI`. An optional continuity adjustment[@laud2017] for both MN and SCAS is provided in the R package, but not yet implemented in the macro. + +## Continuity Adjusted Methods + +SAS provides an option (`CORRECT`) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage, but this only applies for the RISKDIFF contrast, not RELRISK or ODDSRATIO. The Hauck-Anderson (HA) method for RD is a slightly less conservative variation of a continuity adjustment. Note however that all of these methods fail to achieve strictly conservative coverage, although the adjusted Newcombe method comes close[@laud2014]. + +It is important to note that the `CORRECT` sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, the `CORRECT=NO` option removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and equivalent un-corrected score methods for RR and OR). + +No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described[@laud2014] and implemented for MN and SCAS in the ratesci package for R, but not yet added to the %SCORECI macro. + +## Consistency with Hypothesis Tests + +Within SAS PROC FREQ for the asymptotic methods for RD, consistency with a traditional Chi-squared test for association (**Karl** Pearson version, as produced by PROC FREQ), and the Farrington-Manning test for non-inferiority, is provided by the Mee CI (`CL=SCORE(CORRECT=MEE)`). That method is similar to MN but without the 'N-1' correction factor (the omission of which produces coverage that is slightly below the nominal confidence level on average). Note that the MN method (including the correction factor) is consistent with the **Egon** Pearson 'N-1' version of the chi-squared test[@campbell2007]. SAS PROC FREQ does not produce that test, nor does it offer the option to include the 'N-1' adjustment when requesting a non-inferiority test with the `NONINF` option. Consequently, there is a risk of contradictory results if using PROC FREQ to obtain a MN CI with a corresponding non-inferiority test. + +For the SCAS or MN method, the %SCORECI macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. + +If an EXACT statement is used to produce CIs, SAS does not offer any matching hypothesis tests for NI or equivalence testing. + +## Example Code using PROC FREQ + +It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN=` Option in riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". + +Similarly for relrisk, although the output does not state that the RR is calculated as (Row 1) / (Row 2). If treatment groups are labelled differently, you might need to sort by descending trt to obtain the correct contrast (note that unlike for RD, the same result cannot be obtained by setting `COLUMN=2`). + +SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. + +```{sas} + +****************************; +*** Risk Difference examples; +****************************; + +*** Wald, Newcombe, Agresti/Caffo, and Miettinen-Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(wald newcombe ac mn); +run; + +*** Mee score method; +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(mn(mee)); +run; + +*** exact (Chan-Zhang) and continuity-adjusted methods for conservative coverage; +proc freq data=adcibc2 order=data; + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=(exact ha wald(correct) newcombe(correct))); +run; + +*** exact (Santner-Snell) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=exact); +run; + +*** 2-sided exact (Agresti-Min) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=score2); + table trt*resp/riskdiff(CL=exact); +run; + +*** MN and SCAS methods from %SCORECI macro; +*** First manipulate the data to a form for input to the macro; +proc tabulate data=adcibc2 out=tab2; + class trt resp; + table (resp all),trt; +run; +data ds(keep = n1 n0 e1 e0); + set tab2; + by _page_; + retain n1 n0 e1 e0; + * Initialise counts in case there are none in the observed data; + if first._page_ then do; + e1 = 0; + e0 = 0; + end; + if upcase(trt) = "ACT" then do; + if _type_ = "11" and resp = "Yes" then e1 = n; + if _type_ = "10" then n1 = n; + end; + else if upcase(trt) = "PBO" then do; + if _type_ = "11" and resp = "Yes" then e0 = n; + if _type_ = "10" then n0 = n; + end; + if last._page_ then output; +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE); + +*** SCAS CI; +%scoreci(ds, stratify=FALSE); + +*** Farrington-Manning NI test (p=0.0248) contradicts MN interval (LCL < -0.036); +*** (Arbitrary NI margin of -0.036 used for illustration) +proc freq data=adcibc2 order=data; + table trt*resp / riskdiff(CL=mn noninf margin=0.036 method=score); +run; + +*** Miettinen-Nurminen CI with consistent NI test: PVAL_R > 0.025; +%scoreci(ds, stratify=FALSE, skew=FALSE, delta=-0.036); + + +****************************; +*** Relative risk examples; +****************************; + +*** Wald, LR, and Miettinen and Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /relrisk(CL=(wald waldmodified lr score)); +run; +*** MN without the correction; +proc freq data=adcibc2 order=data; + table trt*resp /relrisk(CL=(score(correct=no))); +run; +*** exact method; +proc freq data=adcibc2 order=data; + exact relrisk; + table trt*resp/relrisk(CL=(exact)); +run; +*** santner-snell; +proc freq data=adcibc2 order=data; + exact relrisk (method=noscore); + table trt*resp / relrisk(CL=exact); +run; + +*** Agresti-Min; +proc freq data=adcibc2 order=data; + exact relrisk (method=score2); + table trt*resp / relrisk(CL=exact); +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE, contrast=RR); +*** SCAS CI; +%scoreci(ds, stratify=FALSE, contrast=RR); + + +****************************; +*** Odds Ratio examples; +****************************; + + +* Wald, LR, mid-P and Miettinen and Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /or(CL=(wald waldmodified lr score midp)); +run; +* MN without the correction; +proc freq data=adcibc2 order=data; + table trt*resp /or(CL=(score(correct=no))); +run; +*** 'exact' method; +proc freq data=adcibc2 order=data; + exact or; + table trt*resp/or(CL=(exact)); +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE, orbias=FALSE, contrast=OR); +*** SCAS CI; +%scoreci(ds, stratify=FALSE, contrast=OR); +``` + +\[Selected output screenshots to be added to the below\] + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") +``` + +## References diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd new file mode 100644 index 000000000..4adcad917 --- /dev/null +++ b/SAS/ci_for_paired_prop.qmd @@ -0,0 +1,233 @@ +--- +title: "Confidence intervals for Paired Proportions in SAS" +execute: + eval: false +--- + +## Introduction + +\[See separate page for general introductory information on confidence intervals for proportions.\] + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{sas} +data adcibc2 (keep=trt resp) ; + set adcibc; + if aval gt 4 then resp="Yes"; + else resp="No"; + if trtp="Placebo" then trt="PBO"; + else trt="Act"; +run; +``` + +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). + +```{sas} +proc freq data=adcibc2; + table trt*resp/ nopct nocol; +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") +``` + +## Methods for Calculating Confidence Intervals for Proportion Difference from matched pairs + +You may experience paired data in any of the following types of situation: + +- Tumour assesssments classified as Progressive Disease or Not Progressive Disease performed by an Investigator and separately by an independent panel. + +- A paired case-control study (each subject taking active treatment is matched to a patient taking control) + +- A cross-over trial where the same subjects take both medications + +In all these cases, the calculated proportions for the 2 groups are not independent. + +Using a cross over study as our example, a 2 x 2 table can be formed as follows: + ++-----------------------+---------------+---------------+---------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+===============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+---------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+---------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+---------------+ + +The proportions of subjects responding on each treatment are: + +Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ + +Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ + +Suppose : + ++-----------------------+---------------+---------------+------------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+==================+ +| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | ++-----------------------+---------------+---------------+------------------+ +| Active Response = No | t = 6 | u = 5 | t+u = 11 | ++-----------------------+---------------+---------------+------------------+ +| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | ++-----------------------+---------------+---------------+------------------+ + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) + +In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. Hence the SE for the difference and 95% confidence interval can be calculated using the following equations. + +$SE(D)=\frac{1}{n} * sqrt(s+t-\frac{(s-t)^2}{n})$ + +$D-z_\alpha * SE(D)$ to $D+z_\alpha * SE(D)$ + +where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, + +D=(15-6) /46 = 0.196 + +SE(D) = 1/ 46 \* sqrt (15+6- (((15+6)\^2)/46) ) = 0.0956 + +Lower CI= 0.196 - 1.96 \*0.0956 = 0.009108 + +Upper CI = 0.196 + 1.96 \* 0.0956 = 0.382892 + +### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) + +Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. + +Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. + +Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. + +We then define $\phi$ which is used to correct for $\hat p_1$ and $\hat p_2$ not being independent. As the samples are related, $\phi$ is usually positive and thus makes the confidence interval smaller (narrower). + +If any of r+s, t+u, r+t, s+u are zero, then set $\phi$ to be 0. + +Otherwise we calculate A, B and C, and $\phi=C / sqrt A$ + +In the above: $A=(r+s)(t+u)(r+t)(s+u)$ and $B=(ru-st)$ + +To calculate C follow the table below. n=sample size. + +| Condition of B | Set C equal to | +|---------------------------|----------------| +| If B is greater than n/2 | B - n/2 | +| If B is between 0 and n/2 | 0 | +| If B is less than 0 | B | + +Let D = p1-p2 (the difference between the observed proportions of responders) + +The Confidence interval for the difference between two population proportions is: $D - sqrt((p_1-l_1)^2-2\phi(p_1-l_1)(u_2-p_2)+(u_2-p_2)^2 )$ to + +$D + sqrt((p_2-l_2)^2-2\phi(p_2-l_2)(u_1-p_1)+(u_1-p_1)^2 )$ + +First using the Wilson Method equations for each of the individual single samples 1 and 2. + +| | Active | Placebo | +|----------|------------|------------| +| a | 73.842 | 55.842 | +| b | 13.728 | 11.974 | +| c | 99.683 | 99.683 | +| Lower CI | 0.603 = L1 | 0.440 = L2 | +| Upper CI | 0.878 = U1 | 0.680 = U2 | + +$A=(r+s)(t+u)(r+t)(s+u)$ = 9450000 + +B=10 + +C= 0 (as B is between 0 and n/2) + +$\phi$ = 0. + +Hence the middle part of the equation simplies to 0, and becomes simply: + +Lower CI = $D - sqrt((p_1-l_1)^2+(u_2-p_2)^2 )$ = 0.196 - sqrt \[ (0.761-0.603)\^2 + (0.680-0.565) \^2 \] + +Upper CI = $D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 )$ = 0.196 + sqrt \[ (0.565-0.440)\^2 + (0.878-0.761) \^2 \] + +CI= 0.00032 to 0.367389 + +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with Hypothesis Tests + +\[TBC\] + +## Example Code using PROC FREQ + +Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. A macro is available at \[\] which provides the asymptotic score method by Tango, and a skewness corrected version (paper under review). + +Instead it calculates the risk difference = (r / (r+s) - t / (t+u) ) + +Which in the example above is: 20/ (20+15) - 6 / (6+5) = 0.0296 + +This is more applicable when you have exposed / non-exposed groups looking at who has experienced the outcome. SAS Proc Freq has 3 methods for analysis of paired data using a Common risk difference. + +The default method is Mantel-Haenszel confidence limits. SAS can also Score (Miettinen-Nurminen) CIs and Stratified Newcombe CIs (constructed from stratified Wilson Score CIs). See [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for equations. + +As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. + +```{sas} +data dat_used; + input ID$ PBO ACT @@; + datalines; + 001 0 0 002 0 0 003 0 0 004 0 0 005 0 0 + 006 1 0 007 1 0 008 1 0 009 1 0 010 1 0 + 011 1 0 012 0 1 013 0 1 014 0 1 015 0 1 + 016 0 1 017 0 1 018 0 1 019 0 1 020 0 1 + 021 0 1 022 0 1 023 0 1 024 0 1 025 0 1 + 026 0 1 027 1 1 028 1 1 029 1 1 030 1 1 + 031 1 1 032 1 1 033 1 1 034 1 1 035 1 1 + 036 1 1 037 1 1 038 1 1 039 1 1 040 1 1 + 041 1 1 042 1 1 043 1 1 044 1 1 045 1 1 + 046 1 1 + ; +run; + +proc freq data=dat_used; + table ACT*PBO/commonriskdiff(cl=MH); +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/riskdiff_matched.png") +``` + +## Reference + +1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) + +2. [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f) + +3. [Confidence intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf) + +4. Brown LD, Cai TT, DasGupta A (2001). "Interval estimation for a binomial proportion", Statistical Science 16(2):101-133 + +5. Newcombe RG (1998). "Two-sided confidence intervals for the single proportion: comparison of seven methods", Statistics in Medicine 17(8):857-872 + +6. Clopper,C.J.,and Pearson,E.S.(1934),"The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial", Biometrika 26, 404--413. + +7. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. + +8. Laud PJ (2017) Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 16: 334-348 + +9. Laud PJ (2018) Corrigendum: Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 17: 290-293 + +10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 + +11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. + +12. diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 4a3e77516..60d0166e3 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -1,24 +1,14 @@ --- -title: "Confidence intervals for Proportions in SAS" +title: "Confidence intervals for a Proportion in SAS" execute: - eval: false + eval: false --- ## Introduction -The methods to use for calculating a confidence interval (CI) for a proportion depend on the type of situation you have. +See separate page for general introductory information on confidence intervals for proportions. -- 1 sample proportion (1 proportion calculated from 1 group of subjects) - -- 2 sample proportions and you want a CI for the difference in the 2 proportions (or the ratio, or an odds ratio) e.g. for estimating the magnitude of a treatment effect. - - - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) - - - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). - -Some sources suggest selecting a different method depending on whether your proportion is close to 0 or 1 (or near to the 0.5 midpoint), and your sample size. Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied. However, options are available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection. - -## Data used +## Data Used The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. @@ -32,7 +22,7 @@ data adcibc2 (keep=trt resp) ; run; ``` -The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). ```{sas} proc freq data=adcibc2; @@ -48,37 +38,29 @@ run; knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ``` -## Methods for Calculating Confidence Intervals for a single proportion +## Methods for Calculating Confidence Intervals for a Single Proportion -Here we are calculating a 95% confidence interval for the proportion of responders in the active treatment group. +Here we are calculating a $100(1-\alpha)\%$ (usually 95%) confidence interval for the proportion $p$ of responders in the active treatment group, estimated from the sample proportion $\hat p = x / n$. SAS PROC FREQ in Version 9.4 can compute 11 methods to calculate CIs for a single proportion, an explanation of most of these methods and the code is shown below. See [BINOMIAL](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm)^1^ for more information on SAS parameterization. It is recommended to always sort your data prior to doing a PROC FREQ. For more information about some of these methods in R & SAS, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^2^ and [Confidence Intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf)^3^. Key literature on the subject includes papers by [Brown et al.](https://www.jstor.org/stable/2676784?seq=1)^4^ and [Newcombe](https://pubmed.ncbi.nlm.nih.gov/9595616/)^5^. -### Strictly conservative vs proximate coverage - -Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction") to achieve the former. +### Clopper-Pearson (Exact or binomial CI) Method -### Consistency with hypothesis tests - -Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS for the asymptotic methods, such consistency is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. If an `EXACT BINOMIAL;` statement is used, the resulting test is consistent with the Clopper-Pearson CI. An exact hypothesis test with mid-P adjustment (consistent with the mid-P CI) is available via `EXACT BINOMIAL / MIDP;` but the output only gives the one-sided p-value, so if a 2-sided test is required it needs to be manually calculated by doubling the one-sided mid p-value. - -### Clopper-Pearson (Exact or binomial CI) method - -With binary endpoint data (response/non-response), we make the assumption that the proportion of responders has been derived from a series of Bernoulli trials. Trials (Subjects) are independent and we have a fixed number of repeated trials with an outcome of respond or not respond. This type of data follows the discrete binomial probability distribution, and the Clopper-Pearson^6^ (Exact) method uses this distribution to calculate the CIs. +With binary endpoint data (response/non-response), we make the assumption that the proportion of responders has been derived from a series of Bernoulli trials. Trials (Subjects) are independent and we have a fixed number of repeated trials with an outcome of respond or not respond. This type of data follows the discrete binomial probability distribution, and the Clopper-Pearson^6^ ('exact') method uses this distribution to calculate the CIs. This method guarantees strictly conservative coverage, but has been noted to be excessively conservative, as for any given proportion, the actual coverage probability can be much larger than $(1-\alpha)$. The Clopper-Pearson method is output by SAS as one of the default methods (labelled as "Exact Conf Limits" in the "Proportion" ODS output object), but you can also specify it using `BINOMIAL(LEVEL="Yes" CL=CLOPPERPEARSON);` which creates a separate output object "ProportionCLs". -### Normal Approximation method (Also known as the Wald or asymptotic CI method) +### Normal Approximation Method (Also known as the Wald Method) -The traditional alternative to the Clopper-Pearson (Exact) method is the asymptotic Normal Approximation (Wald) CI. The poor performance of this method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Nevertheless, it remains a default output component in SAS, so is included here for reference. +The traditional alternative to the Clopper-Pearson ('Exact') method is the asymptotic Normal Approximation (Wald) CI. The poor performance of this method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Nevertheless, it remains a default output component in SAS, so is included here for reference. In large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\alpha)\%$ confidence interval for population proportion can be calculated using the normal approximation as follows: -$p\approx \hat p \pm z_{\alpha/2} \sqrt{\hat p(1-\hat p)/n}$, where $\hat p$ is the sample proportion, $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$, and $\sqrt{\hat p(1-\hat p)/n}$ is the estimated standard error. +$p\approx \hat p \pm z_{\alpha/2} \sqrt{\hat p(1-\hat p)/n}$, where $\hat p=x/n$ is the sample proportion, $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$, and $\sqrt{\hat p(1-\hat p)/n}$ is the estimated standard error. One should note that the approximation can become increasingly unreliable as the proportion of responders gets closer to 0 or 1 (e.g. 0 or 100% responding). In this scenario, common issues consist of: @@ -102,12 +84,13 @@ The Wilson (Score) method is also based on an asymptotic normal approximation, b The method can be derived with or without a Yates continuity correction. The corrected interval closely approximates the coverage of the Clopper-Pearson method, but only in terms of overall two-sided coverage - due to the asymmetric coverage, it does not guarantee that the non-coverage probability is less than $\alpha/2$. -Let p=r/n, where r= number of responses, and n=number of subjects, q=1-p, and z= the appropriate value from standard normal distribution:\ -$$ z{_{1-\alpha/2}} $$For example, for 95% confidence intervals, $\alpha=0.05$, using standard normal tables, z in the equations below will take the value =1.96. Calculate 3 quantities +Let $\hat p$ =r/n, where r= number of responses, and n=number of subjects, $\hat q = 1- \hat p$, and z= the appropriate value from standard normal distribution: $z_{1-\alpha/2}$. + +For example, for 95% confidence intervals, $\alpha=0.05$, using standard normal tables, z in the equations below will take the value =1.96. Calculate 3 quantities $$ A= 2r+z^2$$ -$$ B=z\sqrt(z^2 + 4rq) $$ $$ C=2(n+z^2) $$The method calculates the confidence interval \[Lower, Upper\] as: \[(A-B)/C, (A+B)/C\] +$$ B=z\sqrt(z^2 + 4r \hat q) $$ $$ C=2(n+z^2) $$The method calculates the confidence interval \[Lower, Upper\] as: \[(A-B)/C, (A+B)/C\] A = 2 \* 36 + 1.96\^2 = 75.8416 @@ -129,25 +112,29 @@ The only differences in the equations to calculate the Wilson score with continu $$ A= 2r+z^2 -1$$ -$$ B=z\sqrt(z^2 - 2 -\frac{1}{n} + 4rq) $$ +$$ B=z\sqrt(z^2 - 2 -\frac{1}{n} + 4r \hat q) $$ ### Agresti-Coull method -The Agresti-Coull method is a 'simple solution' designed to improve coverage compared to the Wald method and still perform better (i.e. less conservative) than Clopper-Pearson particularly when the probability isn't in the mid-range (0.5). It is less conservative whilst still having good coverage. The only difference compared to the Wald method is that it adds two successes and two failures to the original observations (increasing the sample by 4 observations). In practice it is not often used *\[Ed: what is the basis for this statement?\]*. +The Agresti-Coull method is a 'simple solution' designed to improve coverage compared to the Wald method and still perform better (i.e. less conservative) than Clopper-Pearson particularly when the probability isn't in the mid-range (0.5). It is less conservative whilst still having good coverage. The only difference compared to the Wald method is that it adds $z^2/2$ successes and failures to the original observations (when $\alpha=0.05$ this increases the sample by approximately 4 observations). The Agresti-Coull method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=AGRESTICOULL);` ### Jeffreys method -The Jeffreys method is a particular type of Bayesian Highest Probability Density (HPD) method. For binomial proportions, the beta distribution is generally used for the conjugate prior, which consists of two parameters $\alpha$ and $\beta$. Setting $\alpha=\beta=0.5$ is called the Jeffreys prior. This is considered as non-informative for a binomial proportion. +The Jeffreys method is a particular type of Bayesian Highest Probability Density (HPD) method. For binomial proportions, the beta distribution is generally used for the conjugate prior, which consists of two parameters $\alpha'$ and $\beta'$. Setting $\alpha'=\beta'=0.5$ is called the Jeffreys prior. This is considered as non-informative for a binomial proportion. The resulting posterior density gives the CI as $$ -(Beta (^k/_2 + ^1/_{2}, ^{(n-k)}/_2+^1/_2)_{\alpha}, Beta (^k/_2 + ^1/_{2}, ^{(n-k)}/_2+^1/_2)_{1-\alpha}) -$$The coverage probabilities of the Jeffreys method are centred around the nominal confidence level on average, with symmetric "equal-tailed" 1-sided coverage^8 Appx S3.5^. This interval is output by SAS using `BINOMIAL(LEVEL="Yes" CL=Jeffreys);` +(Beta (x + 0.5, n - x + 0.5)_{\alpha/2}, Beta (x + 0.5, n - x + 0.5)_{1-\alpha/2}) +$$ + +Boundary modifications are applied to force the lower limit to 0 if x=0 and the upper limit to 1 if x=n. + +The coverage probabilities of the Jeffreys method are centred around the nominal confidence level on average, with symmetric "equal-tailed" 1-sided coverage^8 Appx S3.5^. This interval is output by SAS using `BINOMIAL(LEVEL="Yes" CL=Jeffreys);` ### Binomial based Mid-P method -The Mid-P method is similar to the Clopper-Pearson method, in the sense that it is based on exact calculations from the binomial probability distribution, but it aims to reduce the conservatism. It's quite a complex method to compute compared to the methods above and rarely used in practice *\[Ed: what source is there for this statement?\]*. However, like the Jeffreys interval, it has excellent 1-sided and 2-sided coverage properties for those seeking to align mean coverage with the nominal confidence level^9^. +The mid-P method is similar to the Clopper-Pearson method, in the sense that it is based on exact calculations from the binomial probability distribution, but it aims to reduce the conservatism. It's quite a complex method to compute compared to the methods above and rarely used in practice *\[Ed: what source is there for this statement?\]*. However, like the Jeffreys interval, it has excellent 1-sided and 2-sided coverage properties for those seeking to align mean coverage with the nominal confidence level^9^. The mid-P method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=MIDP);` @@ -159,15 +146,27 @@ The Clopper-Pearson CI's are always wider and contain the Blaker CI limits. It's The Blaker method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=BLAKER);` +## Continuity Adjusted Methods + +SAS offers the 'CORRECT' option for producing more conservative versions of the Wald and Wilson Score intervals. + +## Consistency with hypothesis tests + +Within SAS PROC FREQ for the asymptotic methods, consistency with the binomial test is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. + +If an `EXACT BINOMIAL;` statement is used, the resulting test is consistent with the Clopper-Pearson CI. + +An exact hypothesis test with mid-P adjustment (consistent with the mid-P CI) is available via `EXACT BINOMIAL / MIDP;` but the output only gives the one-sided p-value, so if a 2-sided test is required it needs to be manually calculated by doubling the one-sided mid p-value. + ## Example Code using PROC FREQ -By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ', SAS outputs the Normal Approximation (Wald) and Clopper-Pearson (Exact) confidence intervals as two default methods, derived for the `Responders` = `Yes`. If you do not specify the `LEVEL` you want to model, then SAS assumes you want to model the first level that appears in the output (alphabetically). +By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ' TABLES statement, SAS outputs the Normal Approximation (Wald) and Clopper-Pearson (Exact) confidence intervals as two default methods, derived for the `Responders` = `Yes`. If you do not specify the `LEVEL` you want to model, then SAS assumes you want to model the first level that appears in the output (alphabetically). **It is very important to ensure you are calculating the CI for the correct level! Check your output to confirm, you will see below it states `resp=Yes` !** -Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. +Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS PROC FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can (and should) be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. -The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method (Exact), and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. +The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method, and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. ```{sas} proc sort data=adcibc2; @@ -192,11 +191,11 @@ By adding the option `BINOMIAL(LEVEL="Yes" CL=)`, the other C - `BINOMIAL(LEVEL="Yes" CL=CLOPPERPEARSON WALD WILSON AGRESTICOULL JEFFREYS MIDP LIKELIHOODRATIO LOGIT BLAKER)` will return Agresti-Coull, Blaker, Clopper-Pearson(Exact), Wald(without continuity correction) Wilson(without continuity correction), Jeffreys, Mid-P, Likelihood Ratio, and Logit -- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural for Mid-P to be included here instead of Clopper-Pearson!\]* +- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural/logical for Mid-P to be included here instead of Clopper-Pearson! However, note that the ALL option is not mentioned in the current PROC FREQ documentation.\]* -- `BINOMIALc(LEVEL="Yes" CL=ALL);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction) +- `BINOMIAL(LEVEL="Yes" CL=ALL CORRECT);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction). In previous versions of SAS this was coded as `BINOMIALc(...)`, but that is no longer documented. *\[... following the above comment, it is also not logical for Jeffreys and Agresti-Coull to be included here alongside methods designed to achieve conservative coverage\]* -- `BINOMIALc(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson(with continuity correction) and Wald (with continuity correction) +- `BINOMIAL(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson (with continuity correction) and Wald (with continuity correction) ```{sas} proc freq data=adcibc2; @@ -238,345 +237,7 @@ knitr::include_graphics("../images/ci_for_prop/binomial_prop_cc_act.png") SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. `ods output binomialcls=bcl;` and then using the bcl dataset, in a data step to put the variable out to the number of decimal places we require.\ 10 decimal places shown here ! `lowercl2=put(lowercl,12.10);` -## Methods for Calculating Confidence Intervals for 2 independent samples proportion difference - -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The most commonly used are: Wald with continuity correction and Wilson with continuity correction. The Wilson method may be more applicable when sample sizes are smaller and/or the proportion is closer to 0 or 1, since reliance on a normal distribution may be inappropriate. - -SAS is able to calculate CIs using the following methods: Exact, Agresti/Caffo, Wald, Wald with Continuity correction, Wilson/Newcombe Score, Wilson/Newcombe score with continuity correction, Miettinen and Nurminen, and Hauck-Anderson. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. - -```{sas} -# Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods -proc freq data=adcibc2 order=data; - table trt*resp /riskdiff(CL=(wald wilson ac ha mn)); -run; # exact method - -proc freq data=adcibc2 order=data; - exact riskdiff (method=noscore); - table trt*resp/riskdiff(CL=(exact)); -run; -``` - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) - -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. - -The difference between two independent sample proportions is calculated as: $D= \hat p_1-\hat p_2$ - -A confidence interval for the difference between two independent proportions $D$ can be calculated using: - -$D\approx \hat D \pm z_\alpha * SE(\hat p)$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, and - -$SE (\hat p) = sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ - -With continuity correction, the equation becomes - -$D\approx \hat D \pm (CC + z_\alpha * SE(\hat p))$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$,\ -and $SE (\hat p)$ = $sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$\ -and - -$CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ - -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) - -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. - -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. - -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. - -Let D = p1-p2 (the difference between the observed proportions) - -The Confidence interval for the difference between two population proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ - -## Example Code using PROC FREQ - -It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. - -Options for riskdiff(CL=XXX) consist of AC:Agresti-Caffo, EXACT=exact, HA:Hauck-Anderson, MN or SCORE:Miettinen-Nurminen (another type of Score CI), WILSON or NEWCOMBE: Wilson method described above, and WALD: normal approximation wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. - -```{sas} -proc sort data=adcibc2; - by trt descending resp; -run; - -# without continuity correction -proc freq data=adcibc2 order=data; - table trt*resp/riskdiff(CL=(wald wilson)); -run; - -# with continuity correction -proc freq data=adcibc2 order=data; - table trt*resp/riskdiff(CORRECT CL=(wald wilson)); -run; -``` - -```{r} -#| eval: true -#| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") -``` - -## Methods for Calculating Confidence Intervals for a matched pair proportion difference - -You may experience paired data in any of the following types of situation: - -- Tumour assesssments classified as Progressive Disease or Not Progressive Disease performed by an Investigator and separately by an independent panel. - -- A paired case-control study (each subject taking active treatment is matched to a patient taking control) - -- A cross-over trial where the same subjects take both medications - -In all these cases, the calculated proportions for the 2 groups are not independent. - -Using a cross over study as our example, a 2 x 2 table can be formed as follows: - -+-----------------------+---------------+---------------+---------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+===============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+---------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+---------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+---------------+ - -The proportions of subjects responding on each treatment are: - -Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ - -Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ - -Suppose : - -+-----------------------+---------------+---------------+------------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+==================+ -| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | -+-----------------------+---------------+---------------+------------------+ -| Active Response = No | t = 6 | u = 5 | t+u = 11 | -+-----------------------+---------------+---------------+------------------+ -| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | -+-----------------------+---------------+---------------+------------------+ - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) - -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. Hence the SE for the difference and 95% confidence interval can be calculated using the following equations. - -$SE(D)=\frac{1}{n} * sqrt(s+t-\frac{(s-t)^2}{n})$ - -$D-z_\alpha * SE(D)$ to $D+z_\alpha * SE(D)$ - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, - -D=(15-6) /46 = 0.196 - -SE(D) = 1/ 46 \* sqrt (15+6- (((15+6)\^2)/46) ) = 0.0956 - -Lower CI= 0.196 - 1.96 \*0.0956 = 0.009108 - -Upper CI = 0.196 + 1.96 \* 0.0956 = 0.382892 - -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) - -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. - -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. - -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. - -We then define $\phi$ which is used to correct for $\hat p_1$ and $\hat p_2$ not being independent. As the samples are related, $\phi$ is usually positive and thus makes the confidence interval smaller (narrower). - -If any of r+s, t+u, r+t, s+u are zero, then set $\phi$ to be 0. - -Otherwise we calculate A, B and C, and $\phi=C / sqrt A$ - -In the above: $A=(r+s)(t+u)(r+t)(s+u)$ and $B=(ru-st)$ - -To calculate C follow the table below. n=sample size. - -| Condition of B | Set C equal to | -|---------------------------|----------------| -| If B is greater than n/2 | B - n/2 | -| If B is between 0 and n/2 | 0 | -| If B is less than 0 | B | - -Let D = p1-p2 (the difference between the observed proportions of responders) - -The Confidence interval for the difference between two population proportions is: $D - sqrt((p_1-l_1)^2-2\phi(p_1-l_1)(u_2-p_2)+(u_2-p_2)^2 )$ to - -$D + sqrt((p_2-l_2)^2-2\phi(p_2-l_2)(u_1-p_1)+(u_1-p_1)^2 )$ - -First using the Wilson Method equations for each of the individual single samples 1 and 2. - -| | Active | Placebo | -|----------|------------|------------| -| a | 73.842 | 55.842 | -| b | 13.728 | 11.974 | -| c | 99.683 | 99.683 | -| Lower CI | 0.603 = L1 | 0.440 = L2 | -| Upper CI | 0.878 = U1 | 0.680 = U2 | - -$A=(r+s)(t+u)(r+t)(s+u)$ = 9450000 - -B=10 - -C= 0 (as B is between 0 and n/2) - -$\phi$ = 0. - -Hence the middle part of the equation simplies to 0, and becomes simply: - -Lower CI = $D - sqrt((p_1-l_1)^2+(u_2-p_2)^2 )$ = 0.196 - sqrt \[ (0.761-0.603)\^2 + (0.680-0.565) \^2 \] - -Upper CI = $D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 )$ = 0.196 + sqrt \[ (0.565-0.440)\^2 + (0.878-0.761) \^2 \] - -CI= 0.00032 to 0.367389 - -## Example Code using PROC FREQ - -Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. - -Instead it calculates the risk difference = (r / (r+s) - t / (t+u) ) - -Which in the example above is: 20/ (20+15) - 6 / (6+5) = 0.0296 - -This is more applicable when you have exposed / non-exposed groups looking at who has experienced the outcome. SAS Proc Freq has 3 methods for analysis of paired data using a Common risk difference. - -The default method is Mantel-Haenszel confidence limits. SAS can also Score (Miettinen-Nurminen) CIs and Stratified Newcombe CIs (constructed from stratified Wilson Score CIs). See [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for equations. - -As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. - -```{sas} -data dat_used; - input ID$ PBO ACT @@; - datalines; - 001 0 0 002 0 0 003 0 0 004 0 0 005 0 0 - 006 1 0 007 1 0 008 1 0 009 1 0 010 1 0 - 011 1 0 012 0 1 013 0 1 014 0 1 015 0 1 - 016 0 1 017 0 1 018 0 1 019 0 1 020 0 1 - 021 0 1 022 0 1 023 0 1 024 0 1 025 0 1 - 026 0 1 027 1 1 028 1 1 029 1 1 030 1 1 - 031 1 1 032 1 1 033 1 1 034 1 1 035 1 1 - 036 1 1 037 1 1 038 1 1 039 1 1 040 1 1 - 041 1 1 042 1 1 043 1 1 044 1 1 045 1 1 - 046 1 1 - ; -run; - -proc freq data=dat_used; - table ACT*PBO/commonriskdiff(cl=MH); -run; -``` - -```{r} - -#| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/riskdiff_matched.png") -``` - -## Methods for Calculating Confidence Intervals for 2 independent samples proportion - -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^8^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The most commonly used are: Wald with continuity correction and Wilson with continuity correction. The Wilson method may be more applicable when sample sizes are smaller and/or the proportion is closer to 0 or 1, since reliance on a normal distribution may be inappropriate. - -SAS is able to calculate CIs using the following methods: Exact, Agresti/Caffo, Wald, Wald with Continuity correction, Wilson/Newcombe Score, Wilson/Newcombe score with continuity correction, Miettinen and Nurminen, and Hauck-Anderson. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. - -```{sas} -# Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods -proc freq data=adcibc2 order=data; - table trt*resp /riskdiff(CL=(wald wilson ac ha mn)); -run; - -# exact method -proc freq data=adcibc2 order=data; - exact riskdiff (method=noscore); - table trt*resp/riskdiff(CL=(exact)); -run; -``` - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) - -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. - -The difference between two independent sample proportions is calculated as: $D= \hat p_1-\hat p_2$ - -A confidence interval for the difference between two independent proportions $D$ can be calculated using: - -$D\approx \hat D \pm z_\alpha * SE(\hat p)$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, and - -$SE (\hat p) = sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ - -With continuity correction, the equation becomes - -$D\approx \hat D \pm (CC + z_\alpha * SE(\hat p))$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$,\ -and $SE (\hat p)$ = $sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$\ -and - -$CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ - -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) - -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. - -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. - -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. - -Let D = p1-p2 (the difference between the observed proportions) - -The Confidence interval for the difference between two population proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ - -## Example Code using PROC FREQ - -It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. - -Options for riskdiff(CL=XXX) consist of AC:Agresti-Caffo, EXACT=exact, HA:Hauck-Anderson, MN or SCORE:Miettinen-Nurminen (another type of Score CI), WILSON or NEWCOMBE: Wilson method described above, and WALD: normal approximation wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. - -```{sas} -proc sort data=adcibc2; - by trt descending resp; -run; - -# without continuity correction -proc freq data=adcibc2 order=data; - table trt*resp/riskdiff(CL=(wald wilson)); -run; - -# with continuity correction -proc freq data=adcibc2 order=data; - table trt*resp/riskdiff(CORRECT CL=(wald wilson)); -run; -``` - -```{r} -#| echo: false -#| eval: true -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") -``` - -```{r} -#| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") -``` - -## Reference +## References 1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) @@ -592,4 +253,10 @@ knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") 7. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. -8. \ No newline at end of file +8. Laud PJ (2017) Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 16: 334-348 + +9. Laud PJ (2018) Corrigendum: Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 17: 290-293 + +10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 + +11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index d19db8e99..eb742248f 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -6,11 +6,11 @@ execute: # Cochran-Mantel-Haenszel Test -The CMH procedure tests for conditional independence in partial contingency tables for a 2 x 2 x K design. However, it can be generalized to tables of X x Y x K dimensions. This page also details derivation of risk differences and their confidence intervals which often accompnay a CMH test. +The CMH procedure tests for conditional independence in partial contingency tables for a 2 x 2 x K design. However, it can be generalized to tables of X x Y x K dimensions. This page also details derivation of risk differences and their confidence intervals which often accompany a CMH test. ## CMH in SAS -The cmh test is calculated in SAS using the PROC FREQ procedure. By default, it outputs the chi square statistic, degrees of freedom and p-value for each of the three alternative hypothesis: `general association`, `row means differ`, and `nonzero correlation`. It is up to the statistical analyst or statistician to know which result is appropriate for their analysis. +The CMH test is calculated in SAS using the PROC FREQ procedure. By default, it outputs the chi square statistic, degrees of freedom and p-value for each of the three alternative hypothesis: `general association`, `row means differ`, and `nonzero correlation`. It is up to the statistical analyst or statistician to know which result is appropriate for their analysis. When the design of the contingency table is 2 x 2 x K (i.e, X == 2 levels, Y == 2 levels, K \>= 2 levels), the Mantel-Haenszel Common Odds Ratio (odds ratio estimate, 95% CI, P-value) and the Breslow-Day Test for Homogeneity of the Odds Ratios (chi-square statistic, degrees of freedom, P-value) are also output. @@ -66,7 +66,7 @@ The above examples are a test for general association or if the row means scores ### Risk Differences Within each Strata -Risk differences within each strata can be obtained by adding the riskdiff option in SAS. The exact same output is obtained as per example 1 above, with the addition of 2 tables (1 for each age strata), showing the proportion of Female patients within each treatment (including 95% CIs), and the difference between the treatment proportions (including 95% CIs). By default, the CI's are calculated using Wald asymptotic confidence limits and in addition, the `exact` Clopper-Pearson confidence intervals for the risks. See SAS userguide [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) and [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) for more detail on the range of CI's SAS can calculate for the risk differences such as: Agresti-Caffo, exact unconditional, Hauck-Anderson, Miettinen-Nurminen (score), Newcombe (hybrid-score), Wald confidence limits, continuity-corrected Newcombe and continuity corrected Wald CIs. +Risk differences within each strata (for 2 x 2 x K tables) can be obtained by adding the riskdiff option in SAS. The exact same output is obtained as per example 1 above, with the addition of 2 tables (1 for each age strata), showing the proportion of Female patients within each treatment (including 95% CIs), and the difference between the treatment proportions (including 95% CIs). By default, the CI's are calculated using Wald asymptotic confidence limits and in addition, the `exact` Clopper-Pearson confidence intervals for the risks. See SAS userguide [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) and [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) for more detail on the range of CI's SAS can calculate for the risk differences such as: Agresti-Caffo, exact unconditional, Hauck-Anderson, Miettinen-Nurminen (score), Newcombe (hybrid-score), Wald confidence limits, continuity-corrected Newcombe and continuity corrected Wald CIs. The individual treatment comparisons within strata can be useful to explore if the treatment effect is in the same direction and size for each strata, such as to determine if pooling them is in fact sensible. @@ -101,9 +101,11 @@ run; knitr::include_graphics("../images/cmh/saspage_output3b.png") ``` -### SAS Common error: SAS cannot do a stratified Miettinen-Nurminen (score) method! +### SAS Common error: SAS v9.4 cannot do a stratified Miettinen-Nurminen (score) method! -Note: you may think by running the following code, that you would be creating a common risk difference using the stratified Miettinen-Nurminen method. However, this is actually performing an unstratified Miettinen-Nurminen (Score) method. The output contains the same risk differences calculated for each strata separately and then a common risk difference (however this is NOT a stratified approach !!). See [SAS guide](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. +Note: you may think by running the following code, that you would be creating a common risk difference using the stratified Miettinen-Nurminen method. However, this is actually performing an unstratified Miettinen-Nurminen (Score) method. The output contains the same risk differences calculated for each strata separately and then a common risk difference using the 'summary score estimate' approach (however this is NOT a stratified approach !!). See [SAS guide](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. + +PROC FREQ in the SAS Viya platform has introduced a new COMMONRISKDIFF(CL=MN/MNMH) option for the TABLES statement that does use the stratified MN method. See the next section on Common risk differences available in SAS. @@ -124,7 +126,9 @@ knitr::include_graphics("../images/cmh/saspage_output3c.png") ### SAS Common error: Make sure you output the risk difference for the correct level! -Including either column=1 or column=2 tells SAS which is your outcome of interest (ie, often that you want to compare treatment responders and not treatment non-responders!) My default it takes the 1st column sorted alphabetically but you can change this as shown below. +Including either column=1 or column=2 tells SAS which is your outcome of interest (ie, often that you want to compare treatment responders and not treatment non-responders!) By default it takes the 1st column sorted alphabetically but you can change this as shown below. + +Note that if the data are such that there are no responses in either treatment group, the cross-tabulation will only have 1 column, and SAS PROC FREQ will fail to produce a confidence interval. In this unusual situation however, a valid confidence interval can (and should) still be produced, which may be obtained using the %SCORECI macro available at [https://github.com/petelaud/ratesci-sas/tree/main](#0). ```{sas} proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne "\>80")); @@ -137,9 +141,9 @@ run; Proc freq can calculate estimates of the common risk difference with 95% CIs, calculated using the Mantel-Haenszel and summary score (Miettinen-Nurminen) methods for multiway 2x2 tables. It can also provide stratified Newcombe confidence intervals using the method by Yan and Su (2010). The stratified Newcombe CIs are constructed from stratified Wilson CIs for the common (overall) row proportions. See [SAS help](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. -Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nurminen ('MN') score interval for unstratified datasets only. Using 'commonriskdiff' requests risks (binomial proportions) and risk differences for 2x2 tables. But doesn't extend to stratified analysis. +Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nurminen ('MN') score interval for unstratified datasets only. The 'commonriskdiff' option added in SAS/STAT 14.3 requests risks (binomial proportions) and risk differences for 2x2 tables. But doesn't extend to stratified analysis. -The only known way to have SAS produce stratified Miettinen-Nurminen CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](https://urldefense.com/v3/__https:/github.com/petelaud/ratesci-sas/tree/main__;!!GfteaDio!d_y6BtRjdLQgc_Wr-2-HGeyDSL1v71SvjvEQuVXNjzYaqLVDsEH0DdBLCBE3Q6LGFaTjE4LCjtECNYNFYfMSg4G49w$) +Stratified Miettinen-Nurminen CIs have more recently been added to PROC FREQ, but only within the SAS Viya platform (using CL=MN or CL=MNMH in the COMMONRISKDIFF option). The only other known way to have SAS produce these CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](#0). Note that the macro (currently) does not implement Miettinen & Nurminen's proposed iterative weights, but instead the simpler (and similar) MH weights are used. ```{sas} proc freq data = adcibc (where=(trtpn ne 54 and agegr1 ne ">80")); diff --git a/SAS/nature.csl b/SAS/nature.csl new file mode 100644 index 000000000..3b1e3d7a2 --- /dev/null +++ b/SAS/nature.csl @@ -0,0 +1,189 @@ + + diff --git a/SAS/references.bib b/SAS/references.bib new file mode 100644 index 000000000..07114da8b --- /dev/null +++ b/SAS/references.bib @@ -0,0 +1,248 @@ + +@article{brown2001, + title = {Interval Estimation for a Binomial Proportion}, + author = {Brown, Lawrence D. and Cai, T. Tony and DasGupta, Anirban}, + year = {2001}, + month = {05}, + date = {2001-05-01}, + journal = {Statistical Science}, + volume = {16}, + number = {2}, + doi = {10.1214/ss/1009213286}, + url = {http://dx.doi.org/10.1214/ss/1009213286} +} + +@book{newcombe2012, + title = {Confidence Intervals for Proportions and Related Measures of Effect Size}, + author = {Newcombe, Robert Gordon}, + year = {2012}, + month = {08}, + date = {2012-08-25}, + publisher = {CRC Press}, + doi = {10.1201/b12670}, + url = {http://dx.doi.org/10.1201/b12670}, + langid = {en} +} + +@article{newcombe1998c, + title = {Two-sided confidence intervals for the single proportion: comparison of seven methods}, + author = {Newcombe, Robert G.}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {857--872}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + langid = {en} +} + +@article{campbell2007, + title = {Chi{-}squared and Fisher{\textendash}Irwin tests of two{-}by{-}two tables with small sample recommendations}, + author = {Campbell, Ian}, + year = {2007}, + month = {02}, + date = {2007-02-21}, + journal = {Statistics in Medicine}, + pages = {3661--3675}, + volume = {26}, + number = {19}, + doi = {10.1002/sim.2832}, + url = {http://dx.doi.org/10.1002/sim.2832}, + langid = {en} +} + +@article{laud2017, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2017}, + month = {06}, + date = {2017-06-22}, + journal = {Pharmaceutical Statistics}, + pages = {334--348}, + volume = {16}, + number = {5}, + doi = {10.1002/pst.1813}, + url = {http://dx.doi.org/10.1002/pst.1813}, + langid = {en} +} + +@article{miettinen1985, + title = {Comparative analysis of two rates}, + author = {Miettinen, Olli and Nurminen, Markku}, + year = {1985}, + month = {04}, + date = {1985-04}, + journal = {Statistics in Medicine}, + pages = {213--226}, + volume = {4}, + number = {2}, + doi = {10.1002/sim.4780040211}, + url = {http://dx.doi.org/10.1002/sim.4780040211}, + langid = {en} +} + +@article{Mee1984, + ISSN = {0006341X, 15410420}, + URL = {http://www.jstor.org/stable/2531174}, + author = {Robert W. Mee and Dan Anbar}, + journal = {Biometrics}, + number = {4}, + pages = {1175--1176}, + publisher = {International Biometric Society}, + title = {Confidence Bounds for the Difference between Two Probabilities}, + urldate = {2026-02-23}, + volume = {40}, + year = {1984} +} + + +@article{koopman1984, + title = {Confidence Intervals for the Ratio of Two Binomial Proportions}, + author = {Koopman, P. A. R.}, + year = {1984}, + month = {06}, + date = {1984-06}, + journal = {Biometrics}, + pages = {513}, + volume = {40}, + number = {2}, + doi = {10.2307/2531405}, + url = {http://dx.doi.org/10.2307/2531405} +} + +@article{tango1998, + title = {Equivalence test and confidence interval for the difference in proportions for the paired-sample design}, + author = {Tango, Toshiro}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {891--908}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + langid = {en} +} + +@article{tango1999, + title = {Improved confidence intervals for the difference between binomial proportions based on paired data by Robert G. Newcombe,Statistics in Medicine,17, 2635-2650 (1998)}, + author = {Tango, T.}, + year = {1999}, + month = {12}, + date = {1999-12-30}, + journal = {Statistics in Medicine}, + pages = {3511--3513}, + volume = {18}, + number = {24}, + doi = {10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + langid = {en} +} + +@article{tang2003, + title = {On tests of equivalence via non{-}unity relative risk for matched{-}pair design}, + author = {Tang, {Nian{-}Sheng} and Tang, {Man{-}Lai} and Chan, Ivan Siu Fung}, + year = {2003}, + month = {03}, + date = {2003-03-18}, + journal = {Statistics in Medicine}, + pages = {1217--1233}, + volume = {22}, + number = {8}, + doi = {10.1002/sim.1213}, + url = {http://dx.doi.org/10.1002/sim.1213}, + langid = {en} +} + +@article{nam2002, + title = {Analysis of the ratio of marginal probabilities in a matched{-}pair setting}, + author = {Nam, {Jun{-}mo} and Blackwelder, William C.}, + year = {2002}, + month = {02}, + date = {2002-02-19}, + journal = {Statistics in Medicine}, + pages = {689--699}, + volume = {21}, + number = {5}, + doi = {10.1002/sim.1017}, + url = {http://dx.doi.org/10.1002/sim.1017}, + langid = {en} +} + +@article{laud2018, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2018}, + month = {05}, + date = {2018-05}, + journal = {Pharmaceutical Statistics}, + pages = {290--293}, + volume = {17}, + number = {3}, + doi = {10.1002/pst.1855}, + url = {http://dx.doi.org/10.1002/pst.1855}, + langid = {en} +} + +@article{mehrotra2000, + title = {Minimum risk weights for comparing treatments in stratified binomial trials}, + author = {Mehrotra, Devan V. and Railkar, Radha}, + year = {2000}, + month = {03}, + date = {2000-03-30}, + journal = {Statistics in Medicine}, + pages = {811--825}, + volume = {19}, + number = {6}, + doi = {10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + langid = {en} +} + +@article{tonycai2005, + title = {One-sided confidence intervals in discrete distributions}, + author = {Tony Cai, T.}, + year = {2005}, + month = {04}, + date = {2005-04}, + journal = {Journal of Statistical Planning and Inference}, + pages = {63--88}, + volume = {131}, + number = {1}, + doi = {10.1016/j.jspi.2004.01.005}, + url = {http://dx.doi.org/10.1016/j.jspi.2004.01.005}, + langid = {en} +} + +@article{laud2014, + title = {Confidence intervals for the difference between independent binomial proportions: comparison using a graphical approach and moving averages}, + author = {Laud, Peter J. and Dane, Aaron}, + year = {2014}, + month = {08}, + date = {2014-08-27}, + journal = {Pharmaceutical Statistics}, + pages = {294--308}, + volume = {13}, + number = {5}, + doi = {10.1002/pst.1631}, + url = {http://dx.doi.org/10.1002/pst.1631}, + langid = {en} +} + +@article{bai2021, + title = {Confidence interval of risk difference by different statistical methods and its impact on the study conclusion in antibiotic non-inferiority trials}, + author = {Bai, Anthony D. and Komorowski, Adam S. and Lo, Carson K. L. and Tandon, Pranav and Li, Xena X. and Mokashi, Vaibhav and Cvetkovic, Anna and Findlater, Aidan and Liang, Laurel and Tomlinson, George and Loeb, Mark and Mertz, Dominik and , }, + year = {2021}, + month = {10}, + date = {2021-10-16}, + journal = {Trials}, + volume = {22}, + number = {1}, + doi = {10.1186/s13063-021-05686-8}, + url = {http://dx.doi.org/10.1186/s13063-021-05686-8}, + langid = {en} +} diff --git a/data/stat_method_tbl.csv b/data/stat_method_tbl.csv index 695dc845e..0f3a5aad2 100644 --- a/data/stat_method_tbl.csv +++ b/data/stat_method_tbl.csv @@ -1,70 +1,69 @@ -method_grp,method_subgrp,r_links,sas_links,python_links,comparison_links -Summary Statistics,Rounding,[R](R/rounding),[SAS](SAS/rounding),[Python](python/Rounding),[R vs SAS](Comp/r-sas_rounding) -Summary Statistics,Summary statistics,[R](R/summary-stats),[SAS](SAS/summary-stats),[Python](python/Summary_statistics),[R vs SAS](Comp/r-sas-summary-stats) -Summary Statistics,Skewness/Kurtosis,[R](R/summary_skew_kurt),[SAS](SAS/summary_skew_kurt),[Python](python/skewness_kurtosis),[R vs SAS](Comp/r-sas_summary_skew_kurt) -General Linear Models,One-sample t-test,[R](R/ttest_1Sample),[SAS](SAS/ttest_1Sample),[Python](python/one_sample_t_test),[R vs SAS](Comp/r-sas_ttest_1Sample) -General Linear Models,Paired t-test,[R](R/ttest_Paired),[SAS](SAS/ttest_Paired),[Python](python/paired_t_test),[R vs SAS](Comp/r-sas_ttest_Paired) -General Linear Models,Two-sample t-test,[R](R/ttest_2Sample),[SAS](SAS/ttest_2Sample),[Python](python/two_samples_t_test),[R vs SAS](Comp/r-sas_ttest_2Sample) -General Linear Models,ANOVA,[R](R/anova),[SAS](SAS/anova),[Python](python/anova),[R vs SAS](Comp/r-sas_anova) -General Linear Models,ANCOVA,[R](R/ancova),[SAS](SAS/ancova),[Python](python/ancova),[R vs SAS](Comp/r-sas_ancova) -General Linear Models,MANOVA,[R](R/manova),[SAS](SAS/manova),[Python](python/MANOVA),[R vs SAS](Comp/r-sas_manova) -General Linear Models,Linear regression,[R](R/linear-regression),[SAS](SAS/linear-regression),[Python](python/linear_regression),[R vs SAS](Comp/r-sas_linear-regression) -Generalized Linear Models,Logistic regression,[R](R/logistic_regr),[SAS](SAS/logistic-regr),[Python](python/logistic_regression),[R vs SAS](Comp/r-sas_logistic-regr) -Generalized Linear Models,Poisson/negative binomial regression,[R](R/count_data_regression),[SAS](SAS/count_data_regression),,[R vs SAS](Comp/r-sas_negbin) -Non-Parametric Analysis,Wilcoxon signed-rank test,[R](R/wilcoxonsr_hodges_lehman),[SAS/ StatXact](SAS/wilcoxonsr_HL),,[R vs SAS](Comp/r-sas-wilcoxonsr_HL) -Non-Parametric Analysis,Mann-Whitney U/Wilcoxon rank-sum test,[R](R/nonpara_wilcoxon_ranksum),[SAS](SAS/ranksum),,[R vs SAS](Comp/r-sas-wilcoxon-ranksum_hl) -Non-Parametric Analysis,Kolmogorov-Smirnov test,[R](R/kolmogorov-smirnov_test),,, -Non-Parametric Analysis,Kruskal-Wallis test,[R](R/kruskal_wallis),[SAS](SAS/kruskal_wallis),[Python](python/kruskal_wallis),[R vs SAS](Comp/r-sas_kruskalwallis) -Non-Parametric Analysis,Friedman test,[R](R/friedman_test),[SAS](SAS/SAS_Friedmantest),,[R vs SAS](Comp/r-sas_friedman) -Non-Parametric Analysis,Jonckheere-Terpstra test,[R](R/jonckheere),[SAS](SAS/jonchkheere_terpstra),,[R vs SAS](Comp/r-sas_jonckheere) -Non-Parametric Analysis,Hodges-Lehman estimator,[R](R/nparestimate),[SAS](SAS/nparestimate),, -Categorical Data Analysis,Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), -Categorical Data Analysis,McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) -Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, -Categorical Data Analysis,Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) -Categorical Data Analysis,Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) -Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Two Independent Proportions ,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Two Paired Proportions,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Proportions in Stratified Designs,,,, -Categorical Data Analysis,CIs - Poisson Exposure Adjusted Incidence Rates,,,, -Correlated Data Analysis,Linear Mixed Model Repeated Measure (MMRM),[R](R/mmrm),[SAS](SAS/mmrm),,[R vs SAS](Comp/r-sas_mmrm) -Correlated Data Analysis,Linear Mixed Model Random Effect,,,,[R vs SAS](Comp/Sas_vs_R_Mixed_Model) -Correlated Data Analysis,Generalized Linear Mixed Model (GLMM),[R](R/glmm),[SAS](SAS/glmm),,[R vs SAS](Comp/r-sas_glmm) -Correlated Data Analysis,Generalized Estimating Equation (GEE),[R](R/gee),[SAS](SAS/gee),,[R vs SAS](Comp/r-sas_gee) -Multiple Imputation - Continuous Data MAR,Linear regression imputation,[R](R/mi_mar_regression),[SAS](SAS/mi_mar_regression),, -Multiple Imputation - Continuous Data MAR,Predictive mean matching,[R](R/mi_mar_predictive_mean_match),,, -Multiple Imputation - Continuous Data MNAR,Tipping point analysis (delta adjustment),[R](R/tipping_point),[SAS](SAS/tipping_point),,[R vs SAS](Comp/r-sas_tipping_point) -Multiple Imputation - Continuous Data MNAR,Reference-based imputation/joint modeling,[R](R/rbmi_continuous_joint),[SAS](SAS/rbmi_continuous_joint_SAS),,[R vs SAS](Comp/r-sas_rbmi_continuous_joint) -Correlation,Pearson/Spearman/Kendall's Rank,[R](R/correlation),[SAS](SAS/correlation),[Python](python/correlation),[R vs SAS](Comp/r-sas_correlation) -Survival Models,Kaplan-Meier/log-rank test/Cox proportional hazards,[R](R/survival),[SAS](SAS/survival),,[R vs SAS](Comp/r-sas_survival) -Survival Models,Cause-specific hazards,[R](R/survival_csh),[SAS](SAS/survival_csh),,[R vs SAS](Comp/r-sas_survival_csh) -Survival Models,Accelerated failure time,[R](R/Accelerated_Failure_time_model),,, -Survival Models,Weighted log-rank test,[R](R/Weighted-log-rank),,, -Survival Models,Recurrent events,[R](R/recurrent_events),[SAS](SAS/recurrent_events),,[R vs SAS](Comp/r-sas_recurrent_events) -Survival Models,Cumulative incidence functions,[R](R/survival_cif),[SAS](SAS/survival_cif),,[R vs SAS](Comp/r-sas_survival_cif) -Survival Models,Tobit regression,[R](R/tobit regression),[SAS](SAS/tobit regression SAS),,[R vs SAS](Comp/r-sas_tobit) -Survival Models,Restricted Mean Survival Time (RMST),,[SAS](SAS/rmst),, -Bayesian Methods,Intro to Bayesian Analysis,,,, -Bayesian Methods,Repeated Measures MMRM,,,, -Sample size/ Power calculations,Intro to Sample Size,,,,[Summary](method_summary/sample_s_theory) -Sample size/ Power calculations,Superiority Single timepoint,[R](R/sample_s_superiority),[SAS](SAS/sample_s_superiority),,[R vs SAS](Comp/r-sas_s_size_sup) -Sample size/ Power calculations,Equivalence Single timepoint,[R](R/sample_s_equivalence),[SAS](SAS/sample_s_equivalence),, -Sample size/ Power calculations,Non-Inferiority Single timepoint,[R](R/sample_size_non-inferiority),[SAS](SAS/sample_s_noninferiority),, -Sample size/ Power calculations,Average BioEquivalence,[R](R/sample_size_average_bioequivalence),,, -Sample size/ Power calculations,Cochran-Armitage Test For Trend,[R](R/sample_s_test_of_trends),[SAS/ StatXact](SAS/sample_s_StatXact_test_of_trends),, -Sample size/ Power calculations,Group sequential designs-Survival,[R](R/gsd-tte),[SAS](SAS/gsd-tte),[East](East/gsd-tte),[R vs SAS vs East](Comp/r-east_gsd_tte) -Sample size/ Power calculations,Group sequential designs-Binary Endpoint,,,, -Causal inference/ Machine learning,Intro to Machine Learning,,,,[Summary](method_summary/stat_ml_methods) -Causal inference/ Machine learning,Propensity Score Matching,[R](R/causal_ps_matching),,,[R vs SAS](Comp/r-sas_psmatch) -Causal inference/ Machine learning,Propensity Score Weighting,,,,[R vs SAS](Comp/r-sas_psweight) -Causal inference/ Machine learning,Clustering,[R](R/Clustering_Knowhow),,, -Causal inference/ Machine learning,Principal Components Analysis (PCA),[R](R/PCA_analysis),,, -Causal inference/ Machine learning,Lasso,,,, -Causal inference/ Machine learning,Ridge Regression,,,, -Causal inference/ Machine learning,xgboost,[R](R/xgboost),,, -Other Methods,Survey statistics,[R](R/survey-stats-summary),[SAS](SAS/survey-stats-summary),[Python](python/survey-stats-summary),[R vs SAS vs Python](Comp/r-sas-python_survey-stats-summary) - - - - +method_grp,method_subgrp,r_links,sas_links,python_links,comparison_links +Summary Statistics,Rounding,[R](R/rounding),[SAS](SAS/rounding),[Python](python/Rounding),[R vs SAS](Comp/r-sas_rounding) +Summary Statistics,Summary statistics,[R](R/summary-stats),[SAS](SAS/summary-stats),[Python](python/Summary_statistics),[R vs SAS](Comp/r-sas-summary-stats) +Summary Statistics,Skewness/Kurtosis,[R](R/summary_skew_kurt),[SAS](SAS/summary_skew_kurt),[Python](python/skewness_kurtosis),[R vs SAS](Comp/r-sas_summary_skew_kurt) +General Linear Models,One-sample t-test,[R](R/ttest_1Sample),[SAS](SAS/ttest_1Sample),[Python](python/one_sample_t_test),[R vs SAS](Comp/r-sas_ttest_1Sample) +General Linear Models,Paired t-test,[R](R/ttest_Paired),[SAS](SAS/ttest_Paired),[Python](python/paired_t_test),[R vs SAS](Comp/r-sas_ttest_Paired) +General Linear Models,Two-sample t-test,[R](R/ttest_2Sample),[SAS](SAS/ttest_2Sample),[Python](python/two_samples_t_test),[R vs SAS](Comp/r-sas_ttest_2Sample) +General Linear Models,ANOVA,[R](R/anova),[SAS](SAS/anova),[Python](python/anova),[R vs SAS](Comp/r-sas_anova) +General Linear Models,ANCOVA,[R](R/ancova),[SAS](SAS/ancova),[Python](python/ancova),[R vs SAS](Comp/r-sas_ancova) +General Linear Models,MANOVA,[R](R/manova),[SAS](SAS/manova),[Python](python/MANOVA),[R vs SAS](Comp/r-sas_manova) +General Linear Models,Linear regression,[R](R/linear-regression),[SAS](SAS/linear-regression),[Python](python/linear_regression),[R vs SAS](Comp/r-sas_linear-regression) +Generalized Linear Models,Logistic regression,[R](R/logistic_regr),[SAS](SAS/logistic-regr),[Python](python/logistic_regression),[R vs SAS](Comp/r-sas_logistic-regr) +Generalized Linear Models,Poisson/negative binomial regression,[R](R/count_data_regression),[SAS](SAS/count_data_regression),,[R vs SAS](Comp/r-sas_negbin) +Non-Parametric Analysis,Wilcoxon signed-rank test,[R](R/wilcoxonsr_hodges_lehman),[SAS/ StatXact](SAS/wilcoxonsr_HL),,[R vs SAS](Comp/r-sas-wilcoxonsr_HL) +Non-Parametric Analysis,Mann-Whitney U/Wilcoxon rank-sum test,[R](R/nonpara_wilcoxon_ranksum),[SAS](SAS/ranksum),,[R vs SAS](Comp/r-sas-wilcoxon-ranksum_hl) +Non-Parametric Analysis,Kolmogorov-Smirnov test,[R](R/kolmogorov-smirnov_test),,, +Non-Parametric Analysis,Kruskal-Wallis test,[R](R/kruskal_wallis),[SAS](SAS/kruskal_wallis),[Python](python/kruskal_wallis),[R vs SAS](Comp/r-sas_kruskalwallis) +Non-Parametric Analysis,Friedman test,[R](R/friedman_test),[SAS](SAS/SAS_Friedmantest),,[R vs SAS](Comp/r-sas_friedman) +Non-Parametric Analysis,Jonckheere-Terpstra test,[R](R/jonckheere),[SAS](SAS/jonchkheere_terpstra),,[R vs SAS](Comp/r-sas_jonckheere) +Non-Parametric Analysis,Hodges-Lehman estimator,[R](R/nparestimate),[SAS](SAS/nparestimate),, +Categorical Data Analysis,Single proportion - Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), +Categorical Data Analysis,Two paired proportions - McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) +Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, +Categorical Data Analysis,Two independent proportions - Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) +Categorical Data Analysis,Stratified tables - Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) +Categorical Data Analysis,Intro to CIs for proportions,,,,[Summary](method_summary/ci_for_prop_intro) +Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) +Categorical Data Analysis,CIs - Two Independent Proportions ,,[SAS](SAS/ci_for_2indep_prop),, +Categorical Data Analysis,CIs - Two Paired Proportions,,,, +Categorical Data Analysis,CIs - Proportions in Stratified Designs,,,, +Categorical Data Analysis,CIs - Poisson Exposure Adjusted Incidence Rates,,,, +Correlated Data Analysis,Linear Mixed Model Repeated Measure (MMRM),[R](R/mmrm),[SAS](SAS/mmrm),,[R vs SAS](Comp/r-sas_mmrm) +Correlated Data Analysis,Linear Mixed Model Random Effect,,,,[R vs SAS](Comp/Sas_vs_R_Mixed_Model) +Correlated Data Analysis,Generalized Linear Mixed Model (GLMM),[R](R/glmm),[SAS](SAS/glmm),,[R vs SAS](Comp/r-sas_glmm) +Correlated Data Analysis,Generalized Estimating Equation (GEE),[R](R/gee),[SAS](SAS/gee),,[R vs SAS](Comp/r-sas_gee) +Multiple Imputation - Continuous Data MAR,Linear regression imputation,[R](R/mi_mar_regression),[SAS](SAS/mi_mar_regression),, +Multiple Imputation - Continuous Data MAR,Predictive mean matching,[R](R/mi_mar_predictive_mean_match),,, +Multiple Imputation - Continuous Data MNAR,Tipping point analysis (delta adjustment),[R](R/tipping_point),[SAS](SAS/tipping_point),,[R vs SAS](Comp/r-sas_tipping_point) +Multiple Imputation - Continuous Data MNAR,Reference-based imputation/joint modeling,[R](R/rbmi_continuous_joint),[SAS](SAS/rbmi_continuous_joint_SAS),,[R vs SAS](Comp/r-sas_rbmi_continuous_joint) +Correlation,Pearson/Spearman/Kendall's Rank,[R](R/correlation),[SAS](SAS/correlation),[Python](python/correlation),[R vs SAS](Comp/r-sas_correlation) +Survival Models,Kaplan-Meier/log-rank test/Cox proportional hazards,[R](R/survival),[SAS](SAS/survival),,[R vs SAS](Comp/r-sas_survival) +Survival Models,Cause-specific hazards,[R](R/survival_csh),[SAS](SAS/survival_csh),,[R vs SAS](Comp/r-sas_survival_csh) +Survival Models,Accelerated failure time,[R](R/Accelerated_Failure_time_model),,, +Survival Models,Weighted log-rank test,[R](R/Weighted-log-rank),,, +Survival Models,Recurrent events,[R](R/recurrent_events),[SAS](SAS/recurrent_events),,[R vs SAS](Comp/r-sas_recurrent_events) +Survival Models,Cumulative incidence functions,[R](R/survival_cif),[SAS](SAS/survival_cif),,[R vs SAS](Comp/r-sas_survival_cif) +Survival Models,Tobit regression,[R](R/tobit regression),[SAS](SAS/tobit regression SAS),,[R vs SAS](Comp/r-sas_tobit) +Survival Models,Restricted Mean Survival Time (RMST),,[SAS](SAS/rmst),, +Bayesian Methods,Intro to Bayesian Analysis,,,, +Bayesian Methods,Repeated Measures MMRM,,,, +Sample size/ Power calculations,Intro to Sample Size,,,,[Summary](method_summary/sample_s_theory) +Sample size/ Power calculations,Superiority Single timepoint,[R](R/sample_s_superiority),[SAS](SAS/sample_s_superiority),,[R vs SAS](Comp/r-sas_s_size_sup) +Sample size/ Power calculations,Equivalence Single timepoint,[R](R/sample_s_equivalence),[SAS](SAS/sample_s_equivalence),, +Sample size/ Power calculations,Non-Inferiority Single timepoint,[R](R/sample_size_non-inferiority),[SAS](SAS/sample_s_noninferiority),, +Sample size/ Power calculations,Average BioEquivalence,[R](R/sample_size_average_bioequivalence),,, +Sample size/ Power calculations,Cochran-Armitage Test For Trend,[R](R/sample_s_test_of_trends),[SAS/ StatXact](SAS/sample_s_StatXact_test_of_trends),, +Sample size/ Power calculations,Group sequential designs-Survival,[R](R/gsd-tte),[SAS](SAS/gsd-tte),[East](East/gsd-tte),[R vs SAS vs East](Comp/r-east_gsd_tte) +Sample size/ Power calculations,Group sequential designs-Binary Endpoint,,,, +Causal inference/ Machine learning,Intro to Machine Learning,,,,[Summary](method_summary/stat_ml_methods) +Causal inference/ Machine learning,Propensity Score Matching,[R](R/causal_ps_matching),,,[R vs SAS](Comp/r-sas_psmatch) +Causal inference/ Machine learning,Propensity Score Weighting,,,,[R vs SAS](Comp/r-sas_psweight) +Causal inference/ Machine learning,Clustering,[R](R/Clustering_Knowhow),,, +Causal inference/ Machine learning,Principal Components Analysis (PCA),[R](R/PCA_analysis),,, +Causal inference/ Machine learning,Lasso,,,, +Causal inference/ Machine learning,Ridge Regression,,,, +Causal inference/ Machine learning,xgboost,[R](R/xgboost),,, +Other Methods,Survey statistics,[R](R/survey-stats-summary),[SAS](SAS/survey-stats-summary),[Python](python/survey-stats-summary),[R vs SAS vs Python](Comp/r-sas-python_survey-stats-summary) + + diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd new file mode 100644 index 000000000..d79b98db0 --- /dev/null +++ b/method_summary/ci_for_prop_intro.qmd @@ -0,0 +1,91 @@ +--- + title: "Confidence intervals for Proportions: General Information" + bibliography: references.bib + csl: nature.csl +--- + +## Introduction + +The methods to use for calculating confidence intervals (CIs) for binomial proportions depend on the situation: + +- 1 sample proportion, p (1 proportion calculated from 1 group of subjects) + +- 2 sample proportions ($p_1$ and $p_2$) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). + + - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) + + - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). + +Reporting a CI for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the CI spans a pre-specified NI margin or not. + +Some sources suggest selecting a different method depending on your sample size and whether your proportion takes an extreme value (i.e. close to 0 or 1). Similarly, for a 2x2 table, there is a widely-held belief that a so-called 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative, but options are now available for CI methods that have appropriate coverage properties across the whole parameter space. These are easily obtained with modern computing software, so there is no need for a data-driven approach to method selection or reliance on overly conservative methods. + +The poor performance of the approximate normal ('Wald') methods is well documented - they can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that they should be avoided[@brown2001, p128]. Wald intervals remain a default output component in SAS, so continue to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] + +### Strictly conservative vs proximate coverage + +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. Some CI methods are designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either 'strictly conservative' or 'proximate'[@laud2017]. It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001]. The debate makes it difficult to state in simple objective terms whether one method is 'better' than another. + +Many CI methods are designed to achieve proximate coverage, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee strictly conservative coverage, while others include an optional adjustment ('continuity correction') to emulate the same effect using asymptotic formulae. The term 'adjustment' may be used instead of 'correction' to avoid the implication that the non-adjusted methods are inferior[@newcombe1998c][@campbell2007]. + +It is worth noting that although one might assume that regulatory authorities would insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method (e.g. see ). In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. + +### Consistency with hypothesis tests, and one-sided coverage + +It is desirable (and in some analysis contexts, essential) for a reported CI to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This applies for the special case of a test for association (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. + +A related issue is whether it is sufficient for two-sided coverage probability to be aligned with the nominal $(1-\alpha)$, or whether the tail probabilities (i.e. one-sided non-coverage at each end) should also be aligned with the nominal $\alpha/2$. Clearly the latter 'equal-tailed' or symmetric coverage criterion (described as 'central interval location' by Newcombe) is essential for non-inferiority testing, but it is also a desirable general property for any CI. + +# Overarching methods + +There are some methods that apply (or have variants which apply) across all contrast parameters (i.e. $\theta=p$ or $\theta=RD$ etc.), which are outlined below. + +### Normal Approximation Methods (Also known as the Wald Methods) + +The Wald confidence interval is constructed simply from the the point estimate plus and minus a multiple of its estimated standard error: $(L_{Wald}, U_{Wald}) = \hat \theta \pm z_{\alpha/2} \sqrt{\hat V(\hat\theta)}$ where $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$. + +For RR and OR, $\theta$ is first log-transformed to improve the normal approximation, and then $(L_{Wald}, U_{Wald}) = exp[ ln(\hat \theta) \pm z_{\alpha/2} \sqrt{\hat V(ln(\hat\theta))}]$, which requires a modification for boundary cases. + +### Asymptotic Score Methods + +The Score confidence interval is also based on an asymptotic normal approximation, but uses a score statistic $Z(\theta) = S(\theta)/\sqrt {\tilde V}$. This statistic is based on a contrast function $S(\theta)$ (e.g. for the single proportion, $S(p) = \hat p - p$, or for RD, $S(\theta) = \hat p_1 - \hat p_2 - \theta$), and a maximum likelihood estimate of its variance $\tilde V = V(S(\theta))$, which is also function of $\theta$. + +$Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion (Wilson score interval), this process simplifies to solving a quadratic equation in $p$. + +For the contrast parameters, score methods were derived for independent proportions by Miettinen & Nurminen and others in the 1980s[@miettinen1985][@Mee1984][@koopman1984], and related methods for paired data followed later[@tango1998][@tango1999][@nam2002][@tang2003]. In each case, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ (or for the paired case, the cell probability $p_{21}$) for the given value of $\theta$. + +### SCAS Methods + +Originating from papers published in 1985-1990 by Gart and Nam, the skewness-corrected asymptotic score ('SCAS') methods introduce a further term within the score statistic that involves $\tilde \mu$, the third central moment of $S(\theta)$: + +$$ +Z(\theta) = \frac{S(\theta)}{\sqrt {\tilde V}} - \frac{(Z(\theta)^2 - 1)\tilde \mu_3}{6 \tilde V^{3/2}} +$$ + +Hence these are an extension of the asymptotic score methods, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage[@tonycai2005]. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], and a further publication currently under review for paired data. + +SCAS intervals generally achieve one-sided coverage that is very close to the nominal $\alpha/2$, which naturally also leads to excellent two-sided coverage. Symmetric strictly conservative coverage can be achieved by adding a continuity adjustment. + +The SCAS method is not implemented in SAS PROC FREQ, but can be obtained using the %SCORECI and %PAIRBINCI macros from . + +### MOVER Methods (Also known as the Newcombe Method) + +This class of methods was popularised by Newcombe for the RD contrast, and is therefore labelled as the Newcombe method in current SAS PROC FREQ documentation. It has since been generalised to other contrasts and for paired data (though not implemented in SAS), and renamed as the Method of Variance Estimates Recovery (MOVER). It is also referred to as the 'Square-and-add' approach. + +MOVER intervals are produced by combining CIs for each of the two separate proportions involved (and an estimate of their correlation, in the paired data case). Originally Wilson Score intervals were used (hence earlier versions of SAS (e.g. v9.3) labelled this method as WILSON or SCORE), but improved performance may be obtained by using Jeffreys intervals instead. + +### 'Exact' Methods + +So-called 'exact' methods are designed to guarantee strictly conservative coverage. (Newcombe points out some problems with the use of the term 'exact' here, hence the quotation marks.) For the single proportion, the Clopper-Pearson confidence interval is the range of p for which $P(X \ge x | p) = \alpha / 2$ and $P(X \le x | p) = \alpha / 2$. More complex calculations are involved for the contrast parameters, which eliminate the nuisance parameter by taking the supremum of all p-values across the range of combinations of $p_1$ and $p_2$ for a given $\theta$. + +There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). + +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $3/8 \times n$ [@mehrotra2000], or $0.25 \times n$[@laud2017, Appendix S2] (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used to achieve coverage that is at least $(1-\alpha)$ nearly all of the time. + +### 'Mid-P' Methods + +'Exact' methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. + +Although theoretically possible for any contrast, the mid-P method is only implemented in SAS for CIs for the single proportion and the odds ratio contrast. + +## References diff --git a/method_summary/nature.csl b/method_summary/nature.csl new file mode 100644 index 000000000..3b1e3d7a2 --- /dev/null +++ b/method_summary/nature.csl @@ -0,0 +1,189 @@ + + diff --git a/method_summary/references.bib b/method_summary/references.bib new file mode 100644 index 000000000..c7086d9ae --- /dev/null +++ b/method_summary/references.bib @@ -0,0 +1,219 @@ + +@article{brown2001, + title = {Interval Estimation for a Binomial Proportion}, + author = {Brown, Lawrence D. and Cai, T. Tony and DasGupta, Anirban}, + year = {2001}, + month = {05}, + date = {2001-05-01}, + journal = {Statistical Science}, + volume = {16}, + number = {2}, + doi = {10.1214/ss/1009213286}, + url = {http://dx.doi.org/10.1214/ss/1009213286} +} + +@book{newcombe2012, + title = {Confidence Intervals for Proportions and Related Measures of Effect Size}, + author = {Newcombe, Robert Gordon}, + year = {2012}, + month = {08}, + date = {2012-08-25}, + publisher = {CRC Press}, + doi = {10.1201/b12670}, + url = {http://dx.doi.org/10.1201/b12670}, + langid = {en} +} + +@article{newcombe1998c, + title = {Two-sided confidence intervals for the single proportion: comparison of seven methods}, + author = {Newcombe, Robert G.}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {857--872}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + langid = {en} +} + +@article{campbell2007, + title = {Chi{-}squared and Fisher{\textendash}Irwin tests of two{-}by{-}two tables with small sample recommendations}, + author = {Campbell, Ian}, + year = {2007}, + month = {02}, + date = {2007-02-21}, + journal = {Statistics in Medicine}, + pages = {3661--3675}, + volume = {26}, + number = {19}, + doi = {10.1002/sim.2832}, + url = {http://dx.doi.org/10.1002/sim.2832}, + langid = {en} +} + +@article{laud2017, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2017}, + month = {06}, + date = {2017-06-22}, + journal = {Pharmaceutical Statistics}, + pages = {334--348}, + volume = {16}, + number = {5}, + doi = {10.1002/pst.1813}, + url = {http://dx.doi.org/10.1002/pst.1813}, + langid = {en} +} + +@article{miettinen1985, + title = {Comparative analysis of two rates}, + author = {Miettinen, Olli and Nurminen, Markku}, + year = {1985}, + month = {04}, + date = {1985-04}, + journal = {Statistics in Medicine}, + pages = {213--226}, + volume = {4}, + number = {2}, + doi = {10.1002/sim.4780040211}, + url = {http://dx.doi.org/10.1002/sim.4780040211}, + langid = {en} +} + +@article{Mee1984, + ISSN = {0006341X, 15410420}, + URL = {http://www.jstor.org/stable/2531174}, + author = {Robert W. Mee and Dan Anbar}, + journal = {Biometrics}, + number = {4}, + pages = {1175--1176}, + publisher = {International Biometric Society}, + title = {Confidence Bounds for the Difference between Two Probabilities}, + urldate = {2026-02-23}, + volume = {40}, + year = {1984} +} + + +@article{koopman1984, + title = {Confidence Intervals for the Ratio of Two Binomial Proportions}, + author = {Koopman, P. A. R.}, + year = {1984}, + month = {06}, + date = {1984-06}, + journal = {Biometrics}, + pages = {513}, + volume = {40}, + number = {2}, + doi = {10.2307/2531405}, + url = {http://dx.doi.org/10.2307/2531405} +} + +@article{tango1998, + title = {Equivalence test and confidence interval for the difference in proportions for the paired-sample design}, + author = {Tango, Toshiro}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {891--908}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + langid = {en} +} + +@article{tango1999, + title = {Improved confidence intervals for the difference between binomial proportions based on paired data by Robert G. Newcombe,Statistics in Medicine,17, 2635-2650 (1998)}, + author = {Tango, T.}, + year = {1999}, + month = {12}, + date = {1999-12-30}, + journal = {Statistics in Medicine}, + pages = {3511--3513}, + volume = {18}, + number = {24}, + doi = {10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + langid = {en} +} + +@article{tang2003, + title = {On tests of equivalence via non{-}unity relative risk for matched{-}pair design}, + author = {Tang, {Nian{-}Sheng} and Tang, {Man{-}Lai} and Chan, Ivan Siu Fung}, + year = {2003}, + month = {03}, + date = {2003-03-18}, + journal = {Statistics in Medicine}, + pages = {1217--1233}, + volume = {22}, + number = {8}, + doi = {10.1002/sim.1213}, + url = {http://dx.doi.org/10.1002/sim.1213}, + langid = {en} +} + +@article{nam2002, + title = {Analysis of the ratio of marginal probabilities in a matched{-}pair setting}, + author = {Nam, {Jun{-}mo} and Blackwelder, William C.}, + year = {2002}, + month = {02}, + date = {2002-02-19}, + journal = {Statistics in Medicine}, + pages = {689--699}, + volume = {21}, + number = {5}, + doi = {10.1002/sim.1017}, + url = {http://dx.doi.org/10.1002/sim.1017}, + langid = {en} +} + +@article{laud2018, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2018}, + month = {05}, + date = {2018-05}, + journal = {Pharmaceutical Statistics}, + pages = {290--293}, + volume = {17}, + number = {3}, + doi = {10.1002/pst.1855}, + url = {http://dx.doi.org/10.1002/pst.1855}, + langid = {en} +} + +@article{mehrotra2000, + title = {Minimum risk weights for comparing treatments in stratified binomial trials}, + author = {Mehrotra, Devan V. and Railkar, Radha}, + year = {2000}, + month = {03}, + date = {2000-03-30}, + journal = {Statistics in Medicine}, + pages = {811--825}, + volume = {19}, + number = {6}, + doi = {10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + langid = {en} +} + +@article{tonycai2005, + title = {One-sided confidence intervals in discrete distributions}, + author = {Tony Cai, T.}, + year = {2005}, + month = {04}, + date = {2005-04}, + journal = {Journal of Statistical Planning and Inference}, + pages = {63--88}, + volume = {131}, + number = {1}, + doi = {10.1016/j.jspi.2004.01.005}, + url = {http://dx.doi.org/10.1016/j.jspi.2004.01.005}, + langid = {en} +}