From d850a8582b02b76a6232da316e37ab8b5f5b7a7f Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Wed, 18 Feb 2026 18:29:56 +0000 Subject: [PATCH 01/58] split into separate pages for single/independent/paired proportions, and edit introduction text --- SAS/ci_for_2indep_prop.qmd | 168 +++++++++++++++++++++++++ SAS/ci_for_paired_prop.qmd | 238 ++++++++++++++++++++++++++++++++++++ SAS/ci_for_prop.qmd | 243 ------------------------------------- 3 files changed, 406 insertions(+), 243 deletions(-) create mode 100644 SAS/ci_for_2indep_prop.qmd create mode 100644 SAS/ci_for_paired_prop.qmd diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd new file mode 100644 index 000000000..1bee3d230 --- /dev/null +++ b/SAS/ci_for_2indep_prop.qmd @@ -0,0 +1,168 @@ +--- +title: "Confidence Intervals for Independent Proportions in SAS" +--- + +## Introduction + +The methods to use for calculating a confidence interval (CI) for independent proportions (p1 and p2) depend on the contrast parameter of interest. You might wish to quantify the simple difference between the proportions ('risk difference' RD = p1 - p2), or the ratio ( 'relative risk' RR = p1 / p2). Another option (for consistency with logistic regression output for example) is the Odds Ratio OR = (p1 (1-p2) / (p2 (1-p1)). + +The production of a confidence interval for such contrasts is useful for giving a clinically interpretable quantification of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. + +### Strictly conservative vs proximate coverage + +Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p1 and p2. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction"), or employ computationally intensive approaches to achieve the former. It is worth noting that although one might assume that regulatory bodies would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen method. + +### Consistency with hypothesis tests + +Depending on the analysis plan, it may be desirable (or in the case of a non-inferority trial, essential) for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS PROC FREQ for the asymptotic methods, such consistency is provided by the Miettinen-Nurminen CI, which is based on the same test statistic as the Farrington-Manning test\*\*\*. For the SCAS method, the macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. + +If an EXACT statement is used... + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{sas} +#| eval: false +data adcibc2 (keep=trt resp) ; + set adcibc; + if aval gt 4 then resp="Yes"; + else resp="No"; + if trtp="Placebo" then trt="PBO"; + else trt="Act"; +run; +``` + +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558. + +```{sas} +#| eval: false +proc freq data=adcibc2; + table trt*resp/ nopct nocol; +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") +``` + +## Methods for Calculating Confidence Intervals for 2 independent samples proportion difference + +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The most commonly used are: Wald with continuity correction and Wilson with continuity correction. The Wilson method may be more applicable when sample sizes are smaller and/or the proportion is closer to 0 or 1, since reliance on a normal distribution may be inappropriate. + +SAS is able to calculate CIs using the following methods: Exact, Agresti/Caffo, Wald, Wald with Continuity correction, Wilson/Newcombe Score, Wilson/Newcombe score with continuity correction, Miettinen and Nurminen, and Hauck-Anderson. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. + +```{sas} +#| eval: false +# Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(wald newcombe ac ha mn)); +run; +# exact method +proc freq data=adcibc2 order=data; + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=(exact)); +run; +``` + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) + +In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. + +The difference between two independent sample proportions is calculated as: $D= \hat p_1-\hat p_2$ + +A confidence interval for the difference between two independent proportions $D$ can be calculated using: + +$D\approx \hat D \pm z_\alpha * SE(\hat p)$, + +where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, and + +$SE (\hat p) = sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ + +With continuity correction, the equation becomes + +$D\approx \hat D \pm (CC + z_\alpha * SE(\hat p))$, + +where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$,\ +and $SE (\hat p)$ = $sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$\ +and + +$CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ + +### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) + +Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. + +Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. + +Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. + +Let D = p1-p2 (the difference between the observed proportions) + +The Confidence interval for the difference between two population proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ + +## Example Code using PROC FREQ + +It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. + +Options for riskdiff(CL=XXX) consist of AC:Agresti-Caffo, EXACT=exact, HA:Hauck-Anderson, MN or SCORE:Miettinen-Nurminen (another type of Score CI), WILSON or NEWCOMBE: Wilson method described above, and WALD: normal approximation wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. + +```{sas} +#| eval: false +proc sort data=adcibc2; +by trt descending resp; +run; + +# without continuity correction +proc freq data=adcibc2 order=data; +table trt*resp/riskdiff(CL=(wald wilson)); +run; + +# with continuity correction +proc freq data=adcibc2 order=data; +table trt*resp/riskdiff(CORRECT CL=(wald wilson)); +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% + knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png")` +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") +``` + +## Reference + +1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) + +2. [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f) + +3. [Confidence intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf) + +4. Brown LD, Cai TT, DasGupta A (2001). "Interval estimation for a binomial proportion", Statistical Science 16(2):101-133 + +5. Newcombe RG (1998). "Two-sided confidence intervals for the single proportion: comparison of seven methods", Statistics in Medicine 17(8):857-872 + +6. Clopper,C.J.,and Pearson,E.S.(1934),"The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial", Biometrika 26, 404--413. + +7. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. + +8. Laud PJ (2017) Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 16: 334-348 + +9. Laud PJ (2018) Corrigendum: Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 17: 290-293 + +10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 + +11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. + +12. diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd new file mode 100644 index 000000000..da74d4999 --- /dev/null +++ b/SAS/ci_for_paired_prop.qmd @@ -0,0 +1,238 @@ +--- +title: "Confidence intervals for Paired Proportions in SAS" +--- + +## Introduction + +The methods to use for calculating a confidence interval (CI) for independent proportions (p1 and p2) depend on the contrast parameter of interest. You might wish to quantify the simple difference between the proportions ('risk difference' RD = p1 - p2), or the ratio ( 'relative risk' RR = p1 / p2). Another option (slightly more obscure) is the Conditional Odds Ratio OR, but note that this .... + +The production of a confidence interval for such contrasts is useful for giving a clinically interpretable quantification of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. + +### Strictly conservative vs proximate coverage + +Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction") to achieve the former. + +### Consistency with hypothesis tests + +Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS for the asymptotic methods, such consistency is provided by the ... + +### + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{sas} +#| eval: false +data adcibc2 (keep=trt resp) ; + set adcibc; + if aval gt 4 then resp="Yes"; + else resp="No"; + if trtp="Placebo" then trt="PBO"; + else trt="Act"; +run; +``` + +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). + +```{sas} +#| eval: false +proc freq data=adcibc2; + table trt*resp/ nopct nocol; +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") +``` + +## Methods for Calculating Confidence Intervals for a matched pair proportion difference + +You may experience paired data in any of the following types of situation: + +- Tumour assesssments classified as Progressive Disease or Not Progressive Disease performed by an Investigator and separately by an independent panel. + +- A paired case-control study (each subject taking active treatment is matched to a patient taking control) + +- A cross-over trial where the same subjects take both medications + +In all these cases, the calculated proportions for the 2 groups are not independent. + +Using a cross over study as our example, a 2 x 2 table can be formed as follows: + ++-----------------------+---------------+---------------+---------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+===============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+---------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+---------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+---------------+ + +The proportions of subjects responding on each treatment are: + +Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ + +Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ + +Suppose : + ++-----------------------+----------------+----------------+------------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+================+================+==================+ +| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | ++-----------------------+----------------+----------------+------------------+ +| Active Response = No | t = 6 | u = 5 | t+u = 11 | ++-----------------------+----------------+----------------+------------------+ +| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | ++-----------------------+----------------+----------------+------------------+ + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) + +In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. Hence the SE for the difference and 95% confidence interval can be calculated using the following equations. + +$SE(D)=\frac{1}{n} * sqrt(s+t-\frac{(s-t)^2}{n})$ + +$D-z_\alpha * SE(D)$ to $D+z_\alpha * SE(D)$ + +where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, + +D=(15-6) /46 = 0.196 + +SE(D) = 1/ 46 \* sqrt (15+6- (((15+6)\^2)/46) ) = 0.0956 + +Lower CI= 0.196 - 1.96 \*0.0956 = 0.009108 + +Upper CI = 0.196 + 1.96 \* 0.0956 = 0.382892 + +### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) + +Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. + +Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. + +Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. + +We then define $\phi$ which is used to correct for $\hat p_1$ and $\hat p_2$ not being independent. As the samples are related, $\phi$ is usually positive and thus makes the confidence interval smaller (narrower). + +If any of r+s, t+u, r+t, s+u are zero, then set $\phi$ to be 0. + +Otherwise we calculate A, B and C, and $\phi=C / sqrt A$ + +In the above: $A=(r+s)(t+u)(r+t)(s+u)$ and $B=(ru-st)$ + +To calculate C follow the table below. n=sample size. + +| Condition of B | Set C equal to | +|---------------------------|----------------| +| If B is greater than n/2 | B - n/2 | +| If B is between 0 and n/2 | 0 | +| If B is less than 0 | B | + +Let D = p1-p2 (the difference between the observed proportions of responders) + +The Confidence interval for the difference between two population proportions is: $D - sqrt((p_1-l_1)^2-2\phi(p_1-l_1)(u_2-p_2)+(u_2-p_2)^2 )$ to + +$D + sqrt((p_2-l_2)^2-2\phi(p_2-l_2)(u_1-p_1)+(u_1-p_1)^2 )$ + +First using the Wilson Method equations for each of the individual single samples 1 and 2. + +| | Active | Placebo | +|----------|------------|------------| +| a | 73.842 | 55.842 | +| b | 13.728 | 11.974 | +| c | 99.683 | 99.683 | +| Lower CI | 0.603 = L1 | 0.440 = L2 | +| Upper CI | 0.878 = U1 | 0.680 = U2 | + +$A=(r+s)(t+u)(r+t)(s+u)$ = 9450000 + +B=10 + +C= 0 (as B is between 0 and n/2) + +$\phi$ = 0. + +Hence the middle part of the equation simplies to 0, and becomes simply: + +Lower CI = $D - sqrt((p_1-l_1)^2+(u_2-p_2)^2 )$ = 0.196 - sqrt \[ (0.761-0.603)\^2 + (0.680-0.565) \^2 \] + +Upper CI = $D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 )$ = 0.196 + sqrt \[ (0.565-0.440)\^2 + (0.878-0.761) \^2 \] + +CI= 0.00032 to 0.367389 + +## Example Code using PROC FREQ + +Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. + +Instead it calculates the risk difference = (r / (r+s) - t / (t+u) ) + +Which in the example above is: 20/ (20+15) - 6 / (6+5) = 0.0296 + +This is more applicable when you have exposed / non-exposed groups looking at who has experienced the outcome. SAS Proc Freq has 3 methods for analysis of paired data using a Common risk difference. + +The default method is Mantel-Haenszel confidence limits. SAS can also Score (Miettinen-Nurminen) CIs and Stratified Newcombe CIs (constructed from stratified Wilson Score CIs). See [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for equations. + +As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. + +```{sas} +#| eval: false +data dat_used; + input ID$ PBO ACT @@; + datalines; + 001 0 0 002 0 0 003 0 0 004 0 0 005 0 0 + 006 1 0 007 1 0 008 1 0 009 1 0 010 1 0 + 011 1 0 012 0 1 013 0 1 014 0 1 015 0 1 + 016 0 1 017 0 1 018 0 1 019 0 1 020 0 1 + 021 0 1 022 0 1 023 0 1 024 0 1 025 0 1 + 026 0 1 027 1 1 028 1 1 029 1 1 030 1 1 + 031 1 1 032 1 1 033 1 1 034 1 1 035 1 1 + 036 1 1 037 1 1 038 1 1 039 1 1 040 1 1 + 041 1 1 042 1 1 043 1 1 044 1 1 045 1 1 + 046 1 1 + ; +run; + +proc freq data=dat_used; + table ACT*PBO/commonriskdiff(cl=MH); +run; +``` + +```{r} +#| echo: false +#| fig-align: center +#| out-width: 50% +knitr::include_graphics("../images/ci_for_prop/riskdiff_matched.png") +``` + +## Reference + +1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) + +2. [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f) + +3. [Confidence intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf) + +4. Brown LD, Cai TT, DasGupta A (2001). "Interval estimation for a binomial proportion", Statistical Science 16(2):101-133 + +5. Newcombe RG (1998). "Two-sided confidence intervals for the single proportion: comparison of seven methods", Statistics in Medicine 17(8):857-872 + +6. Clopper,C.J.,and Pearson,E.S.(1934),"The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial", Biometrika 26, 404--413. + +7. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. + +8. Laud PJ (2017) Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 16: 334-348 + +9. Laud PJ (2018) Corrigendum: Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 17: 290-293 + +10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 + +11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. + +12. diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 2b6728dae..771e22764 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -237,249 +237,6 @@ knitr::include_graphics("../images/ci_for_prop/binomial_prop_cc_act.png") SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. `ods output binomialcls=bcl;` and then using the bcl dataset, in a data step to put the variable out to the number of decimal places we require.\ 10 decimal places shown here ! `lowercl2=put(lowercl,12.10);` -## Methods for Calculating Confidence Intervals for 2 independent samples proportion difference - -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The most commonly used are: Wald with continuity correction and Wilson with continuity correction. The Wilson method may be more applicable when sample sizes are smaller and/or the proportion is closer to 0 or 1, since reliance on a normal distribution may be inappropriate. - -SAS is able to calculate CIs using the following methods: Exact, Agresti/Caffo, Wald, Wald with Continuity correction, Wilson/Newcombe Score, Wilson/Newcombe score with continuity correction, Miettinen and Nurminen, and Hauck-Anderson. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. - -`{sas} #| eval: false # Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods proc freq data=adcibc2 order=data; table trt*resp /riskdiff(CL=(wald wilson ac ha mn)); run; # exact method proc freq data=adcibc2 order=data; exact riskdiff (method=noscore); table trt*resp/riskdiff(CL=(exact)); run;}` - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) - -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. - -The difference between two independent sample proportions is calculated as: $D= \hat p_1-\hat p_2$ - -A confidence interval for the difference between two independent proportions $D$ can be calculated using: - -$D\approx \hat D \pm z_\alpha * SE(\hat p)$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, and - -$SE (\hat p) = sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ - -With continuity correction, the equation becomes - -$D\approx \hat D \pm (CC + z_\alpha * SE(\hat p))$, - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$,\ -and $SE (\hat p)$ = $sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$\ -and - -$CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ - -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) - -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. - -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. - -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. - -Let D = p1-p2 (the difference between the observed proportions) - -The Confidence interval for the difference between two population proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ - -## Example Code using PROC FREQ - -It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. - -Options for riskdiff(CL=XXX) consist of AC:Agresti-Caffo, EXACT=exact, HA:Hauck-Anderson, MN or SCORE:Miettinen-Nurminen (another type of Score CI), WILSON or NEWCOMBE: Wilson method described above, and WALD: normal approximation wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. - -```{sas} -#| eval: false -proc sort data=adcibc2; -by trt descending resp; -run; - -# without continuity correction -proc freq data=adcibc2 order=data; -table trt*resp/riskdiff(CL=(wald wilson)); -run; - -# with continuity correction -proc freq data=adcibc2 order=data; -table trt*resp/riskdiff(CORRECT CL=(wald wilson)); -run; -``` - - -```{r} #| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") -``` - -```{r} -#| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") -``` - -## Methods for Calculating Confidence Intervals for a matched pair proportion difference - -You may experience paired data in any of the following types of situation: - -- Tumour assesssments classified as Progressive Disease or Not Progressive Disease performed by an Investigator and separately by an independent panel. - -- A paired case-control study (each subject taking active treatment is matched to a patient taking control) - -- A cross-over trial where the same subjects take both medications - -In all these cases, the calculated proportions for the 2 groups are not independent. - -Using a cross over study as our example, a 2 x 2 table can be formed as follows: - -+-----------------------+---------------+---------------+---------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+===============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+---------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+---------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+---------------+ - -The proportions of subjects responding on each treatment are: - -Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ - -Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ - -Suppose : - -+-----------------------+---------------+---------------+------------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+==================+ -| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | -+-----------------------+---------------+---------------+------------------+ -| Active Response = No | t = 6 | u = 5 | t+u = 11 | -+-----------------------+---------------+---------------+------------------+ -| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | -+-----------------------+---------------+---------------+------------------+ - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) - -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. Hence the SE for the difference and 95% confidence interval can be calculated using the following equations. - -$SE(D)=\frac{1}{n} * sqrt(s+t-\frac{(s-t)^2}{n})$ - -$D-z_\alpha * SE(D)$ to $D+z_\alpha * SE(D)$ - -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, - -D=(15-6) /46 = 0.196 - -SE(D) = 1/ 46 \* sqrt (15+6- (((15+6)\^2)/46) ) = 0.0956 - -Lower CI= 0.196 - 1.96 \*0.0956 = 0.009108 - -Upper CI = 0.196 + 1.96 \* 0.0956 = 0.382892 - -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) - -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. - -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. - -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. - -We then define $\phi$ which is used to correct for $\hat p_1$ and $\hat p_2$ not being independent. As the samples are related, $\phi$ is usually positive and thus makes the confidence interval smaller (narrower). - -If any of r+s, t+u, r+t, s+u are zero, then set $\phi$ to be 0. - -Otherwise we calculate A, B and C, and $\phi=C / sqrt A$ - -In the above: $A=(r+s)(t+u)(r+t)(s+u)$ and $B=(ru-st)$ - -To calculate C follow the table below. n=sample size. - -| Condition of B | Set C equal to | -|---------------------------|----------------| -| If B is greater than n/2 | B - n/2 | -| If B is between 0 and n/2 | 0 | -| If B is less than 0 | B | - -Let D = p1-p2 (the difference between the observed proportions of responders) - -The Confidence interval for the difference between two population proportions is: $D - sqrt((p_1-l_1)^2-2\phi(p_1-l_1)(u_2-p_2)+(u_2-p_2)^2 )$ to - -$D + sqrt((p_2-l_2)^2-2\phi(p_2-l_2)(u_1-p_1)+(u_1-p_1)^2 )$ - -First using the Wilson Method equations for each of the individual single samples 1 and 2. - -| | Active | Placebo | -|----------|------------|------------| -| a | 73.842 | 55.842 | -| b | 13.728 | 11.974 | -| c | 99.683 | 99.683 | -| Lower CI | 0.603 = L1 | 0.440 = L2 | -| Upper CI | 0.878 = U1 | 0.680 = U2 | - -$A=(r+s)(t+u)(r+t)(s+u)$ = 9450000 - -B=10 - -C= 0 (as B is between 0 and n/2) - -$\phi$ = 0. - -Hence the middle part of the equation simplies to 0, and becomes simply: - -Lower CI = $D - sqrt((p_1-l_1)^2+(u_2-p_2)^2 )$ = 0.196 - sqrt \[ (0.761-0.603)\^2 + (0.680-0.565) \^2 \] - -Upper CI = $D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 )$ = 0.196 + sqrt \[ (0.565-0.440)\^2 + (0.878-0.761) \^2 \] - -CI= 0.00032 to 0.367389 - -## Example Code using PROC FREQ - -Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. - -Instead it calculates the risk difference = (r / (r+s) - t / (t+u) ) - -Which in the example above is: 20/ (20+15) - 6 / (6+5) = 0.0296 - -This is more applicable when you have exposed / non-exposed groups looking at who has experienced the outcome. SAS Proc Freq has 3 methods for analysis of paired data using a Common risk difference. - -The default method is Mantel-Haenszel confidence limits. SAS can also Score (Miettinen-Nurminen) CIs and Stratified Newcombe CIs (constructed from stratified Wilson Score CIs). See [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for equations. - -As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. - -```{sas} -#| eval: false -data dat_used; - input ID$ PBO ACT @@; - datalines; - 001 0 0 002 0 0 003 0 0 004 0 0 005 0 0 - 006 1 0 007 1 0 008 1 0 009 1 0 010 1 0 - 011 1 0 012 0 1 013 0 1 014 0 1 015 0 1 - 016 0 1 017 0 1 018 0 1 019 0 1 020 0 1 - 021 0 1 022 0 1 023 0 1 024 0 1 025 0 1 - 026 0 1 027 1 1 028 1 1 029 1 1 030 1 1 - 031 1 1 032 1 1 033 1 1 034 1 1 035 1 1 - 036 1 1 037 1 1 038 1 1 039 1 1 040 1 1 - 041 1 1 042 1 1 043 1 1 044 1 1 045 1 1 - 046 1 1 - ; -run; - -proc freq data=dat_used; - table ACT*PBO/commonriskdiff(cl=MH); -run; -``` - -```{r} -#| echo: false -#| fig-align: center -#| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/riskdiff_matched.png") -``` - ## Reference 1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) From 79b7d2ed014a10935c4142afe9a6639d47cd0a92 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Wed, 18 Feb 2026 18:38:01 +0000 Subject: [PATCH 02/58] reflect updated SAS option name for CL=Newcombe instead of CL=Wilson (though the latter still works for now). Add details for SCAS method and GitHub repo for the macro, and citation for 'most commonly used method' --- SAS/ci_for_2indep_prop.qmd | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 1bee3d230..bfe39b147 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -49,11 +49,11 @@ run; knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ``` -## Methods for Calculating Confidence Intervals for 2 independent samples proportion difference +## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The most commonly used are: Wald with continuity correction and Wilson with continuity correction. The Wilson method may be more applicable when sample sizes are smaller and/or the proportion is closer to 0 or 1, since reliance on a normal distribution may be inappropriate. +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The performance of many of the same methods are compared graphically here\[add ref\]. According to a recent paper\[\], the most commonly reported method in non-inferiority clinical trials is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). -SAS is able to calculate CIs using the following methods: Exact, Agresti/Caffo, Wald, Wald with Continuity correction, Wilson/Newcombe Score, Wilson/Newcombe score with continuity correction, Miettinen and Nurminen, and Hauck-Anderson. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. +SAS PROC FREQ is able to calculate CIs using the following methods: Exact, Agresti/Caffo (AC), Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, Miettinen and Nurminen (MN), and Hauck-Anderson (HA). The SCAS method is not available in PROC FREQ, but a SAS macro is available in a GitHub repository\[\], which also provides an optional continuity correction for both SCAS and MN. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. ```{sas} #| eval: false @@ -92,9 +92,9 @@ and $CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ -### Wilson Method (Also known as the Score method or the Altman, Newcombe method^7^ ) +### Newcombe Method (Also known as the Hybrid Score method or the Method of Variance Estimates Recovery (MOVER) ) -Derive the confidence intervals using the Wilson Method equations above for each of the individual single samples 1 and 2. +Derive the confidence intervals for the separate proportions in each group, p1 and p2, using the Wilson Method equations \***above**. Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. @@ -102,7 +102,11 @@ Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. Let D = p1-p2 (the difference between the observed proportions) -The Confidence interval for the difference between two population proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ +The CI for the difference between two proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ + +### Miettinen-Nurminen Asymptotic Score Method + +\[Details to be added, noting options for continuity 'correction' and skewness correction offered by the macro\] ## Example Code using PROC FREQ From 99316f02ecc7a44edde2576bde37acff8d9aa88b Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Wed, 18 Feb 2026 18:38:38 +0000 Subject: [PATCH 03/58] initial minor edits, add reference to macro available on GitHub --- SAS/ci_for_paired_prop.qmd | 44 +++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index da74d4999..66fc3bb7a 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -49,7 +49,7 @@ run; knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ``` -## Methods for Calculating Confidence Intervals for a matched pair proportion difference +## Methods for Calculating Confidence Intervals for Proportion Difference from matched pairs You may experience paired data in any of the following types of situation: @@ -63,16 +63,16 @@ In all these cases, the calculated proportions for the 2 groups are not independ Using a cross over study as our example, a 2 x 2 table can be formed as follows: -+-----------------------+---------------+---------------+---------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+===============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+---------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+---------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+---------------+ ++-----------------------+---------------+---------------+-------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+=============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+-------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+-------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+-------------+ The proportions of subjects responding on each treatment are: @@ -82,16 +82,16 @@ Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ Suppose : -+-----------------------+----------------+----------------+------------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+================+================+==================+ -| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | -+-----------------------+----------------+----------------+------------------+ -| Active Response = No | t = 6 | u = 5 | t+u = 11 | -+-----------------------+----------------+----------------+------------------+ -| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | -+-----------------------+----------------+----------------+------------------+ ++-----------------------+---------------+---------------+------------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+==================+ +| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | ++-----------------------+---------------+---------------+------------------+ +| Active Response = No | t = 6 | u = 5 | t+u = 11 | ++-----------------------+---------------+---------------+------------------+ +| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | ++-----------------------+---------------+---------------+------------------+ ### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) @@ -169,7 +169,7 @@ CI= 0.00032 to 0.367389 ## Example Code using PROC FREQ -Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. +Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. A macro is available at \[\] which provides the asymptotic score method by Tango, and a skewness corrected version (paper under review). Instead it calculates the risk difference = (r / (r+s) - t / (t+u) ) From 84bf12c9018e9209b9577dc7d01b763d43b1735f Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 10:51:24 +0000 Subject: [PATCH 04/58] move introduction to separate page for general information on CIs for proportions, to avoid repetition --- SAS/ci_for_prop.qmd | 30 ++++++++++-------------------- SAS/ci_for_prop_intro.qmd | 31 +++++++++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 20 deletions(-) create mode 100644 SAS/ci_for_prop_intro.qmd diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 771e22764..48204738a 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -4,17 +4,7 @@ title: "Confidence intervals for Proportions in SAS" ## Introduction -The methods to use for calculating a confidence interval (CI) for a proportion depend on the type of situation you have. - -- 1 sample proportion (1 proportion calculated from 1 group of subjects) - -- 2 sample proportions and you want a CI for the difference in the 2 proportions (or the ratio, or an odds ratio) e.g. for estimating the magnitude of a treatment effect. - - - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) - - - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). - -Some sources suggest selecting a different method depending on whether your proportion is close to 0 or 1 (or near to the 0.5 midpoint), and your sample size. Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied. However, options are available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection. +\[See separate page for general introductory information on confidence intervals for proportions.\] ## Data used @@ -55,14 +45,6 @@ SAS PROC FREQ in Version 9.4 can compute 11 methods to calculate CIs for a singl For more information about some of these methods in R & SAS, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^2^ and [Confidence Intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf)^3^. Key literature on the subject includes papers by [Brown et al.](https://www.jstor.org/stable/2676784?seq=1)^4^ and [Newcombe](https://pubmed.ncbi.nlm.nih.gov/9595616/)^5^. -### Strictly conservative vs proximate coverage - -Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction") to achieve the former. - -### Consistency with hypothesis tests - -Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS for the asymptotic methods, such consistency is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. If an `EXACT BINOMIAL;` statement is used, the resulting test is consistent with the Clopper-Pearson CI. An exact hypothesis test with mid-P adjustment (consistent with the mid-P CI) is available via `EXACT BINOMIAL / MIDP;` but the output only gives the one-sided p-value, so if a 2-sided test is required it needs to be manually calculated by doubling the one-sided mid p-value. - ### Clopper-Pearson (Exact or binomial CI) method With binary endpoint data (response/non-response), we make the assumption that the proportion of responders has been derived from a series of Bernoulli trials. Trials (Subjects) are independent and we have a fixed number of repeated trials with an outcome of respond or not respond. This type of data follows the discrete binomial probability distribution, and the Clopper-Pearson^6^ (Exact) method uses this distribution to calculate the CIs. @@ -146,7 +128,7 @@ $$The coverage probabilities of the Jeffreys method are centred around the nomin ### Binomial based Mid-P method -The Mid-P method is similar to the Clopper-Pearson method, in the sense that it is based on exact calculations from the binomial probability distribution, but it aims to reduce the conservatism. It's quite a complex method to compute compared to the methods above and rarely used in practice *\[Ed: what source is there for this statement?\]*. However, like the Jeffreys interval, it has excellent 1-sided and 2-sided coverage properties for those seeking to align mean coverage with the nominal confidence level^9^. +The mid-P method is similar to the Clopper-Pearson method, in the sense that it is based on exact calculations from the binomial probability distribution, but it aims to reduce the conservatism. It's quite a complex method to compute compared to the methods above and rarely used in practice *\[Ed: what source is there for this statement?\]*. However, like the Jeffreys interval, it has excellent 1-sided and 2-sided coverage properties for those seeking to align mean coverage with the nominal confidence level^9^. The mid-P method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=MIDP);` @@ -158,6 +140,14 @@ The Clopper-Pearson CI's are always wider and contain the Blaker CI limits. It's The Blaker method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=BLAKER);` +## Consistency with hypothesis tests + +Within SAS for the asymptotic methods, consistency with the binomial test is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. + +If an `EXACT BINOMIAL;` statement is used, the resulting test is consistent with the Clopper-Pearson CI. + +An exact hypothesis test with mid-P adjustment (consistent with the mid-P CI) is available via `EXACT BINOMIAL / MIDP;` but the output only gives the one-sided p-value, so if a 2-sided test is required it needs to be manually calculated by doubling the one-sided mid p-value. + ## Example Code using PROC FREQ By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ', SAS outputs the Normal Approximation (Wald) and Clopper-Pearson (Exact) confidence intervals as two default methods, derived for the `Responders` = `Yes`. If you do not specify the `LEVEL` you want to model, then SAS assumes you want to model the first level that appears in the output (alphabetically). diff --git a/SAS/ci_for_prop_intro.qmd b/SAS/ci_for_prop_intro.qmd new file mode 100644 index 000000000..37b99fe9a --- /dev/null +++ b/SAS/ci_for_prop_intro.qmd @@ -0,0 +1,31 @@ +--- + title: "Confidence intervals for Proportions: General Information" +--- + +## Introduction + +The methods to use for calculating confidence intervals (CI) for binomial proportions depend on the type of situation you have. + +- 1 sample proportion, p (1 proportion calculated from 1 group of subjects) + +- 2 sample proportions (p1 and p2) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, RD = p1 - p2), the ratio (relative risk, RR = p1 / p2), or the odds ratio (OR = p1 (1-p2) / (p2 (1-p1))). + + - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) + + - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). + +The production of a confidence interval for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. + +Some sources suggest selecting a different method depending on whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint), and your sample size. Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied. However, options are available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection. + +The poor performance of the Wald methods is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. + +### Strictly conservative vs proximate coverage + +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. + +It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems to be more commonly preferred. + +### Consistency with hypothesis tests + +Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). It should be noted that not all CI methods satisfy this criterion. From 1610a9827ca8f43fd890058f4edb9de270e80e24 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 10:52:59 +0000 Subject: [PATCH 05/58] typos & clarification --- SAS/cmh.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index 4c5762e44..c178e9b2f 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -4,11 +4,11 @@ title: "CMH Test" # Cochran-Mantel-Haenszel Test -The CMH procedure tests for conditional independence in partial contingency tables for a 2 x 2 x K design. However, it can be generalized to tables of X x Y x K dimensions. This page also details derivation of risk differences and their confidence intervals which often accompnay a CMH test. +The CMH procedure tests for conditional independence in partial contingency tables for a 2 x 2 x K design. However, it can be generalized to tables of X x Y x K dimensions. This page also details derivation of risk differences and their confidence intervals which often accompany a CMH test. ## CMH in SAS -The cmh test is calculated in SAS using the PROC FREQ procedure. By default, it outputs the chi square statistic, degrees of freedom and p-value for each of the three alternative hypothesis: `general association`, `row means differ`, and `nonzero correlation`. It is up to the statistical analyst or statistician to know which result is appropriate for their analysis. +The CMH test is calculated in SAS using the PROC FREQ procedure. By default, it outputs the chi square statistic, degrees of freedom and p-value for each of the three alternative hypothesis: `general association`, `row means differ`, and `nonzero correlation`. It is up to the statistical analyst or statistician to know which result is appropriate for their analysis. When the design of the contingency table is 2 x 2 x K (i.e, X == 2 levels, Y == 2 levels, K \>= 2 levels), the Mantel-Haenszel Common Odds Ratio (odds ratio estimate, 95% CI, P-value) and the Breslow-Day Test for Homogeneity of the Odds Ratios (chi-square statistic, degrees of freedom, P-value) are also output. @@ -65,7 +65,7 @@ The above examples are a test for general association or if the row means scores ### Risk Differences Within each Strata -Risk differences within each strata can be obtained by adding the riskdiff option in SAS. The exact same output is obtained as per example 1 above, with the addition of 2 tables (1 for each age strata), showing the proportion of Female patients within each treatment (including 95% CIs), and the difference between the treatment proportions (including 95% CIs). By default, the CI's are calculated using Wald asymptotic confidence limits and in addition, the `exact` Clopper-Pearson confidence intervals for the risks. See SAS userguide [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) and [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) for more detail on the range of CI's SAS can calculate for the risk differences such as: Agresti-Caffo, exact unconditional, Hauck-Anderson, Miettinen-Nurminen (score), Newcombe (hybrid-score), Wald confidence limits, continuity-corrected Newcombe and continuity corrected Wald CIs. +Risk differences within each strata (for 2 x 2 x K tables) can be obtained by adding the riskdiff option in SAS. The exact same output is obtained as per example 1 above, with the addition of 2 tables (1 for each age strata), showing the proportion of Female patients within each treatment (including 95% CIs), and the difference between the treatment proportions (including 95% CIs). By default, the CI's are calculated using Wald asymptotic confidence limits and in addition, the `exact` Clopper-Pearson confidence intervals for the risks. See SAS userguide [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) and [here](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details53.htm) for more detail on the range of CI's SAS can calculate for the risk differences such as: Agresti-Caffo, exact unconditional, Hauck-Anderson, Miettinen-Nurminen (score), Newcombe (hybrid-score), Wald confidence limits, continuity-corrected Newcombe and continuity corrected Wald CIs. The individual treatment comparisons within strata can be useful to explore if the treatment effect is in the same direction and size for each strata, such as to determine if pooling them is in fact sensible. From 3fb8a767ed51febfda8c499039b0925fa8273a0f Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 10:53:59 +0000 Subject: [PATCH 06/58] add reference to new COMMONRISKDIFF option in SAS Viya --- SAS/cmh.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index c178e9b2f..da96e1a57 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -101,9 +101,11 @@ run; knitr::include_graphics("../images/cmh/saspage_output3b.png") ``` -### SAS Common error: SAS cannot do a stratified Miettinen-Nurminen (score) method! +### SAS Common error: SAS v9.4 cannot do a stratified Miettinen-Nurminen (score) method! -Note: you may think by running the following code, that you would be creating a common risk difference using the stratified Miettinen-Nurminen method. However, this is actually performing an unstratified Miettinen-Nurminen (Score) method. The output contains the same risk differences calculated for each strata separately and then a common risk difference (however this is NOT a stratified approach !!). See [SAS guide](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. +Note: you may think by running the following code, that you would be creating a common risk difference using the stratified Miettinen-Nurminen method. However, this is actually performing an unstratified Miettinen-Nurminen (Score) method. The output contains the same risk differences calculated for each strata separately and then a common risk difference using the 'summary score estimate' approach (however this is NOT a stratified approach !!). See [SAS guide](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. + +PROC FREQ in the SAS Viya platform has introduced a new COMMONRISKDIFF option for the TABLES statement that does use the stratified MN method. See the next section on Common risk differences available in SAS. @@ -140,7 +142,7 @@ Proc freq can calculate estimates of the common risk difference with 95% CIs, ca Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nurminen ('MN') score interval for unstratified datasets only. Using 'commonriskdiff' requests risks (binomial proportions) and risk differences for 2x2 tables. But doesn't extend to stratified analysis. -The only known way to have SAS produce stratified Miettinen-Nurminen CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](https://urldefense.com/v3/__https:/github.com/petelaud/ratesci-sas/tree/main__;!!GfteaDio!d_y6BtRjdLQgc_Wr-2-HGeyDSL1v71SvjvEQuVXNjzYaqLVDsEH0DdBLCBE3Q6LGFaTjE4LCjtECNYNFYfMSg4G49w$) +Stratified Miettinen-Nurminen CIs have recently been added to SAS PROC FREQ, but only within the SAS Viya platform. The only other known way to have SAS produce these CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](#0) ```{sas} #| eval: false From 3060c331ea4701a614f090ee89ab1fb8f1492fb1 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 10:55:06 +0000 Subject: [PATCH 07/58] add warning that PROC FREQ fails to produce an interval for 0/n1 vs 0/n2 --- SAS/cmh.qmd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index da96e1a57..e57152bdb 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -126,7 +126,9 @@ knitr::include_graphics("../images/cmh/saspage_output3c.png") ### SAS Common error: Make sure you output the risk difference for the correct level! -Including either column=1 or column=2 tells SAS which is your outcome of interest (ie, often that you want to compare treatment responders and not treatment non-responders!) My default it takes the 1st column sorted alphabetically but you can change this as shown below. +Including either column=1 or column=2 tells SAS which is your outcome of interest (ie, often that you want to compare treatment responders and not treatment non-responders!) By default it takes the 1st column sorted alphabetically but you can change this as shown below. + +Note that if the data are such that there are no responses in either treatment group, the cross-tabulation will only have 1 column, and SAS PROC FREQ will fail to produce a confidence interval. In this unusual situation however, a valid confidence interval can (and should) still be produced, which may be obtained using the %SCORECI macro available at [https://github.com/petelaud/ratesci-sas/tree/main](#0). ```{sas} #| eval: false From 797fc4b1bffc4604b79142e5a47907c654749abc Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 10:56:44 +0000 Subject: [PATCH 08/58] add text to categorical data subheadings to make connection with the corresponding CI pages --- data/stat_method_tbl.csv | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/data/stat_method_tbl.csv b/data/stat_method_tbl.csv index ca437445c..7c82fc6a5 100644 --- a/data/stat_method_tbl.csv +++ b/data/stat_method_tbl.csv @@ -18,11 +18,11 @@ Non-Parametric Analysis,Kruskal-Wallis test,[R](R/kruskal_wallis),[SAS](SAS/krus Non-Parametric Analysis,Friedman test,[R](R/friedman_test),[SAS](SAS/SAS_Friedmantest),,[R vs SAS](Comp/r-sas_friedman) Non-Parametric Analysis,Jonckheere-Terpstra test,[R](R/jonckheere),[SAS](SAS/jonchkheere_terpstra),,[R vs SAS](Comp/r-sas_jonckheere) Non-Parametric Analysis,Hodges-Lehman estimator,[R](R/nparestimate),[SAS](SAS/nparestimate),, -Categorical Data Analysis,Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), -Categorical Data Analysis,McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) +Categorical Data Analysis,Single proportion - Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), +Categorical Data Analysis,Two paired proportions - McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, -Categorical Data Analysis,Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) -Categorical Data Analysis,Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) +Categorical Data Analysis,Two independent proportions - Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) +Categorical Data Analysis,Stratified tables - Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) Categorical Data Analysis,CIs - Two Independent Proportions ,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) Categorical Data Analysis,CIs - Two Paired Proportions,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) From 507f8e8cfab7522c66fd56791d74047cec5b51c1 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 11:38:24 +0000 Subject: [PATCH 09/58] add further details about continuity adjustment and consistency with hypothesis tests --- SAS/ci_for_prop_intro.qmd | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/SAS/ci_for_prop_intro.qmd b/SAS/ci_for_prop_intro.qmd index 37b99fe9a..b2bd7f45e 100644 --- a/SAS/ci_for_prop_intro.qmd +++ b/SAS/ci_for_prop_intro.qmd @@ -22,10 +22,14 @@ The poor performance of the Wald methods is well documented - it can fail to ach ### Strictly conservative vs proximate coverage -Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" has been said to be something of a misnomer, and "continuity adjustment" is sometimes preferred \[see Campbell 2007\]. It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems to be more commonly preferred. ### Consistency with hypothesis tests -Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). It should be noted that not all CI methods satisfy this criterion. +Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a superiority test (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. + +## Reference + +1. \[References to be added\] From 34049738e4fa10750edb7d954c16bdc27e9eb20f Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 11:40:12 +0000 Subject: [PATCH 10/58] move intro page to method_summary folder, and add to index page --- data/stat_method_tbl.csv | 1 + {SAS => method_summary}/ci_for_prop_intro.qmd | 0 2 files changed, 1 insertion(+) rename {SAS => method_summary}/ci_for_prop_intro.qmd (100%) diff --git a/data/stat_method_tbl.csv b/data/stat_method_tbl.csv index 7c82fc6a5..2c1f0a8e0 100644 --- a/data/stat_method_tbl.csv +++ b/data/stat_method_tbl.csv @@ -23,6 +23,7 @@ Categorical Data Analysis,Two paired proportions - McNemar's test,[R](R/mcnemar) Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, Categorical Data Analysis,Two independent proportions - Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) Categorical Data Analysis,Stratified tables - Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) +Categorical Data Analysis,Intro to CIs for proportions,,,,[Summary](method_summary/ci_for_prop_intro) Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) Categorical Data Analysis,CIs - Two Independent Proportions ,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) Categorical Data Analysis,CIs - Two Paired Proportions,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) diff --git a/SAS/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd similarity index 100% rename from SAS/ci_for_prop_intro.qmd rename to method_summary/ci_for_prop_intro.qmd From 01ee908efd9a6bd0730b16e4cdeaab426f6c1b11 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 13:03:10 +0000 Subject: [PATCH 11/58] clarification about COMMONRISKDIFF option and new sub-options in SAS Viya --- SAS/cmh.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SAS/cmh.qmd b/SAS/cmh.qmd index e57152bdb..8d072b17a 100644 --- a/SAS/cmh.qmd +++ b/SAS/cmh.qmd @@ -105,7 +105,7 @@ knitr::include_graphics("../images/cmh/saspage_output3b.png") Note: you may think by running the following code, that you would be creating a common risk difference using the stratified Miettinen-Nurminen method. However, this is actually performing an unstratified Miettinen-Nurminen (Score) method. The output contains the same risk differences calculated for each strata separately and then a common risk difference using the 'summary score estimate' approach (however this is NOT a stratified approach !!). See [SAS guide](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. -PROC FREQ in the SAS Viya platform has introduced a new COMMONRISKDIFF option for the TABLES statement that does use the stratified MN method. +PROC FREQ in the SAS Viya platform has introduced a new COMMONRISKDIFF(CL=MN/MNMH) option for the TABLES statement that does use the stratified MN method. See the next section on Common risk differences available in SAS. @@ -142,9 +142,9 @@ run; Proc freq can calculate estimates of the common risk difference with 95% CIs, calculated using the Mantel-Haenszel and summary score (Miettinen-Nurminen) methods for multiway 2x2 tables. It can also provide stratified Newcombe confidence intervals using the method by Yan and Su (2010). The stratified Newcombe CIs are constructed from stratified Wilson CIs for the common (overall) row proportions. See [SAS help](https://support.sas.com/documentation/cdl/en/procstat/67528/HTML/default/viewer.htm#procstat_freq_details63.htm) for more detail. -Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nurminen ('MN') score interval for unstratified datasets only. Using 'commonriskdiff' requests risks (binomial proportions) and risk differences for 2x2 tables. But doesn't extend to stratified analysis. +Note that SAS (since v9.3M2 / STAT 12.1) PROC FREQ will produce the Miettinen-Nurminen ('MN') score interval for unstratified datasets only. The 'commonriskdiff' option added in SAS/STAT 14.3 requests risks (binomial proportions) and risk differences for 2x2 tables. But doesn't extend to stratified analysis. -Stratified Miettinen-Nurminen CIs have recently been added to SAS PROC FREQ, but only within the SAS Viya platform. The only other known way to have SAS produce these CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](#0) +Stratified Miettinen-Nurminen CIs have more recently been added to PROC FREQ, but only within the SAS Viya platform (using CL=MN or CL=MNMH in the COMMONRISKDIFF option). The only other known way to have SAS produce these CIs is to use this publicly available macro: [https://github.com/petelaud/ratesci-sas/tree/main](#0). Note that the macro (currently) does not implement Miettinen & Nurminen's proposed iterative weights, but instead the simpler (and similar) MH weights are used. ```{sas} #| eval: false From a9b365912f8f24d44dc3589e96c21327ed7bc1be Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 13:04:11 +0000 Subject: [PATCH 12/58] remove intro text. Initial outline sections added and first draft of text --- SAS/ci_for_2indep_prop.qmd | 48 +++++++++++++++++++++++++------------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index bfe39b147..6b0af0171 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -4,19 +4,7 @@ title: "Confidence Intervals for Independent Proportions in SAS" ## Introduction -The methods to use for calculating a confidence interval (CI) for independent proportions (p1 and p2) depend on the contrast parameter of interest. You might wish to quantify the simple difference between the proportions ('risk difference' RD = p1 - p2), or the ratio ( 'relative risk' RR = p1 / p2). Another option (for consistency with logistic regression output for example) is the Odds Ratio OR = (p1 (1-p2) / (p2 (1-p1)). - -The production of a confidence interval for such contrasts is useful for giving a clinically interpretable quantification of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. - -### Strictly conservative vs proximate coverage - -Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p1 and p2. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction"), or employ computationally intensive approaches to achieve the former. It is worth noting that although one might assume that regulatory bodies would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen method. - -### Consistency with hypothesis tests - -Depending on the analysis plan, it may be desirable (or in the case of a non-inferority trial, essential) for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS PROC FREQ for the asymptotic methods, such consistency is provided by the Miettinen-Nurminen CI, which is based on the same test statistic as the Farrington-Manning test\*\*\*. For the SCAS method, the macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. - -If an EXACT statement is used... +\[See separate page for general introductory information on confidence intervals for proportions.\] ## Data used @@ -53,7 +41,7 @@ knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The performance of many of the same methods are compared graphically here\[add ref\]. According to a recent paper\[\], the most commonly reported method in non-inferiority clinical trials is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). -SAS PROC FREQ is able to calculate CIs using the following methods: Exact, Agresti/Caffo (AC), Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, Miettinen and Nurminen (MN), and Hauck-Anderson (HA). The SCAS method is not available in PROC FREQ, but a SAS macro is available in a GitHub repository\[\], which also provides an optional continuity correction for both SCAS and MN. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. +SAS PROC FREQ is able to calculate CIs using the following methods: Exact, Agresti/Caffo (AC), Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, Miettinen and Nurminen (MN or SCORE), Mee (MN(correct=Mee)), and Hauck-Anderson (HA). The SCAS method is not available in PROC FREQ, but a SAS macro is available in a GitHub repository\[\], which also provides an optional continuity correction for both SCAS and MN. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. ```{sas} #| eval: false @@ -106,13 +94,41 @@ The CI for the difference between two proportions is: $$ D - sqrt((p_1-l_1)^2+(u ### Miettinen-Nurminen Asymptotic Score Method -\[Details to be added, noting options for continuity 'correction' and skewness correction offered by the macro\] +\[Details to be added, noting options for skewness correction offered by the %SCORECI macro\] + +## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples + +SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nurminen (SCORE), Exact, Likelihood Ratio (LR), Wald, Haldane Modified Wald. The SCAS method (which addresses asymmetric one-sided coverage of the MN method which is particularly pronounced for RR) is not available in PROC FREQ, but is in the SAS macro %SCORECI. + +## Methods for Calculating Confidence Intervals for Odds Ratio from 2 independent samples + +SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nurminen (SCORE), Exact, Likelihood Ratio (LR), mid-P (MIDP), Wald, Haldane Modified Wald. The SCAS method is not available in PROC FREQ, but is in the SAS macro %SCORECI. + +## + +### + +## Continuity Adjusted Methods + +SAS provides an option (CORRECT) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage. Note however that the Wald-cc method fails to achieve strictly conservative coverage \[See Laud & Dane 2014\]. + +It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. It removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method. No continuity correction is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described \[Laud & Dane 2014\] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. + +## Consistency with Hypothesis Tests + +Within SAS PROC FREQ for the asymptotic methods for RD, consistency with a traditional Chi-squared test for superiority (Karl Pearson version as produced by PROC FREQ), and the Farrington-Manning test for non-inferiority, is provided by the Mee CI (CL=SCORE(CORRECT=MEE)), which is similar to MN but omitting the N/(N-1) correction factor. Note that the MN method (including the correction factor) is consistent with the Egon Pearson 'N-1' version of the chi-squared test. SAS PROC FREQ does not produce that, nor does it offer the option to include the 'N-1' adjustment when requesting a non-inferiority test with the NONINF option. Consequently, there is a risk of contradictory results if using PROC FREQ to obtain a MN CI with a corresponding non-inferiority test. + +For example... + +For the SCAS method, the macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. + +If an EXACT statement is used... ## Example Code using PROC FREQ It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. -Options for riskdiff(CL=XXX) consist of AC:Agresti-Caffo, EXACT=exact, HA:Hauck-Anderson, MN or SCORE:Miettinen-Nurminen (another type of Score CI), WILSON or NEWCOMBE: Wilson method described above, and WALD: normal approximation wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. +Options for riskdiff(CL=XXX) consist of AC: Agresti-Caffo, EXACT: exact, HA: Hauck-Anderson, MN or SCORE: Miettinen-Nurminen (another type of Score CI), SCORE(CORRECT=MEE): Mee (Score CI omitting variance bias correction), WILSON or NEWCOMBE: Newcombe hybrid score method described above, and WALD: normal approximation Wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. ```{sas} #| eval: false From feffacce27fab3c95db640dce306f8f3ea87a859 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 16:48:50 +0000 Subject: [PATCH 13/58] split R page into separate sections for single/independent/paired proportions, with reference to new general information page --- R/ci_for_2indep_prop.qmd | 180 +++++++++++++++++++++++++++++++++++++++ R/ci_for_paired_prop.qmd | 145 +++++++++++++++++++++++++++++++ R/ci_for_prop.qmd | 161 +--------------------------------- 3 files changed, 327 insertions(+), 159 deletions(-) create mode 100644 R/ci_for_2indep_prop.qmd create mode 100644 R/ci_for_paired_prop.qmd diff --git a/R/ci_for_2indep_prop.qmd b/R/ci_for_2indep_prop.qmd new file mode 100644 index 000000000..b56224a5d --- /dev/null +++ b/R/ci_for_2indep_prop.qmd @@ -0,0 +1,180 @@ +--- +title: "Confidence Intervals for Independent Proportions in R" +--- + +## Introduction + +\[See separate page for general introductory information on confidence intervals for proportions.\] + +\[Note: information about cicalc package will be added to this page soon.\] + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{r} +#| echo: false +#| include: false +library(tidyverse) +library(cardx) +library(DescTools) +adcibc2 <- read_csv("../data/adcibc.csv") + +adcibc <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 0, 1) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +act <- filter(adcibc, trt == "ACT") |> + select(respn) +act2 <- act$respn +``` + +The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders). + +```{r} +#| echo: false +adcibc |> + group_by(trt, resp) |> + tally() +``` + +## Packages + +**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for. + +See [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations. + +If calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector). + +Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. + +**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. + +**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. + +**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc. + +1) x = 0 (0% responders), in which case the lower limit does not match. +2) x = n (100% responders), in which case the upper limit does not match. + +Because of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals. + +$$ B(\alpha/2;x, n-x+1) < p < B(1-\alpha/2; x+1, n-x)$$ + +RBesT equations are:\ +pLow \<- qbeta(Low, r + (r == 0), n - r + 1)\ +pHigh \<- qbeta(High, r + 1, n - r + ((n - r) == 0)) + +In Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\ +pLow \<- qbeta(Low, r, n - r + 1)\ +pHigh \<- qbeta(High, r + 1, n - r) + +**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs). + +**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail. + +**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios. + +## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples + +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions. + +### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx} + +For more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html). + +#### Example code + +`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied. + +Although this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method). + +Both the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables. + +The prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI. + +```{r} +indat1 <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 1, 0) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +# To get the comparison the correct way around Placebo must be 1, and Active 0 + +indat <- select(indat1, trtn, respn) + +cardx::ard_stats_prop_test( + data = indat, + by = trtn, + variables = respn, + conf.level = 0.95, + correct = FALSE +) +cardx::ard_stats_prop_test( + data = indat, + by = trtn, + variables = respn, + conf.level = 0.95, + correct = TRUE +) +``` + +### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools} + +For more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks. + +#### Example code + +With 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI. + +```{r} +count_dat <- indat |> + count(trtn, respn) +count_dat + +# BinomDiffCI requires +# x1 = successes in active, n1 = total subjects in active, +# x2 = successes in placebo, n2 = total subjects in placebo + +DescTools::BinomDiffCI( + x1 = 36, + n1 = 154, + x2 = 12, + n2 = 77, + conf.level = 0.95, + sides = c("two.sided"), + method = c( + "wald", + "waldcc", + "score", + "scorecc", + "ac", + "mn", + "mee", + "blj", + "ha", + "hal", + "jp" + ) +) +``` + +## References + +1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) +2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf) +3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. +4. diff --git a/R/ci_for_paired_prop.qmd b/R/ci_for_paired_prop.qmd new file mode 100644 index 000000000..a6c00abd7 --- /dev/null +++ b/R/ci_for_paired_prop.qmd @@ -0,0 +1,145 @@ +--- +title: "Confidence Intervals for Proportions in R" +--- + +## Introduction + +\[See separate page for general introductory information on confidence intervals for proportions.\] + +\[Note: information about cicalc package will be added to this page soon.\] + +## Data used + +The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `ACT` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. + +```{r} +#| echo: false +#| include: false +library(tidyverse) +library(cardx) +library(DescTools) +adcibc2 <- read_csv("../data/adcibc.csv") + +adcibc <- adcibc2 |> + select(AVAL, TRTP) |> + mutate( + resp = if_else(AVAL > 4, "Yes", "No"), + respn = if_else(AVAL > 4, 1, 0), + trt = if_else(TRTP == "Placebo", "PBO", "ACT"), + trtn = if_else(TRTP == "Placebo", 0, 1) + ) |> + select(trt, trtn, resp, respn) + +# cardx package required a vector with 0 and 1s for a single proportion CI +act <- filter(adcibc, trt == "ACT") |> + select(respn) +act2 <- act$respn +``` + +The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.234 (23.4% responders). + +```{r} +#| echo: false +adcibc |> + group_by(trt, resp) |> + tally() +``` + +## Packages + +**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for. + +See [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations. + +If calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector). + +Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. + +**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. + +**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. + +**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc. + +1) x = 0 (0% responders), in which case the lower limit does not match. +2) x = n (100% responders), in which case the upper limit does not match. + +Because of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals. + +$$ B(\alpha/2;x, n-x+1) < p < B(1-\alpha/2; x+1, n-x)$$ + +RBesT equations are:\ +pLow \<- qbeta(Low, r + (r == 0), n - r + 1)\ +pHigh \<- qbeta(High, r + 1, n - r + ((n - r) == 0)) + +In Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\ +pLow \<- qbeta(Low, r, n - r + 1)\ +pHigh \<- qbeta(High, r + 1, n - r) + +**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs). + +**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail. + +**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios. + +## Methods for Calculating Confidence Intervals for a matched pair proportion using {ExactCIdiff} + +For more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses. + +To date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above. + +**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\ +NOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf). + +Using a cross over study as our example, a 2 x 2 table can be formed as follows: + ++-----------------------+---------------+---------------+---------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+===============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+---------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+---------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+---------------+ + +: The proportions of subjects responding on each treatment are: + +Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ + +Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ + +Suppose : + ++-----------------------+---------------+---------------+------------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+==================+ +| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | ++-----------------------+---------------+---------------+------------------+ +| Active Response = No | t = 6 | u = 5 | t+u = 11 | ++-----------------------+---------------+---------------+------------------+ +| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | ++-----------------------+---------------+---------------+------------------+ + +Active: $\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\hat p_2= (r+t)/n$ = 26/46 =0.565 + +Difference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below + +-0.00339 to 0.38065 + +```{r} +#| eval: false +# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95) + +CI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI +CI +``` + +## References + +1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) +2. [PropCIs package](https://cran.r-project.org/web//packages/PropCIs/PropCIs.pdf) +3. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. +4. diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index e015f8a45..e3f250797 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -4,21 +4,9 @@ title: "Confidence Intervals for Proportions in R" ## Introduction -The methods to use for calculating a confidence interval (CI) for a proportion depend on the type of proportion you have. +\[See separate page for general introductory information on confidence intervals for proportions.\] -- 1 sample proportion (1 proportion calculated from 1 group of subjects) - -- 2 sample proportions and you want a CI for the difference in the 2 proportions. - - - If the 2 samples come from 2 independent samples (different subjects in each of the 2 groups) - - - If the 2 samples are matched (i.e. the same subject has 2 results, one on each group \[paired data\]). - -The method selected is also dependent on whether your proportion is close to 0 or 1 (or near to the 0.5 midpoint), and your sample size. - -For more information about these methods, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^1^. - -Note: information about cicalc package will be added to this page soon. +\[Note: information about cicalc package will be added to this page soon.\] ## Data used @@ -191,151 +179,6 @@ cardx::proportion_ci_jeffreys(act2, conf.level = 0.95) |> as_tibble() ``` -## Methods for Calculating Confidence Intervals for a matched pair proportion using {ExactCIdiff} - -For more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses. - -To date we have not found an R package which calculates a CI for matched pair proportions using the normal approximation or Wilson methods although they can be done by hand using the equations provided on the SAS page link above. - -**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs), claiming to be the first package in R to do this method. However, it should only be used when the sample size is not too large as it can be computationally intensive.\ -NOTE that the {ExactNumCI} package should not be used for this task. More detail on these two packages can be found [here](RJ-2013-026.pdf). - -Using a cross over study as our example, a 2 x 2 table can be formed as follows: - -+-----------------------+---------------+---------------+---------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+===============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+---------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+---------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+---------------+ - -: The proportions of subjects responding on each treatment are: - -Active: $\hat p_1 = (r+s)/n$ and Placebo: $\hat p_2= (r+t)/n$ - -Difference between the proportions for each treatment are: $D=p1-p2=(s-t)/n$ - -Suppose : - -+-----------------------+---------------+---------------+------------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+==================+ -| Active Response = Yes | r = 20 | s = 15 | r+s = 35 | -+-----------------------+---------------+---------------+------------------+ -| Active Response = No | t = 6 | u = 5 | t+u = 11 | -+-----------------------+---------------+---------------+------------------+ -| Total | r+t = 26 | s+u = 20 | N = r+s+t+u = 46 | -+-----------------------+---------------+---------------+------------------+ - -Active: $\hat p_1 = (r+s)/n$ =35/46 =0.761 and Placebo: $\hat p_2= (r+t)/n$ = 26/46 =0.565 - -Difference = 0.761-0.565 = 0.196, then PairedCI() function can provide an exact confidence interval as shown below - --0.00339 to 0.38065 - -```{r} -#| eval: false -# ExactCIdiff::PairedCI(s, r+u, t, conf.level = 0.95) - -CI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI -CI -``` - -## Methods for Calculating Confidence Intervals for 2 independent samples proportion - -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^4^ describes many methods for the calculation of confidence intervals for 2 independent proportions. - -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) using {cardx} - -For more technical information regarding the Wald method see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html). - -#### Example code - -`cardx::ard_stats_prop_test function` uses `stats::prop.test` which also allows a continuity correction to be applied. - -Although this website [here](https://rdrr.io/r/stats/prop.test.html) and this one [here](https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/prop.test) both reference Newcombe for the CI that this function uses, replication of the results by hand and compared to SAS show that the results below match the Normal Approximation (Wald method). - -Both the Treatment variable (ACT,PBO) and the Response variable (Yes,No) have to be numeric (0,1) or Logit (TRUE,FALSE) variables. - -The prop.test default with 2 groups, is the null hypothesis that the proportions in each group are the same and a 2-sided CI. - -```{r} -indat1 <- adcibc2 |> - select(AVAL, TRTP) |> - mutate( - resp = if_else(AVAL > 4, "Yes", "No"), - respn = if_else(AVAL > 4, 1, 0), - trt = if_else(TRTP == "Placebo", "PBO", "ACT"), - trtn = if_else(TRTP == "Placebo", 1, 0) - ) |> - select(trt, trtn, resp, respn) - -# cardx package required a vector with 0 and 1s for a single proportion CI -# To get the comparison the correct way around Placebo must be 1, and Active 0 - -indat <- select(indat1, trtn, respn) - -cardx::ard_stats_prop_test( - data = indat, - by = trtn, - variables = respn, - conf.level = 0.95, - correct = FALSE -) -cardx::ard_stats_prop_test( - data = indat, - by = trtn, - variables = respn, - conf.level = 0.95, - correct = TRUE -) -``` - -### Normal Approximation (Wald) and Other Methods for 2 independent samples using {DescTools} - -For more technical information regarding the derivations of these methods see the corresponding [SAS page](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html) or {DescTools} package documentation [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html). **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive (less commonly used) methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson, Haldane and Jeffreys-Perks. - -#### Example code - -With 2 groups, the null hypothesis that the proportions in each group are the same and a 2-sided CI. - -```{r} -count_dat <- indat |> - count(trtn, respn) -count_dat - -# BinomDiffCI requires -# x1 = successes in active, n1 = total subjects in active, -# x2 = successes in placebo, n2 = total subjects in placebo - -DescTools::BinomDiffCI( - x1 = 36, - n1 = 154, - x2 = 12, - n2 = 77, - conf.level = 0.95, - sides = c("two.sided"), - method = c( - "wald", - "waldcc", - "score", - "scorecc", - "ac", - "mn", - "mee", - "blj", - "ha", - "hal", - "jp" - ) -) -``` - ## References 1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) From 5c4a92f27858a52334a97e963f707725aee2c53c Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 17:13:32 +0000 Subject: [PATCH 14/58] remove packages not relevant to the section, & add placeholder for adding {ratesci} --- R/ci_for_2indep_prop.qmd | 25 +++---------------------- R/ci_for_paired_prop.qmd | 35 ++--------------------------------- R/ci_for_prop.qmd | 4 +++- 3 files changed, 8 insertions(+), 56 deletions(-) diff --git a/R/ci_for_2indep_prop.qmd b/R/ci_for_2indep_prop.qmd index b56224a5d..4ba279f89 100644 --- a/R/ci_for_2indep_prop.qmd +++ b/R/ci_for_2indep_prop.qmd @@ -53,30 +53,11 @@ See [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Interv If calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector). -Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. +Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. -**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. +Methods included are \[TBC\] methods for 2 independent samples. -**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. - -**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc. - -1) x = 0 (0% responders), in which case the lower limit does not match. -2) x = n (100% responders), in which case the upper limit does not match. - -Because of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals. - -$$ B(\alpha/2;x, n-x+1) < p < B(1-\alpha/2; x+1, n-x)$$ - -RBesT equations are:\ -pLow \<- qbeta(Low, r + (r == 0), n - r + 1)\ -pHigh \<- qbeta(High, r + 1, n - r + ((n - r) == 0)) - -In Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\ -pLow \<- qbeta(Low, r, n - r + 1)\ -pHigh \<- qbeta(High, r + 1, n - r) - -**The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs). +**The {ratesci} package** is ... \[TBC\] **The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail. diff --git a/R/ci_for_paired_prop.qmd b/R/ci_for_paired_prop.qmd index a6c00abd7..5e1c1d135 100644 --- a/R/ci_for_paired_prop.qmd +++ b/R/ci_for_paired_prop.qmd @@ -47,42 +47,11 @@ adcibc |> ## Packages -**The {cardx} package** is an extension of the {cards} package, providing additional functions to create Analysis Results Data Objects (ARDs)^1^. It was developed as part of {NEST} and pharmaverse. This package requires the binary endpoint to be a logical (TRUE/FALSE) vector or a numeric/integer coded as (0, 1) with 1 (TRUE) being the success you want to calculate the confidence interval for. - -See [here](R:%20Functions%20for%20Calculating%20Proportion%20Confidence%20Intervals) for full description of the {cardx} proportions equations. - -If calculating the CI for a difference in proportions, the package requires both the response and the treatment variable to be numeric/integer coded as (0, 1) (or logical vector). - -Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. - -**The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. - -**The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. - -**The {RBesT} package (Prior to Version 1.8-0)** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. However, as described below, there are 2 cases where the results using RBesT package do not match cardx or Hmisc. - -1) x = 0 (0% responders), in which case the lower limit does not match. -2) x = n (100% responders), in which case the upper limit does not match. - -Because of the relationship between the binomial distribution and the beta distribution. This package uses quantiles of the beta distribution to derive exact confidence intervals. - -$$ B(\alpha/2;x, n-x+1) < p < B(1-\alpha/2; x+1, n-x)$$ - -RBesT equations are:\ -pLow \<- qbeta(Low, r + (r == 0), n - r + 1)\ -pHigh \<- qbeta(High, r + 1, n - r + ((n - r) == 0)) - -In Version 1.8-0 onwards the equations were updated as follows, which then match the Hmisc intervals:\ -pLow \<- qbeta(Low, r, n - r + 1)\ -pHigh \<- qbeta(High, r + 1, n - r) +**The {ratesci} package** is ... \[TBC\] **The {ExactCIdiff} package** produces exact CIs for two dependent proportions (matched pairs). -**The {DescTools} package** has a function BinomDiffCI which produces CIs for two independent proportions (unmatched pairs) including methods for Agresti/Caffo, Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, and more computationally intensive methods such as Miettinen and Nurminen, Mee, Brown Li's Jeffreys, Hauck-Anderson and Haldane. See [here](https://search.r-project.org/CRAN/refmans/DescTools/html/BinomDiffCI.html) for more detail. - -**The {presize} package** has a function prec_prop() which also calculates CIs for 2 independent samples using the Wilson, Agresti-Coull, Exact or Wald approaches. The package is not described in further detail here since in most cases **{DescTools}** will be able to compute what is needed. However, it's mentioned due to other functionality it has available such as sample size and precision calculations for AUC, correlations, cronbach's alpha, intraclass correlation, Cohen's kappa, likelihood ratios, means, mean differences, odds ratios, rates, rate ratios, risk differences and risk ratios. - -## Methods for Calculating Confidence Intervals for a matched pair proportion using {ExactCIdiff} +## Methods for Calculating Confidence Intervals for Proportion Difference from matched pairs using {ExactCIdiff} and {ratesci} For more information about the detailed methods for calculating confidence intervals for a matched pair proportion see [here](https://psiaims.github.io/CAMIS/SAS/ci_for_prop.html#methods-for-calculating-confidence-intervals-for-a-matched-pair-proportion). When you have 2 measurements on the same subject, the 2 sets of measures are not independent and you have matched pair of responses. diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index e3f250797..0f10de26b 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -1,5 +1,5 @@ --- -title: "Confidence Intervals for Proportions in R" +title: "Confidence Intervals for a Proportion in R" --- ## Introduction @@ -55,6 +55,8 @@ If calculating the CI for a difference in proportions, the package requires both Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. +**The {ratesci} package** is ... \[TBC\] + **The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. **The {Hmisc} package** produces CIs using the Clopper-Pearson method. In this example (x=36 and n=154), the results match the cardx package. Documentation reports that the method uses F distribution to compute exact intervals based on the binomial cdf. However, if the percentage of responders is 100% then the upper limit is set to 1. Similarly if the percentage of responders is 0%, then the lower limit is set to 0. Hence, in extreme cases there may be differences between this package and the standard implementation of Clopper-Pearson method. From c12abf16d019e6b3e5a092e040dec6592e689f63 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 17:14:38 +0000 Subject: [PATCH 15/58] replace intro text with reference to new summary section --- SAS/ci_for_paired_prop.qmd | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index 66fc3bb7a..4c593f11e 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -4,19 +4,7 @@ title: "Confidence intervals for Paired Proportions in SAS" ## Introduction -The methods to use for calculating a confidence interval (CI) for independent proportions (p1 and p2) depend on the contrast parameter of interest. You might wish to quantify the simple difference between the proportions ('risk difference' RD = p1 - p2), or the ratio ( 'relative risk' RR = p1 / p2). Another option (slightly more obscure) is the Conditional Odds Ratio OR, but note that this .... - -The production of a confidence interval for such contrasts is useful for giving a clinically interpretable quantification of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. - -### Strictly conservative vs proximate coverage - -Because of the discrete nature of binomial data, it is impossible for any CI to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some include an optional adjustment ("continuity correction") to achieve the former. - -### Consistency with hypothesis tests - -Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). Within SAS for the asymptotic methods, such consistency is provided by the ... - -### +\[See separate page for general introductory information on confidence intervals for proportions.\] ## Data used @@ -63,16 +51,16 @@ In all these cases, the calculated proportions for the 2 groups are not independ Using a cross over study as our example, a 2 x 2 table can be formed as follows: -+-----------------------+---------------+---------------+-------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+=============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+-------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+-------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+-------------+ ++-----------------------+---------------+---------------+--------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+==============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+--------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+--------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+--------------+ The proportions of subjects responding on each treatment are: From 9b08c89dc8b4f1818ea5b1fda6714196c8337e43 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 17:15:37 +0000 Subject: [PATCH 16/58] add placeholders for new sections - content to be added --- R/ci_for_2indep_prop.qmd | 18 ++++++++++++++++++ R/ci_for_paired_prop.qmd | 22 +++++++++++++++++++++- R/ci_for_prop.qmd | 8 ++++++++ SAS/ci_for_2indep_prop.qmd | 2 -- SAS/ci_for_paired_prop.qmd | 8 ++++++++ SAS/ci_for_prop.qmd | 6 +++++- 6 files changed, 60 insertions(+), 4 deletions(-) diff --git a/R/ci_for_2indep_prop.qmd b/R/ci_for_2indep_prop.qmd index 4ba279f89..8825f2570 100644 --- a/R/ci_for_2indep_prop.qmd +++ b/R/ci_for_2indep_prop.qmd @@ -153,6 +153,24 @@ DescTools::BinomDiffCI( ) ``` +## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples + +\[TBC\] + +### + +## Methods for Calculating Confidence Intervals for Odds Ratio from 2 independent samples + +\[TBC\] + +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with hypothesis tests + +\[TBC\] - cf. chi-squared tests + ## References 1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) diff --git a/R/ci_for_paired_prop.qmd b/R/ci_for_paired_prop.qmd index 5e1c1d135..6b409645f 100644 --- a/R/ci_for_paired_prop.qmd +++ b/R/ci_for_paired_prop.qmd @@ -1,5 +1,5 @@ --- -title: "Confidence Intervals for Proportions in R" +title: "Confidence Intervals for Paired Proportions in R" --- ## Introduction @@ -106,6 +106,26 @@ CI <- ExactCIdiff::PairedCI(15, 25, 6, conf.level = 0.95)$ExactCI CI ``` +## Methods for Calculating Confidence Intervals for Relative Risk from matched pairs using {ratesci} + +\[TBC\] + +## Methods for Calculating Confidence Intervals for Conditional Odds Ratio from matched pairs using {ratesci} + +\[TBC\] + +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with Hypothesis Tests + +\[TBC\] - cf. McNemar test + +## + +## + ## References 1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index 0f10de26b..7b54c0a5f 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -181,6 +181,14 @@ cardx::proportion_ci_jeffreys(act2, conf.level = 0.95) |> as_tibble() ``` +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with hypothesis tests + +\[TBC\] + ## References 1. [pharmaverse cardx package](https://insightsengineering.github.io/cardx/main/#:~:text=The%20%7Bcardx%7D%20package%20is%20an%20extension%20of%20the,Data%20Objects%20%28ARDs%29%20using%20the%20R%20programming%20language.) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 6b0af0171..38f5f6a3b 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -104,8 +104,6 @@ SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nu SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nurminen (SCORE), Exact, Likelihood Ratio (LR), mid-P (MIDP), Wald, Haldane Modified Wald. The SCAS method is not available in PROC FREQ, but is in the SAS macro %SCORECI. -## - ### ## Continuity Adjusted Methods diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index 4c593f11e..5c3762b25 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -155,6 +155,14 @@ Upper CI = $D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 )$ = 0.196 + sqrt \[ (0.565-0.440)\ CI= 0.00032 to 0.367389 +## Continuity Adjusted Methods + +\[TBC\] + +## Consistency with Hypothesis Tests + +\[TBC\] + ## Example Code using PROC FREQ Unfortunately, SAS does not have a procedure which outputs the confidence intervals for matched proportions. A macro is available at \[\] which provides the asymptotic score method by Tango, and a skewness corrected version (paper under review). diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 48204738a..e4257425b 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -1,5 +1,5 @@ --- -title: "Confidence intervals for Proportions in SAS" +title: "Confidence intervals for a Proportion in SAS" --- ## Introduction @@ -140,6 +140,10 @@ The Clopper-Pearson CI's are always wider and contain the Blaker CI limits. It's The Blaker method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=BLAKER);` +## Continuity Adjusted Methods + +\[TBC\] + ## Consistency with hypothesis tests Within SAS for the asymptotic methods, consistency with the binomial test is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. From 8e3a83f96747060d7d9fcbb47d9f3d6aec281dd3 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Thu, 19 Feb 2026 17:28:33 +0000 Subject: [PATCH 17/58] add note about development version of ratesci::rateci() --- R/ci_for_prop.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index 7b54c0a5f..632edf152 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -55,7 +55,7 @@ If calculating the CI for a difference in proportions, the package requires both Instead of the code presented below, you can use `ard_categorical_ci(data, variables=resp, method ='wilson')` for example. This invokes the code below but returns an analysis results dataset (ARD) format as the output. Methods included are waldcc, wald, clopper-pearson, wilson, wilsoncc, strat_wilson, strat_wilsoncc, agresti-coull and jeffreys for one-sample proportions and methods for 2 independent samples, however currently does not have a method for 2 matched proportions. -**The {ratesci} package** is ... \[TBC\] +**The {ratesci} package** is ... \[TBC\] - note, current development version at has new features in the rateci() function (including more of the CI methods described below) compared to the CRAN release. **The {PropCIs} package** produces CIs for methods such as Blaker's exact method and Midp which aren't available in {cardx} but are available in SAS. We found results agreed with SAS to the 5th decimal place. The package also calculates CIs for Clopper-Pearson, Wald, Wilson, Agresti-Coull and these align to results obtained in cardx to at least the 7th decimal place. The {PropsCIs} package requires just the number of events (numerator number of successes) & total number of subjects (denominator) as an input dataset. Given Blaker and Midp are rarely used in practice, and {PropsCIs} isn't a package commonly downloaded from CRAN, further details are not provided here. From af55a7bd7fd102dbe2e93e5a56c185f37ce69063 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Sun, 22 Feb 2026 15:26:09 +0000 Subject: [PATCH 18/58] clarification for MN/Mee --- SAS/ci_for_2indep_prop.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 38f5f6a3b..008bfdcab 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -110,7 +110,7 @@ SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nu SAS provides an option (CORRECT) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage. Note however that the Wald-cc method fails to achieve strictly conservative coverage \[See Laud & Dane 2014\]. -It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. It removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method. No continuity correction is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described \[Laud & Dane 2014\] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. +It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, CORRECT=NO removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and an equivalent un-corrected score method for RR and OR). No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described \[Laud & Dane 2014\] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. ## Consistency with Hypothesis Tests From 1a9a5e660a8a4732bd3831177d32f9eb3496125c Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Sun, 22 Feb 2026 15:26:42 +0000 Subject: [PATCH 19/58] remove 'asymptotic', which is non-specific --- R/ci_for_prop.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/R/ci_for_prop.qmd b/R/ci_for_prop.qmd index 632edf152..8822671eb 100644 --- a/R/ci_for_prop.qmd +++ b/R/ci_for_prop.qmd @@ -101,7 +101,7 @@ cardx::proportion_ci_clopper_pearson(act2, conf.level = 0.95) |> as_tibble() ``` -### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) +### Normal Approximation Method (Also known as the Wald Method) In large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\alpha)\%$ confidence interval for population proportion can be calculated using normal approximation as follows: From fd3bd471c3bae365477d6bfe499a468ee50928ce Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Sun, 22 Feb 2026 15:28:33 +0000 Subject: [PATCH 20/58] add section describing methods that apply across different contrast parameters --- method_summary/ci_for_prop_intro.qmd | 30 ++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index b2bd7f45e..2a0cec8ec 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -30,6 +30,36 @@ It is worth noting that although one might assume that regulatory authorities wo Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a superiority test (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. +# Overarching methods + +There are some methods that apply (or have variants which apply) across all contrast parameters (i.e. $\theta=p$ or $\theta=RD$ etc.), which are outlined below. + +### Normal Approximation Method (Also known as the Wald Method) + +The Wald confidence interval is constructed simply from the the point estimate plus and minus a multiple of its estimated standard error: $(L_{Wald}, U_{Wald}) = \hat \theta \pm z_{\alpha/2} \sqrt{\hat V(\hat\theta)}$ where $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$. + +For RR and OR, $\theta$ is log-transformed to improve the normal approximation, and $(L_{Wald}, U_{Wald}) = exp[ ln(\hat \theta) \pm z_{\alpha/2} \sqrt{\hat V(ln(\hat\theta))}]$, which requires a modification for boundary cases. + +### Asymptotic Score Method + +The Score confidence interval is also based on an asymptotic normal approximation, but uses a score statistic $Z(\theta) = S(\theta)/\sqrt {\tilde V}$ based on contrast function $S(\theta)$ and its true variance $\tilde V = V(S(\theta))$. For example, for the single proportion, $S(p) = \hat p - p$, or for RD, $S(\theta) = \hat p_1 - \hat p_2 - \theta$. + +$Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion, this process simplifies to solving a quadratic equation in $p$. + +For the contrast parameters, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ for the given value of $\theta$. + +### Skewness Corrected Asymptotic Score Method + +... + +### 'Exact' Method(s) + +So-called 'exact' methods are designed to guarantee strictly conservative coverage. (Newcombe points out some problems with the use of the term 'exact' here, hence the quotation marks.) For the single proportion, the Clopper-Pearson confidence interval is the range of p for which $P(X \ge x | p) = \alpha / 2$ and $P(X \le x | p) = \alpha / 2$. More complex calculations are involved for the contrast parameters, which eliminate the nuisance parameter by taking the supremum of all p-values across the range of combinations of $p_1$ and $p_2$ for a given $\theta$. + +There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). + +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ could be used for almost-but-not-quite-strictly conservative coverage. + ## Reference 1. \[References to be added\] From 43c1d720bc71a547bf6de8626d6fb1c51d991b20 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Sun, 22 Feb 2026 15:30:54 +0000 Subject: [PATCH 21/58] add detail about historic advice for small cell counts, & some other edits --- method_summary/ci_for_prop_intro.qmd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 2a0cec8ec..d58161944 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -4,7 +4,7 @@ ## Introduction -The methods to use for calculating confidence intervals (CI) for binomial proportions depend on the type of situation you have. +The methods to use for calculating confidence intervals (CI) for binomial proportions depend on the situation: - 1 sample proportion, p (1 proportion calculated from 1 group of subjects) @@ -16,15 +16,15 @@ The methods to use for calculating confidence intervals (CI) for binomial propor The production of a confidence interval for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. -Some sources suggest selecting a different method depending on whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint), and your sample size. Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied. However, options are available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection. +Some sources suggest selecting a different method depending on your sample size and whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint). \[Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied (presumably for ease of computation).\] Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions may well have been useful at a time when the Wald method was the only available alternative. However, options are now available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection or reliance on overly conservative methods. -The poor performance of the Wald methods is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. +The poor performance of the Wald methods is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] ### Strictly conservative vs proximate coverage -Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" has been said to be something of a misnomer, and "continuity adjustment" is sometimes preferred \[see Campbell 2007\]. +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". \[Brown et al. point out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions.\] Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" has been said to be something of a misnomer, and "continuity adjustment" is sometimes preferred \[see Campbell 2007, Newcombe 1998a\]. -It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems to be more commonly preferred. +It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method *\[Ref\]*. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. ### Consistency with hypothesis tests From 532dfa67d27b9e8971b3a5d5b619571802abe8ec Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Mon, 23 Feb 2026 15:52:51 +0000 Subject: [PATCH 22/58] initial creation of references file for easier management of citations. Needs to be tested on CAMIS site --- method_summary/ci_for_prop_intro.qmd | 3 ++- method_summary/references.bib | 13 +++++++++++++ 2 files changed, 15 insertions(+), 1 deletion(-) create mode 100644 method_summary/references.bib diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index d58161944..e8207b192 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -1,5 +1,6 @@ --- title: "Confidence intervals for Proportions: General Information" + bibliography: references.bib --- ## Introduction @@ -18,7 +19,7 @@ The production of a confidence interval for such contrasts is useful for giving Some sources suggest selecting a different method depending on your sample size and whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint). \[Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied (presumably for ease of computation).\] Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions may well have been useful at a time when the Wald method was the only available alternative. However, options are now available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection or reliance on overly conservative methods. -The poor performance of the Wald methods is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] +The poor performance of the Wald method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided[@brown2001, p128]. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] ### Strictly conservative vs proximate coverage diff --git a/method_summary/references.bib b/method_summary/references.bib new file mode 100644 index 000000000..05972a594 --- /dev/null +++ b/method_summary/references.bib @@ -0,0 +1,13 @@ + +@article{brown2001, + title = {Interval Estimation for a Binomial Proportion}, + author = {Brown, Lawrence D. and Cai, T. Tony and DasGupta, Anirban}, + year = {2001}, + month = {05}, + date = {2001-05-01}, + journal = {Statistical Science}, + volume = {16}, + number = {2}, + doi = {10.1214/ss/1009213286}, + url = {http://dx.doi.org/10.1214/ss/1009213286} +} From 181b59bea06aa3e1c4f3bd5df1ed5b9322d3a153 Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Mon, 23 Feb 2026 15:54:45 +0000 Subject: [PATCH 23/58] tidy up text about small cell counts --- method_summary/ci_for_prop_intro.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index e8207b192..89307e1f1 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -17,7 +17,7 @@ The methods to use for calculating confidence intervals (CI) for binomial propor The production of a confidence interval for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. -Some sources suggest selecting a different method depending on your sample size and whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint). \[Seemingly this advice stems from a preference to use the Wald method when the normal approximation assumptions are adequately satisfied (presumably for ease of computation).\] Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions may well have been useful at a time when the Wald method was the only available alternative. However, options are now available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection or reliance on overly conservative methods. +Some sources suggest selecting a different method depending on your sample size and whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint). Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative. However, options are now available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection or reliance on overly conservative methods. The poor performance of the Wald method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided[@brown2001, p128]. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] From 4ae7bba25cbc7ad8123ac7a9c2be4f287eaebbad Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Mon, 23 Feb 2026 15:55:29 +0000 Subject: [PATCH 24/58] add comment about one-sided coverage and central location --- method_summary/ci_for_prop_intro.qmd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 89307e1f1..875888b6d 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -27,10 +27,12 @@ Because of the discrete nature of binomial data, it is impossible for any CI met It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method *\[Ref\]*. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. -### Consistency with hypothesis tests +### Consistency with hypothesis tests, and one-sided coverage Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a superiority test (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. +A related issue is whether it is sufficient for two-sided coverage probability to be aligned with the nominal $(1-\alpha)$, or whether the tail probabilities (i.e. one-sided non-coverage at each end) should also be aligned with the nominal $\alpha/2$. Clearly the latter criterion (described as 'central interval location' by Newcombe) is essential for non-inferiority testing, but it is also a desirable property for any confidence interval. + # Overarching methods There are some methods that apply (or have variants which apply) across all contrast parameters (i.e. $\theta=p$ or $\theta=RD$ etc.), which are outlined below. From 12ba11bd1962bcfe8f43d1a7f85f21328d197bbc Mon Sep 17 00:00:00 2001 From: PeteLaud Date: Mon, 23 Feb 2026 15:56:57 +0000 Subject: [PATCH 25/58] extend details on asymptotic score, SCAS, MOVER & mid-P methods --- method_summary/ci_for_prop_intro.qmd | 32 ++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 875888b6d..f84e96e43 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -37,23 +37,37 @@ A related issue is whether it is sufficient for two-sided coverage probability t There are some methods that apply (or have variants which apply) across all contrast parameters (i.e. $\theta=p$ or $\theta=RD$ etc.), which are outlined below. -### Normal Approximation Method (Also known as the Wald Method) +### Normal Approximation Methods (Also known as the Wald Method) The Wald confidence interval is constructed simply from the the point estimate plus and minus a multiple of its estimated standard error: $(L_{Wald}, U_{Wald}) = \hat \theta \pm z_{\alpha/2} \sqrt{\hat V(\hat\theta)}$ where $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$. For RR and OR, $\theta$ is log-transformed to improve the normal approximation, and $(L_{Wald}, U_{Wald}) = exp[ ln(\hat \theta) \pm z_{\alpha/2} \sqrt{\hat V(ln(\hat\theta))}]$, which requires a modification for boundary cases. -### Asymptotic Score Method +### Asymptotic Score Methods The Score confidence interval is also based on an asymptotic normal approximation, but uses a score statistic $Z(\theta) = S(\theta)/\sqrt {\tilde V}$ based on contrast function $S(\theta)$ and its true variance $\tilde V = V(S(\theta))$. For example, for the single proportion, $S(p) = \hat p - p$, or for RD, $S(\theta) = \hat p_1 - \hat p_2 - \theta$. -$Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion, this process simplifies to solving a quadratic equation in $p$. +$Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion (Wilson score interval), this process simplifies to solving a quadratic equation in $p$. -For the contrast parameters, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ for the given value of $\theta$. +For the contrast parameters, score methods were derived for independent proportions by Miettinen & Nurminen, Mee, and Koopman in the 1980s, and for paired data by Tango (1998), Nam & Blackwelder (2002) and Tang et al. (2003). In each case, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ (or for the paired case, the cell probability $p_{21}$) for the given value of $\theta$. -### Skewness Corrected Asymptotic Score Method +### SCAS Methods -... +Originating from papers published in 1985-1990 by Gart and Nam, the skewness-corrected asymptotic score ('SCAS') methods introduce a further term within the score statistic that involves $\tilde \mu$, the third central moment of $S(\theta)$ + +$$ +Z(\theta) = \frac{S(\theta)}{\sqrt {\tilde V}} - \frac{(Z(\theta)^2 - 1)\tilde \mu_3}{6 \tilde V^{3/2}} +$$ + +Hence these are an extension of the asymptotic score intervals, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is also applied to the Wilson score method, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud (2017), with a further publication under review for paired data. + +\[SCAS intervals are not yet implemented in SAS PROC freq, but can be obtained using the %SCORECI and %PAIRBINCI macros.\] + +### MOVER Method + +This class of methods was popularised by Newcombe for the RD contrast, and is therefore labelled as the Newcombe method in current SAS PROC FREQ documentation. It has since been generalised to other contrasts and for paired data (though not implemented in SAS), and renamed as the Method of Variance Estimates Recovery (MOVER). It is also referred to as the 'Square-and-add' approach. + +MOVER intervals are produced by combining CIs for each of the two separate proportions involved (and an estimate of their correlation, in the paired data case). Originally Wilson intervals were used (hence earlier versions of SAS labelled the method as WILSON), but improved performance may be obtained by using Jeffreys intervals instead. ### 'Exact' Method(s) @@ -63,6 +77,12 @@ There are actually a few different versions of 'exact' methods for each contrast Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ could be used for almost-but-not-quite-strictly conservative coverage. +### 'Mid-P' Method(s) + +Exact methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. + +Although theoretically possible for any contrast, the mid-P method is only implemented in SAS for CIs for the single proportion and the odds ratio contrast. + ## Reference 1. \[References to be added\] From a0dd7c8548f3852934998665000699ee8661defa Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 16:39:29 +0000 Subject: [PATCH 26/58] remove rogue quotation mark (and test commit/push from RStudio on Mac) --- SAS/ci_for_2indep_prop.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 008bfdcab..247e76742 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -149,7 +149,7 @@ run; #| echo: false #| fig-align: center #| out-width: 50% - knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png")` + knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") ``` ```{r} From b8bf4d79701d6640372708cd427a4f474dd789d2 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 19:33:12 +0000 Subject: [PATCH 27/58] add citations --- method_summary/ci_for_prop_intro.qmd | 19 ++- method_summary/nature.csl | 189 +++++++++++++++++++++++++++ method_summary/references.bib | 176 +++++++++++++++++++++++++ 3 files changed, 373 insertions(+), 11 deletions(-) create mode 100644 method_summary/nature.csl diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index f84e96e43..b9abe1f35 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -1,6 +1,7 @@ --- title: "Confidence intervals for Proportions: General Information" bibliography: references.bib + csl: nature.csl --- ## Introduction @@ -9,7 +10,7 @@ The methods to use for calculating confidence intervals (CI) for binomial propor - 1 sample proportion, p (1 proportion calculated from 1 group of subjects) -- 2 sample proportions (p1 and p2) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, RD = p1 - p2), the ratio (relative risk, RR = p1 / p2), or the odds ratio (OR = p1 (1-p2) / (p2 (1-p1))). +- 2 sample proportions (p1 and p2) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) @@ -23,9 +24,9 @@ The poor performance of the Wald method is well documented - it can fail to achi ### Strictly conservative vs proximate coverage -Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described by Newcombe^5^ as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$", or in other words aiming for coverage to be either "strictly conservative" or "proximate". \[Brown et al. point out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions.\] Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" has been said to be something of a misnomer, and "continuity adjustment" is sometimes preferred \[see Campbell 2007, Newcombe 1998a\]. +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either "strictly conservative" or "proximate"[@laud2017]. \[It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001].\] Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" may be more reasonably termed a "continuity adjustment"[@newcombe1998c][@campbell2007]. -It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method *\[Ref\]*. In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. +It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method (e.g. see ). In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. ### Consistency with hypothesis tests, and one-sided coverage @@ -49,17 +50,17 @@ The Score confidence interval is also based on an asymptotic normal approximatio $Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion (Wilson score interval), this process simplifies to solving a quadratic equation in $p$. -For the contrast parameters, score methods were derived for independent proportions by Miettinen & Nurminen, Mee, and Koopman in the 1980s, and for paired data by Tango (1998), Nam & Blackwelder (2002) and Tang et al. (2003). In each case, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ (or for the paired case, the cell probability $p_{21}$) for the given value of $\theta$. +For the contrast parameters, score methods were derived for independent proportions by Miettinen & Nurminen and others in the 1980s[@miettinen1985][@Mee1984][@koopman1984], and related methods for paired data followed later[@tango1998][@tango1999][@nam2002][@tang2003]. In each case, the variance is a function of both $p_1$ and $p_2$, or of $\theta$ and a 'nuisance parameter', which is eliminated using the maximum likelihood estimate for $p_2$ (or for the paired case, the cell probability $p_{21}$) for the given value of $\theta$. ### SCAS Methods -Originating from papers published in 1985-1990 by Gart and Nam, the skewness-corrected asymptotic score ('SCAS') methods introduce a further term within the score statistic that involves $\tilde \mu$, the third central moment of $S(\theta)$ +Originating from papers published in 1985-1990 by Gart and Nam, the skewness-corrected asymptotic score ('SCAS') methods introduce a further term within the score statistic that involves $\tilde \mu$, the third central moment of $S(\theta)$: $$ Z(\theta) = \frac{S(\theta)}{\sqrt {\tilde V}} - \frac{(Z(\theta)^2 - 1)\tilde \mu_3}{6 \tilde V^{3/2}} $$ -Hence these are an extension of the asymptotic score intervals, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is also applied to the Wilson score method, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud (2017), with a further publication under review for paired data. +Hence these are an extension of the asymptotic score intervals, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], amd a further publication currently under review for paired data. \[SCAS intervals are not yet implemented in SAS PROC freq, but can be obtained using the %SCORECI and %PAIRBINCI macros.\] @@ -75,14 +76,10 @@ So-called 'exact' methods are designed to guarantee strictly conservative covera There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). -Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ could be used for almost-but-not-quite-strictly conservative coverage. +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. ### 'Mid-P' Method(s) Exact methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. Although theoretically possible for any contrast, the mid-P method is only implemented in SAS for CIs for the single proportion and the odds ratio contrast. - -## Reference - -1. \[References to be added\] diff --git a/method_summary/nature.csl b/method_summary/nature.csl new file mode 100644 index 000000000..3b1e3d7a2 --- /dev/null +++ b/method_summary/nature.csl @@ -0,0 +1,189 @@ + + diff --git a/method_summary/references.bib b/method_summary/references.bib index 05972a594..3ad933e25 100644 --- a/method_summary/references.bib +++ b/method_summary/references.bib @@ -11,3 +11,179 @@ @article{brown2001 doi = {10.1214/ss/1009213286}, url = {http://dx.doi.org/10.1214/ss/1009213286} } + +@book{newcombe2012, + title = {Confidence Intervals for Proportions and Related Measures of Effect Size}, + author = {Newcombe, Robert Gordon}, + year = {2012}, + month = {08}, + date = {2012-08-25}, + publisher = {CRC Press}, + doi = {10.1201/b12670}, + url = {http://dx.doi.org/10.1201/b12670}, + langid = {en} +} + +@article{newcombe1998c, + title = {Two-sided confidence intervals for the single proportion: comparison of seven methods}, + author = {Newcombe, Robert G.}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {857--872}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + langid = {en} +} + +@article{campbell2007, + title = {Chi{-}squared and Fisher{\textendash}Irwin tests of two{-}by{-}two tables with small sample recommendations}, + author = {Campbell, Ian}, + year = {2007}, + month = {02}, + date = {2007-02-21}, + journal = {Statistics in Medicine}, + pages = {3661--3675}, + volume = {26}, + number = {19}, + doi = {10.1002/sim.2832}, + url = {http://dx.doi.org/10.1002/sim.2832}, + langid = {en} +} + +@article{laud2017, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2017}, + month = {06}, + date = {2017-06-22}, + journal = {Pharmaceutical Statistics}, + pages = {334--348}, + volume = {16}, + number = {5}, + doi = {10.1002/pst.1813}, + url = {http://dx.doi.org/10.1002/pst.1813}, + langid = {en} +} + +@article{miettinen1985, + title = {Comparative analysis of two rates}, + author = {Miettinen, Olli and Nurminen, Markku}, + year = {1985}, + month = {04}, + date = {1985-04}, + journal = {Statistics in Medicine}, + pages = {213--226}, + volume = {4}, + number = {2}, + doi = {10.1002/sim.4780040211}, + url = {http://dx.doi.org/10.1002/sim.4780040211}, + langid = {en} +} + +@article{Mee1984, + ISSN = {0006341X, 15410420}, + URL = {http://www.jstor.org/stable/2531174}, + author = {Robert W. Mee and Dan Anbar}, + journal = {Biometrics}, + number = {4}, + pages = {1175--1176}, + publisher = {International Biometric Society}, + title = {Confidence Bounds for the Difference between Two Probabilities}, + urldate = {2026-02-23}, + volume = {40}, + year = {1984} +} + + +@article{koopman1984, + title = {Confidence Intervals for the Ratio of Two Binomial Proportions}, + author = {Koopman, P. A. R.}, + year = {1984}, + month = {06}, + date = {1984-06}, + journal = {Biometrics}, + pages = {513}, + volume = {40}, + number = {2}, + doi = {10.2307/2531405}, + url = {http://dx.doi.org/10.2307/2531405} +} + +@article{tango1998, + title = {Equivalence test and confidence interval for the difference in proportions for the paired-sample design}, + author = {Tango, Toshiro}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {891--908}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + langid = {en} +} + +@article{tango1999, + title = {Improved confidence intervals for the difference between binomial proportions based on paired data by Robert G. Newcombe,Statistics in Medicine,17, 2635-2650 (1998)}, + author = {Tango, T.}, + year = {1999}, + month = {12}, + date = {1999-12-30}, + journal = {Statistics in Medicine}, + pages = {3511--3513}, + volume = {18}, + number = {24}, + doi = {10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + langid = {en} +} + +@article{tang2003, + title = {On tests of equivalence via non{-}unity relative risk for matched{-}pair design}, + author = {Tang, {Nian{-}Sheng} and Tang, {Man{-}Lai} and Chan, Ivan Siu Fung}, + year = {2003}, + month = {03}, + date = {2003-03-18}, + journal = {Statistics in Medicine}, + pages = {1217--1233}, + volume = {22}, + number = {8}, + doi = {10.1002/sim.1213}, + url = {http://dx.doi.org/10.1002/sim.1213}, + langid = {en} +} + +@article{nam2002, + title = {Analysis of the ratio of marginal probabilities in a matched{-}pair setting}, + author = {Nam, {Jun{-}mo} and Blackwelder, William C.}, + year = {2002}, + month = {02}, + date = {2002-02-19}, + journal = {Statistics in Medicine}, + pages = {689--699}, + volume = {21}, + number = {5}, + doi = {10.1002/sim.1017}, + url = {http://dx.doi.org/10.1002/sim.1017}, + langid = {en} +} + +@article{laud2018, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2018}, + month = {05}, + date = {2018-05}, + journal = {Pharmaceutical Statistics}, + pages = {290--293}, + volume = {17}, + number = {3}, + doi = {10.1002/pst.1855}, + url = {http://dx.doi.org/10.1002/pst.1855}, + langid = {en} +} From 15bf6713bf4518e66d7880e8bae4dfedc5067f7f Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 19:34:27 +0000 Subject: [PATCH 28/58] add subscripts for p1 and p2 --- method_summary/ci_for_prop_intro.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index b9abe1f35..09bfd749d 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -10,7 +10,7 @@ The methods to use for calculating confidence intervals (CI) for binomial propor - 1 sample proportion, p (1 proportion calculated from 1 group of subjects) -- 2 sample proportions (p1 and p2) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). +- 2 sample proportions (\$p_1\$ and $p_2$) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) From 3d8eb574ff2febc72e897ceadd5cae418a09f917 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 19:36:37 +0000 Subject: [PATCH 29/58] make headings consistent --- method_summary/ci_for_prop_intro.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 09bfd749d..3a9a91b27 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -64,13 +64,13 @@ Hence these are an extension of the asymptotic score intervals, designed to addr \[SCAS intervals are not yet implemented in SAS PROC freq, but can be obtained using the %SCORECI and %PAIRBINCI macros.\] -### MOVER Method +### MOVER Methods This class of methods was popularised by Newcombe for the RD contrast, and is therefore labelled as the Newcombe method in current SAS PROC FREQ documentation. It has since been generalised to other contrasts and for paired data (though not implemented in SAS), and renamed as the Method of Variance Estimates Recovery (MOVER). It is also referred to as the 'Square-and-add' approach. MOVER intervals are produced by combining CIs for each of the two separate proportions involved (and an estimate of their correlation, in the paired data case). Originally Wilson intervals were used (hence earlier versions of SAS labelled the method as WILSON), but improved performance may be obtained by using Jeffreys intervals instead. -### 'Exact' Method(s) +### 'Exact' Methods So-called 'exact' methods are designed to guarantee strictly conservative coverage. (Newcombe points out some problems with the use of the term 'exact' here, hence the quotation marks.) For the single proportion, the Clopper-Pearson confidence interval is the range of p for which $P(X \ge x | p) = \alpha / 2$ and $P(X \le x | p) = \alpha / 2$. More complex calculations are involved for the contrast parameters, which eliminate the nuisance parameter by taking the supremum of all p-values across the range of combinations of $p_1$ and $p_2$ for a given $\theta$. @@ -78,7 +78,7 @@ There are actually a few different versions of 'exact' methods for each contrast Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. -### 'Mid-P' Method(s) +### 'Mid-P' Methods Exact methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. From 949103a0e3aae222f5bbcf26fd8db221d30fdb46 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 20:02:05 +0000 Subject: [PATCH 30/58] mention continuity adjustment for Jeffreys --- method_summary/ci_for_prop_intro.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 3a9a91b27..ddea9d1bc 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -76,7 +76,7 @@ So-called 'exact' methods are designed to guarantee strictly conservative covera There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). -Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. (Note that such an adjustment is also possible for methods based on Jeffreys intervals, such as MOVER.) ### 'Mid-P' Methods From 18e1dd56ad5e33a354038e9ec9dd783e233aa742 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Mon, 23 Feb 2026 20:02:33 +0000 Subject: [PATCH 31/58] add details for continuity adjusted methods and SAS PROC FREQ options --- SAS/ci_for_prop.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index e4257425b..76d20b567 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -142,7 +142,7 @@ The Blaker method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=BLAKER);` ## Continuity Adjusted Methods -\[TBC\] +SAS offers the 'CORRECT' option for producing more conservative versions of the Wald and Wilson Score intervals. ## Consistency with hypothesis tests @@ -185,9 +185,11 @@ By adding the option `BINOMIAL(LEVEL="Yes" CL=)`, the other C - `BINOMIAL(LEVEL="Yes" CL=CLOPPERPEARSON WALD WILSON AGRESTICOULL JEFFREYS MIDP LIKELIHOODRATIO LOGIT BLAKER)` will return Agresti-Coull, Blaker, Clopper-Pearson(Exact), Wald(without continuity correction) Wilson(without continuity correction), Jeffreys, Mid-P, Likelihood Ratio, and Logit -- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural for Mid-P to be included here instead of Clopper-Pearson!\]* +- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural/logical for Mid-P to be included here instead of Clopper-Pearson!\]* -- `BINOMIALc(LEVEL="Yes" CL=ALL);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction) +- `BINOMIALc(LEVEL="Yes" CL=ALL);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction). *\[... following the above comment, it is also not logical for Jeffreys and Agresti-Coull to be included here\]* + + - Note that this option does not appear in current SAS/STAT documentation, but appears to still work. - `BINOMIALc(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson(with continuity correction) and Wald (with continuity correction) From 2daec26e6d7c1e285649775773b66f8d6b29ba9d Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 26 Feb 2026 18:03:38 +0000 Subject: [PATCH 32/58] intro improvements & additions --- method_summary/ci_for_prop_intro.qmd | 32 ++++++++++++++++------------ method_summary/references.bib | 15 +++++++++++++ 2 files changed, 33 insertions(+), 14 deletions(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index ddea9d1bc..9baac0ef0 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -6,7 +6,7 @@ ## Introduction -The methods to use for calculating confidence intervals (CI) for binomial proportions depend on the situation: +The methods to use for calculating confidence intervals (CIs) for binomial proportions depend on the situation: - 1 sample proportion, p (1 proportion calculated from 1 group of subjects) @@ -16,37 +16,39 @@ The methods to use for calculating confidence intervals (CI) for binomial propor - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). -The production of a confidence interval for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the confidence interval spans a pre-specified NI margin or not. +The production of a CI for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the CI spans a pre-specified NI margin or not. -Some sources suggest selecting a different method depending on your sample size and whether your proportion is close to 0 or 1 (rather than near to the 0.5 midpoint). Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative. However, options are now available for CI methods that have appropriate coverage properties across the whole parameter space, which with modern computing software are easily obtained, so there is no need for such a data-driven approach to method selection or reliance on overly conservative methods. +Some sources suggest selecting a different method depending on your sample size and whether your proportion takes an extreme value (i.e. close to 0 or 1). Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative, but options are now available for CI methods that have appropriate coverage properties across the whole parameter space. These are easily obtained with modern computing software, so there is no need for a data-driven approach to method selection or reliance on overly conservative methods. -The poor performance of the Wald method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided[@brown2001, p128]. Inexplicably, it remains a default output component in SAS, so continues to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] +The poor performance of the approximate normal ('Wald') methods is well documented - they can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that they should be avoided[@brown2001, p128]. Wald intervals remain a default output component in SAS, so continue to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] ### Strictly conservative vs proximate coverage -Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. CI methods may be designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either "strictly conservative" or "proximate"[@laud2017]. \[It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001].\] Many alternative CI methods are designed to meet the latter criterion, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee the former, or include an optional adjustment ("continuity correction") to emulate the same effect using asymptotic formulae. The term "continuity correction" may be more reasonably termed a "continuity adjustment"[@newcombe1998c][@campbell2007]. +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. Some CI methods are designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either "strictly conservative" or "proximate"[@laud2017]. \[It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001].\] The debate makes it difficult to state in simple objective terms whether one method is 'better' than another. -It is worth noting that although one might assume that regulatory authorities would take the latter stance and insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method (e.g. see ). In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. +Many CI methods are designed to achieve proximate coverage, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee strictly conservative coverage, while others include an optional adjustment ('continuity correction') to emulate the same effect using asymptotic formulae. The term 'continuity adjustment' is used to avoid the implication that the non-adjusted methods are inferior[@newcombe1998c][@campbell2007]. + +It is worth noting that although one might assume that regulatory authorities would insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method (e.g. see ). In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. ### Consistency with hypothesis tests, and one-sided coverage -Depending on the analysis plan, it may be desirable for a reported confidence interval to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a superiority test (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. +Depending on the analysis plan, it may be desirable for a reported CI to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a test for association (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. -A related issue is whether it is sufficient for two-sided coverage probability to be aligned with the nominal $(1-\alpha)$, or whether the tail probabilities (i.e. one-sided non-coverage at each end) should also be aligned with the nominal $\alpha/2$. Clearly the latter criterion (described as 'central interval location' by Newcombe) is essential for non-inferiority testing, but it is also a desirable property for any confidence interval. +A related issue is whether it is sufficient for two-sided coverage probability to be aligned with the nominal $(1-\alpha)$, or whether the tail probabilities (i.e. one-sided non-coverage at each end) should also be aligned with the nominal $\alpha/2$. Clearly the latter 'equal-tailed' or symmetric coverage criterion (described as 'central interval location' by Newcombe) is essential for non-inferiority testing, but it is also a desirable general property for any CI. # Overarching methods There are some methods that apply (or have variants which apply) across all contrast parameters (i.e. $\theta=p$ or $\theta=RD$ etc.), which are outlined below. -### Normal Approximation Methods (Also known as the Wald Method) +### Normal Approximation Methods (Also known as the Wald Methods) The Wald confidence interval is constructed simply from the the point estimate plus and minus a multiple of its estimated standard error: $(L_{Wald}, U_{Wald}) = \hat \theta \pm z_{\alpha/2} \sqrt{\hat V(\hat\theta)}$ where $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$. -For RR and OR, $\theta$ is log-transformed to improve the normal approximation, and $(L_{Wald}, U_{Wald}) = exp[ ln(\hat \theta) \pm z_{\alpha/2} \sqrt{\hat V(ln(\hat\theta))}]$, which requires a modification for boundary cases. +For RR and OR, $\theta$ is first log-transformed to improve the normal approximation, and then $(L_{Wald}, U_{Wald}) = exp[ ln(\hat \theta) \pm z_{\alpha/2} \sqrt{\hat V(ln(\hat\theta))}]$, which requires a modification for boundary cases. ### Asymptotic Score Methods -The Score confidence interval is also based on an asymptotic normal approximation, but uses a score statistic $Z(\theta) = S(\theta)/\sqrt {\tilde V}$ based on contrast function $S(\theta)$ and its true variance $\tilde V = V(S(\theta))$. For example, for the single proportion, $S(p) = \hat p - p$, or for RD, $S(\theta) = \hat p_1 - \hat p_2 - \theta$. +The Score confidence interval is also based on an asymptotic normal approximation, but uses a score statistic $Z(\theta) = S(\theta)/\sqrt {\tilde V}$. This statistic is based on a contrast function $S(\theta)$ (e.g. for the single proportion, $S(p) = \hat p - p$, or for RD, $S(\theta) = \hat p_1 - \hat p_2 - \theta$), and a maximum likelihood estimate of its variance $\tilde V = V(S(\theta))$, which is also function of $\theta$. $Z(\theta)$ can be evaluated at any value of $\theta$ to find the range of values for which $|Z(\theta)|< z_{\alpha/2}$. For the single proportion (Wilson score interval), this process simplifies to solving a quadratic equation in $p$. @@ -60,9 +62,11 @@ $$ Z(\theta) = \frac{S(\theta)}{\sqrt {\tilde V}} - \frac{(Z(\theta)^2 - 1)\tilde \mu_3}{6 \tilde V^{3/2}} $$ -Hence these are an extension of the asymptotic score intervals, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], amd a further publication currently under review for paired data. +Hence these are an extension of the asymptotic score methods, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], and a further publication currently under review for paired data. + +SCAS intervals generally achieve one-sided coverage that is very close to the nominal $\alpha/2$, which naturally also leads to excellent two-sided coverage. Symmetric strictly conservative coverage can be achieved by adding a continuity adjustment. -\[SCAS intervals are not yet implemented in SAS PROC freq, but can be obtained using the %SCORECI and %PAIRBINCI macros.\] +The SCAS method is not implemented in SAS PROC FREQ, but can be obtained using the %SCORECI and %PAIRBINCI macros from . ### MOVER Methods @@ -76,7 +80,7 @@ So-called 'exact' methods are designed to guarantee strictly conservative covera There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). -Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $0.25 \times n$ (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. (Note that such an adjustment is also possible for methods based on Jeffreys intervals, such as MOVER.) +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $3/8 \times n$ [@mehrotra2000], or $0.25 \times n$[@laud2017, Appendix S2] (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. (Note that such an adjustment is also possible for methods based on Jeffreys intervals, such as MOVER.) ### 'Mid-P' Methods diff --git a/method_summary/references.bib b/method_summary/references.bib index 3ad933e25..8b061cbf8 100644 --- a/method_summary/references.bib +++ b/method_summary/references.bib @@ -187,3 +187,18 @@ @article{laud2018 url = {http://dx.doi.org/10.1002/pst.1855}, langid = {en} } + +@article{mehrotra2000, + title = {Minimum risk weights for comparing treatments in stratified binomial trials}, + author = {Mehrotra, Devan V. and Railkar, Radha}, + year = {2000}, + month = {03}, + date = {2000-03-30}, + journal = {Statistics in Medicine}, + pages = {811--825}, + volume = {19}, + number = {6}, + doi = {10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + langid = {en} +} From fe1bf3775c20d795521a3d49ea2e16c73cac2569 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 26 Feb 2026 18:47:21 +0000 Subject: [PATCH 33/58] several general edits/clarifications, plus correction to Jeffreys and Agresti-Coull sections --- SAS/ci_for_prop.qmd | 57 ++++++++++++++++++++++++--------------------- 1 file changed, 30 insertions(+), 27 deletions(-) diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 76d20b567..6de961560 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -4,9 +4,9 @@ title: "Confidence intervals for a Proportion in SAS" ## Introduction -\[See separate page for general introductory information on confidence intervals for proportions.\] +See separate page for general introductory information on confidence intervals for proportions. -## Data used +## Data Used The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. @@ -21,7 +21,7 @@ data adcibc2 (keep=trt resp) ; run; ``` -The below shows that for the Actual Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). ```{sas} #| eval: false @@ -37,29 +37,29 @@ run; knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ``` -## Methods for Calculating Confidence Intervals for a single proportion +## Methods for Calculating Confidence Intervals for a Single Proportion -Here we are calculating a 95% confidence interval for the proportion of responders in the active treatment group. +Here we are calculating a $100(1-\alpha)\%$ (usually 95%) confidence interval for the proportion $p$ of responders in the active treatment group, estimated from the sample proportion $\hat p = x / n$. SAS PROC FREQ in Version 9.4 can compute 11 methods to calculate CIs for a single proportion, an explanation of most of these methods and the code is shown below. See [BINOMIAL](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm)^1^ for more information on SAS parameterization. It is recommended to always sort your data prior to doing a PROC FREQ. For more information about some of these methods in R & SAS, including which performs better in different scenarios see [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f)^2^ and [Confidence Intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf)^3^. Key literature on the subject includes papers by [Brown et al.](https://www.jstor.org/stable/2676784?seq=1)^4^ and [Newcombe](https://pubmed.ncbi.nlm.nih.gov/9595616/)^5^. -### Clopper-Pearson (Exact or binomial CI) method +### Clopper-Pearson (Exact or binomial CI) Method -With binary endpoint data (response/non-response), we make the assumption that the proportion of responders has been derived from a series of Bernoulli trials. Trials (Subjects) are independent and we have a fixed number of repeated trials with an outcome of respond or not respond. This type of data follows the discrete binomial probability distribution, and the Clopper-Pearson^6^ (Exact) method uses this distribution to calculate the CIs. +With binary endpoint data (response/non-response), we make the assumption that the proportion of responders has been derived from a series of Bernoulli trials. Trials (Subjects) are independent and we have a fixed number of repeated trials with an outcome of respond or not respond. This type of data follows the discrete binomial probability distribution, and the Clopper-Pearson^6^ ('exact') method uses this distribution to calculate the CIs. This method guarantees strictly conservative coverage, but has been noted to be excessively conservative, as for any given proportion, the actual coverage probability can be much larger than $(1-\alpha)$. The Clopper-Pearson method is output by SAS as one of the default methods (labelled as "Exact Conf Limits" in the "Proportion" ODS output object), but you can also specify it using `BINOMIAL(LEVEL="Yes" CL=CLOPPERPEARSON);` which creates a separate output object "ProportionCLs". -### Normal Approximation method (Also known as the Wald or asymptotic CI method) +### Normal Approximation Method (Also known as the Wald Method) -The traditional alternative to the Clopper-Pearson (Exact) method is the asymptotic Normal Approximation (Wald) CI. The poor performance of this method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Nevertheless, it remains a default output component in SAS, so is included here for reference. +The traditional alternative to the Clopper-Pearson ('Exact') method is the asymptotic Normal Approximation (Wald) CI. The poor performance of this method is well documented - it can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that it should be avoided^4, p128^. Nevertheless, it remains a default output component in SAS, so is included here for reference. In large random samples from independent trials, the sampling distribution of proportions approximately follows the normal distribution. The expectation of a sample proportion is the corresponding population proportion. Therefore, based on a sample of size $n$, a $(1-\alpha)\%$ confidence interval for population proportion can be calculated using the normal approximation as follows: -$p\approx \hat p \pm z_{\alpha/2} \sqrt{\hat p(1-\hat p)/n}$, where $\hat p$ is the sample proportion, $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$, and $\sqrt{\hat p(1-\hat p)/n}$ is the estimated standard error. +$p\approx \hat p \pm z_{\alpha/2} \sqrt{\hat p(1-\hat p)/n}$, where $\hat p=x/n$ is the sample proportion, $z_{\alpha/2}$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to the confidence level $(1-\alpha)$, and $\sqrt{\hat p(1-\hat p)/n}$ is the estimated standard error. One should note that the approximation can become increasingly unreliable as the proportion of responders gets closer to 0 or 1 (e.g. 0 or 100% responding). In this scenario, common issues consist of: @@ -83,12 +83,13 @@ The Wilson (Score) method is also based on an asymptotic normal approximation, b The method can be derived with or without a Yates continuity correction. The corrected interval closely approximates the coverage of the Clopper-Pearson method, but only in terms of overall two-sided coverage - due to the asymmetric coverage, it does not guarantee that the non-coverage probability is less than $\alpha/2$. -Let p=r/n, where r= number of responses, and n=number of subjects, q=1-p, and z= the appropriate value from standard normal distribution:\ -$$ z{_{1-\alpha/2}} $$For example, for 95% confidence intervals, $\alpha=0.05$, using standard normal tables, z in the equations below will take the value =1.96. Calculate 3 quantities +Let $\hat p$ =r/n, where r= number of responses, and n=number of subjects, $\hat q = 1- \hat p$, and z= the appropriate value from standard normal distribution: $z_{1-\alpha/2}$. + +For example, for 95% confidence intervals, $\alpha=0.05$, using standard normal tables, z in the equations below will take the value =1.96. Calculate 3 quantities $$ A= 2r+z^2$$ -$$ B=z\sqrt(z^2 + 4rq) $$ $$ C=2(n+z^2) $$The method calculates the confidence interval \[Lower, Upper\] as: \[(A-B)/C, (A+B)/C\] +$$ B=z\sqrt(z^2 + 4r \hat q) $$ $$ C=2(n+z^2) $$The method calculates the confidence interval \[Lower, Upper\] as: \[(A-B)/C, (A+B)/C\] A = 2 \* 36 + 1.96\^2 = 75.8416 @@ -110,21 +111,25 @@ The only differences in the equations to calculate the Wilson score with continu $$ A= 2r+z^2 -1$$ -$$ B=z\sqrt(z^2 - 2 -\frac{1}{n} + 4rq) $$ +$$ B=z\sqrt(z^2 - 2 -\frac{1}{n} + 4r \hat q) $$ ### Agresti-Coull method -The Agresti-Coull method is a 'simple solution' designed to improve coverage compared to the Wald method and still perform better (i.e. less conservative) than Clopper-Pearson particularly when the probability isn't in the mid-range (0.5). It is less conservative whilst still having good coverage. The only difference compared to the Wald method is that it adds two successes and two failures to the original observations (increasing the sample by 4 observations). In practice it is not often used *\[Ed: what is the basis for this statement?\]*. +The Agresti-Coull method is a 'simple solution' designed to improve coverage compared to the Wald method and still perform better (i.e. less conservative) than Clopper-Pearson particularly when the probability isn't in the mid-range (0.5). It is less conservative whilst still having good coverage. The only difference compared to the Wald method is that it adds $z^2/2$ successes and failures to the original observations (when $\alpha=0.05$ this increases the sample by approximately 4 observations). The Agresti-Coull method is output by SAS using `BINOMIAL(LEVEL="Yes" CL=AGRESTICOULL);` ### Jeffreys method -The Jeffreys method is a particular type of Bayesian Highest Probability Density (HPD) method. For binomial proportions, the beta distribution is generally used for the conjugate prior, which consists of two parameters $\alpha$ and $\beta$. Setting $\alpha=\beta=0.5$ is called the Jeffreys prior. This is considered as non-informative for a binomial proportion. +The Jeffreys method is a particular type of Bayesian Highest Probability Density (HPD) method. For binomial proportions, the beta distribution is generally used for the conjugate prior, which consists of two parameters $\alpha'$ and $\beta'$. Setting $\alpha'=\beta'=0.5$ is called the Jeffreys prior. This is considered as non-informative for a binomial proportion. The resulting posterior density gives the CI as $$ -(Beta (^k/_2 + ^1/_{2}, ^{(n-k)}/_2+^1/_2)_{\alpha}, Beta (^k/_2 + ^1/_{2}, ^{(n-k)}/_2+^1/_2)_{1-\alpha}) -$$The coverage probabilities of the Jeffreys method are centred around the nominal confidence level on average, with symmetric "equal-tailed" 1-sided coverage^8 Appx S3.5^. This interval is output by SAS using `BINOMIAL(LEVEL="Yes" CL=Jeffreys);` +(Beta (x + 0.5, n - x + 0.5)_{\alpha/2}, Beta (x + 0.5, n - x + 0.5)_{1-\alpha/2}) +$$ + +Boundary modifications are applied to force the lower limit to 0 if x=0 and the upper limit to 1 if x=n. + +The coverage probabilities of the Jeffreys method are centred around the nominal confidence level on average, with symmetric "equal-tailed" 1-sided coverage^8 Appx S3.5^. This interval is output by SAS using `BINOMIAL(LEVEL="Yes" CL=Jeffreys);` ### Binomial based Mid-P method @@ -146,7 +151,7 @@ SAS offers the 'CORRECT' option for producing more conservative versions of the ## Consistency with hypothesis tests -Within SAS for the asymptotic methods, consistency with the binomial test is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. +Within SAS PROC FREQ for the asymptotic methods, consistency with the binomial test is provided by the Wilson CI, because the asymptotic test uses the standard error estimated under the null hypothesis (i.e. the value specified in the `BINOMIAL(P=...)` option of the `TABLES` statement. If an `EXACT BINOMIAL;` statement is used, the resulting test is consistent with the Clopper-Pearson CI. @@ -154,13 +159,13 @@ An exact hypothesis test with mid-P adjustment (consistent with the mid-P CI) is ## Example Code using PROC FREQ -By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ', SAS outputs the Normal Approximation (Wald) and Clopper-Pearson (Exact) confidence intervals as two default methods, derived for the `Responders` = `Yes`. If you do not specify the `LEVEL` you want to model, then SAS assumes you want to model the first level that appears in the output (alphabetically). +By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ' TABLES statement, SAS outputs the Normal Approximation (Wald) and Clopper-Pearson (Exact) confidence intervals as two default methods, derived for the `Responders` = `Yes`. If you do not specify the `LEVEL` you want to model, then SAS assumes you want to model the first level that appears in the output (alphabetically). **It is very important to ensure you are calculating the CI for the correct level! Check your output to confirm, you will see below it states `resp=Yes` !** -Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. +Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can (and should) be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. -The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method (Exact), and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. +The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method, and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. ```{sas} #| eval: false @@ -185,13 +190,11 @@ By adding the option `BINOMIAL(LEVEL="Yes" CL=)`, the other C - `BINOMIAL(LEVEL="Yes" CL=CLOPPERPEARSON WALD WILSON AGRESTICOULL JEFFREYS MIDP LIKELIHOODRATIO LOGIT BLAKER)` will return Agresti-Coull, Blaker, Clopper-Pearson(Exact), Wald(without continuity correction) Wilson(without continuity correction), Jeffreys, Mid-P, Likelihood Ratio, and Logit -- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural/logical for Mid-P to be included here instead of Clopper-Pearson!\]* - -- `BINOMIALc(LEVEL="Yes" CL=ALL);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction). *\[... following the above comment, it is also not logical for Jeffreys and Agresti-Coull to be included here\]* +- `BINOMIAL(LEVEL="Yes" CL=ALL);` will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald(without continuity correction), Wilson (without continuity correction). *\[If the developers of SAS are reading this, it would seem more natural/logical for Mid-P to be included here instead of Clopper-Pearson! However, note that the ALL option is not mentioned in the current PROC FREQ documentation.\]* - - Note that this option does not appear in current SAS/STAT documentation, but appears to still work. +- `BINOMIAL(LEVEL="Yes" CL=ALL CORRECT);`will return Agresti-Coull, Clopper-Pearson (Exact), Jeffreys, Wald (with continuity correction), Wilson(with continuity correction). In previous versions of SAS this was coded as `BINOMIALc(...)`, but that is no longer documented. *\[... following the above comment, it is also not logical for Jeffreys and Agresti-Coull to be included here alongside methods designed to achieve conservative coverage\]* -- `BINOMIALc(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson(with continuity correction) and Wald (with continuity correction) +- `BINOMIAL(LEVEL="Yes" CL=WILSON(CORRECT) WALD(CORRECT));`will return Wilson (with continuity correction) and Wald (with continuity correction) ```{sas} #| eval: false From 5d7307ac16d9a1848a649ed6a1a1ad59f41d7046 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:23:55 +0000 Subject: [PATCH 34/58] make some subtle language improvements and add Cai reference --- method_summary/ci_for_prop_intro.qmd | 22 +++++++++++----------- method_summary/references.bib | 15 +++++++++++++++ 2 files changed, 26 insertions(+), 11 deletions(-) diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 9baac0ef0..62f655bf8 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -10,29 +10,29 @@ The methods to use for calculating confidence intervals (CIs) for binomial propo - 1 sample proportion, p (1 proportion calculated from 1 group of subjects) -- 2 sample proportions (\$p_1\$ and $p_2$) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). +- 2 sample proportions ($p_1$ and $p_2$) and you want a CI for a contrast parameter, such as the difference in the 2 proportions (risk difference, $RD = p_1 - p_2$), the ratio (relative risk, $RR = p_1 / p_2$), or the odds ratio ($OR = p_1 (1-p_2) / (p_2 (1-p_1))$). - If the 2 samples come from 2 independent samples (different subjects in each of the 2 treatment groups) - If the 2 samples are matched (i.e. the same subject has 2 results, one from each treatment \[paired data\]). -The production of a CI for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the CI spans a pre-specified NI margin or not. +Reporting a CI for such contrasts is useful for giving a clinically interpretable estimate of the magnitude of a treatment effect. Moreover, it is particularly relevant for non-inferiority (NI) or equivalence trials, where the hypothesis test is an assessment of whether the CI spans a pre-specified NI margin or not. -Some sources suggest selecting a different method depending on your sample size and whether your proportion takes an extreme value (i.e. close to 0 or 1). Similarly, for a 2x2 table, there is a widely-held belief that an 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative, but options are now available for CI methods that have appropriate coverage properties across the whole parameter space. These are easily obtained with modern computing software, so there is no need for a data-driven approach to method selection or reliance on overly conservative methods. +Some sources suggest selecting a different method depending on your sample size and whether your proportion takes an extreme value (i.e. close to 0 or 1). Similarly, for a 2x2 table, there is a widely-held belief that a so-called 'exact' method should be used if expected cell counts are less than 5. These conventions originated at a time when the Wald method was the only available alternative, but options are now available for CI methods that have appropriate coverage properties across the whole parameter space. These are easily obtained with modern computing software, so there is no need for a data-driven approach to method selection or reliance on overly conservative methods. The poor performance of the approximate normal ('Wald') methods is well documented - they can fail to achieve the nominal confidence level even with large sample sizes, and there is a consensus in the literature that they should be avoided[@brown2001, p128]. Wald intervals remain a default output component in SAS, so continue to be widely used, but alternative methods are strongly recommended. \[Note in particular that most other methods become conservative for very small cell counts, which suggests there is no need to resort to 'exact' methods.\] ### Strictly conservative vs proximate coverage -Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. Some CI methods are designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either "strictly conservative" or "proximate"[@laud2017]. \[It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001].\] The debate makes it difficult to state in simple objective terms whether one method is 'better' than another. +Because of the discrete nature of binomial data, it is impossible for any CI method to cover the true value of the estimated proportion precisely 95% of the time for all values of p. Some CI methods are designed to achieve the nominal confidence level **as a minimum**, but many researchers find such a criterion to be excessive, producing CIs that are unnecessarily wide. The alternative position is to aim for coverage probability that is close to the nominal confidence level **on average.** These two opposing positions are described as "aiming to align either the minimum or the mean coverage with the nominal $(1-\alpha)$"[@newcombe2012], or in other words aiming for coverage to be either 'strictly conservative' or 'proximate'[@laud2017]. It has been pointed out that the proximate stance is consistent with most other types of statistical models, based on approximate assumptions[@brown2001]. The debate makes it difficult to state in simple objective terms whether one method is 'better' than another. -Many CI methods are designed to achieve proximate coverage, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee strictly conservative coverage, while others include an optional adjustment ('continuity correction') to emulate the same effect using asymptotic formulae. The term 'continuity adjustment' is used to avoid the implication that the non-adjusted methods are inferior[@newcombe1998c][@campbell2007]. +Many CI methods are designed to achieve proximate coverage, but some employ computationally intensive approaches (often labelled as 'exact' methods) to guarantee strictly conservative coverage, while others include an optional adjustment ('continuity correction') to emulate the same effect using asymptotic formulae. The term 'adjustment' may be used instead of 'correction' to avoid the implication that the non-adjusted methods are inferior[@newcombe1998c][@campbell2007]. It is worth noting that although one might assume that regulatory authorities would insist on strictly conservative coverage (noting that one-sided non-coverage equates to the type 1 error of a NI test), in recent years it appears that the FDA's preferred method for RD is the Miettinen-Nurminen (MN) method (e.g. see ). In contrast, for a single proportion, the 'exact' Clopper-Pearson method seems (anecdotally) to be more commonly preferred. ### Consistency with hypothesis tests, and one-sided coverage -Depending on the analysis plan, it may be desirable for a reported CI to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This includes the special case of a test for association (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. +It is desirable (and in some analysis contexts, essential) for a reported CI to be consistent with the result of a hypothesis test, so that for example the 95% CI will exclude the null hypothesis value if and only if the test p-value is less than 0.05 (or 0.025 one-sided). This applies for the special case of a test for association (where the null hypothesis is of 0 difference between the groups), but also the more general case of a non-zero null hypothesis for NI or equivalence testing. It should be noted that not all CI methods satisfy this criterion. A related issue is whether it is sufficient for two-sided coverage probability to be aligned with the nominal $(1-\alpha)$, or whether the tail probabilities (i.e. one-sided non-coverage at each end) should also be aligned with the nominal $\alpha/2$. Clearly the latter 'equal-tailed' or symmetric coverage criterion (described as 'central interval location' by Newcombe) is essential for non-inferiority testing, but it is also a desirable general property for any CI. @@ -62,17 +62,17 @@ $$ Z(\theta) = \frac{S(\theta)}{\sqrt {\tilde V}} - \frac{(Z(\theta)^2 - 1)\tilde \mu_3}{6 \tilde V^{3/2}} $$ -Hence these are an extension of the asymptotic score methods, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], and a further publication currently under review for paired data. +Hence these are an extension of the asymptotic score methods, designed to address asymmetric coverage which was observed for the score method, particularly for the RR contrast. The same principle is applied to the Wilson score interval for a single proportion, which has also been noted to have a systematic bias in one-sided coverage[@tonycai2005]. A unified family of methods covering the single proportion and all independent contrasts was described by Laud[@laud2017], with the addition of a bias correction for the OR case[@laud2018], and a further publication currently under review for paired data. SCAS intervals generally achieve one-sided coverage that is very close to the nominal $\alpha/2$, which naturally also leads to excellent two-sided coverage. Symmetric strictly conservative coverage can be achieved by adding a continuity adjustment. The SCAS method is not implemented in SAS PROC FREQ, but can be obtained using the %SCORECI and %PAIRBINCI macros from . -### MOVER Methods +### MOVER Methods (Also known as the Newcombe Method) This class of methods was popularised by Newcombe for the RD contrast, and is therefore labelled as the Newcombe method in current SAS PROC FREQ documentation. It has since been generalised to other contrasts and for paired data (though not implemented in SAS), and renamed as the Method of Variance Estimates Recovery (MOVER). It is also referred to as the 'Square-and-add' approach. -MOVER intervals are produced by combining CIs for each of the two separate proportions involved (and an estimate of their correlation, in the paired data case). Originally Wilson intervals were used (hence earlier versions of SAS labelled the method as WILSON), but improved performance may be obtained by using Jeffreys intervals instead. +MOVER intervals are produced by combining CIs for each of the two separate proportions involved (and an estimate of their correlation, in the paired data case). Originally Wilson Score intervals were used (hence earlier versions of SAS (e.g. v9.3) labelled this method as WILSON or SCORE), but improved performance may be obtained by using Jeffreys intervals instead. ### 'Exact' Methods @@ -80,10 +80,10 @@ So-called 'exact' methods are designed to guarantee strictly conservative covera There are actually a few different versions of 'exact' methods for each contrast, with alternatives aimed at reducing the conservatism of coverage. For example, Chan-Zhang is an improvement on Santner-Snell. However, it should be noted that some methods (e.g. Blaker, Agresti-Min) achieve reduced conservatism by inverting a two-sided exact test, but as a result they do not satisfy the strictly conservative criterion for one-sided coverage (and therefore would produce inflated type 1 error rates if used for a non-inferiority test). -Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $3/8 \times n$ [@mehrotra2000], or $0.25 \times n$[@laud2017, Appendix S2] (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used for almost-but-not-quite-strictly conservative coverage. (Note that such an adjustment is also possible for methods based on Jeffreys intervals, such as MOVER.) +Several asymptotic methods offer a continuity adjustment, which aims to emulate 'exact' methods and achieve almost conservative coverage, by for example adding a quantity of the order $0.5 \times n$ to the formula. Generally this is not a successful modification to the Wald method, but can produce good results for the score methods. It has been suggested that an adjustment of smaller magnitude such as $3/8 \times n$ [@mehrotra2000], or $0.25 \times n$[@laud2017, Appendix S2] (or any other selected value on a 'sliding scale' from 0 to 0.5) could be used to achieve coverage that is at least $(1-\alpha)$ nearly all of the time. ### 'Mid-P' Methods -Exact methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. +'Exact' methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. Although theoretically possible for any contrast, the mid-P method is only implemented in SAS for CIs for the single proportion and the odds ratio contrast. diff --git a/method_summary/references.bib b/method_summary/references.bib index 8b061cbf8..c7086d9ae 100644 --- a/method_summary/references.bib +++ b/method_summary/references.bib @@ -202,3 +202,18 @@ @article{mehrotra2000 url = {http://dx.doi.org/10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, langid = {en} } + +@article{tonycai2005, + title = {One-sided confidence intervals in discrete distributions}, + author = {Tony Cai, T.}, + year = {2005}, + month = {04}, + date = {2005-04}, + journal = {Journal of Statistical Planning and Inference}, + pages = {63--88}, + volume = {131}, + number = {1}, + doi = {10.1016/j.jspi.2004.01.005}, + url = {http://dx.doi.org/10.1016/j.jspi.2004.01.005}, + langid = {en} +} From 498a517bb3237d67f9643694bd3a04ec706a459b Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:24:33 +0000 Subject: [PATCH 35/58] add bibliography file --- SAS/ci_for_2indep_prop.qmd | 2 + SAS/references.bib | 248 +++++++++++++++++++++++++++++++++++++ 2 files changed, 250 insertions(+) create mode 100644 SAS/references.bib diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 247e76742..8abfd49a0 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -1,5 +1,7 @@ --- title: "Confidence Intervals for Independent Proportions in SAS" +bibliography: references.bib +csl: nature.csl --- ## Introduction diff --git a/SAS/references.bib b/SAS/references.bib new file mode 100644 index 000000000..07114da8b --- /dev/null +++ b/SAS/references.bib @@ -0,0 +1,248 @@ + +@article{brown2001, + title = {Interval Estimation for a Binomial Proportion}, + author = {Brown, Lawrence D. and Cai, T. Tony and DasGupta, Anirban}, + year = {2001}, + month = {05}, + date = {2001-05-01}, + journal = {Statistical Science}, + volume = {16}, + number = {2}, + doi = {10.1214/ss/1009213286}, + url = {http://dx.doi.org/10.1214/ss/1009213286} +} + +@book{newcombe2012, + title = {Confidence Intervals for Proportions and Related Measures of Effect Size}, + author = {Newcombe, Robert Gordon}, + year = {2012}, + month = {08}, + date = {2012-08-25}, + publisher = {CRC Press}, + doi = {10.1201/b12670}, + url = {http://dx.doi.org/10.1201/b12670}, + langid = {en} +} + +@article{newcombe1998c, + title = {Two-sided confidence intervals for the single proportion: comparison of seven methods}, + author = {Newcombe, Robert G.}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {857--872}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<857::aid-sim777>3.0.co;2-e}, + langid = {en} +} + +@article{campbell2007, + title = {Chi{-}squared and Fisher{\textendash}Irwin tests of two{-}by{-}two tables with small sample recommendations}, + author = {Campbell, Ian}, + year = {2007}, + month = {02}, + date = {2007-02-21}, + journal = {Statistics in Medicine}, + pages = {3661--3675}, + volume = {26}, + number = {19}, + doi = {10.1002/sim.2832}, + url = {http://dx.doi.org/10.1002/sim.2832}, + langid = {en} +} + +@article{laud2017, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2017}, + month = {06}, + date = {2017-06-22}, + journal = {Pharmaceutical Statistics}, + pages = {334--348}, + volume = {16}, + number = {5}, + doi = {10.1002/pst.1813}, + url = {http://dx.doi.org/10.1002/pst.1813}, + langid = {en} +} + +@article{miettinen1985, + title = {Comparative analysis of two rates}, + author = {Miettinen, Olli and Nurminen, Markku}, + year = {1985}, + month = {04}, + date = {1985-04}, + journal = {Statistics in Medicine}, + pages = {213--226}, + volume = {4}, + number = {2}, + doi = {10.1002/sim.4780040211}, + url = {http://dx.doi.org/10.1002/sim.4780040211}, + langid = {en} +} + +@article{Mee1984, + ISSN = {0006341X, 15410420}, + URL = {http://www.jstor.org/stable/2531174}, + author = {Robert W. Mee and Dan Anbar}, + journal = {Biometrics}, + number = {4}, + pages = {1175--1176}, + publisher = {International Biometric Society}, + title = {Confidence Bounds for the Difference between Two Probabilities}, + urldate = {2026-02-23}, + volume = {40}, + year = {1984} +} + + +@article{koopman1984, + title = {Confidence Intervals for the Ratio of Two Binomial Proportions}, + author = {Koopman, P. A. R.}, + year = {1984}, + month = {06}, + date = {1984-06}, + journal = {Biometrics}, + pages = {513}, + volume = {40}, + number = {2}, + doi = {10.2307/2531405}, + url = {http://dx.doi.org/10.2307/2531405} +} + +@article{tango1998, + title = {Equivalence test and confidence interval for the difference in proportions for the paired-sample design}, + author = {Tango, Toshiro}, + year = {1998}, + month = {04}, + date = {1998-04-30}, + journal = {Statistics in Medicine}, + pages = {891--908}, + volume = {17}, + number = {8}, + doi = {10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19980430)17:8<891::aid-sim780>3.0.co;2-b}, + langid = {en} +} + +@article{tango1999, + title = {Improved confidence intervals for the difference between binomial proportions based on paired data by Robert G. Newcombe,Statistics in Medicine,17, 2635-2650 (1998)}, + author = {Tango, T.}, + year = {1999}, + month = {12}, + date = {1999-12-30}, + journal = {Statistics in Medicine}, + pages = {3511--3513}, + volume = {18}, + number = {24}, + doi = {10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(19991230)18:24<3511::aid-sim303>3.0.co;2-a}, + langid = {en} +} + +@article{tang2003, + title = {On tests of equivalence via non{-}unity relative risk for matched{-}pair design}, + author = {Tang, {Nian{-}Sheng} and Tang, {Man{-}Lai} and Chan, Ivan Siu Fung}, + year = {2003}, + month = {03}, + date = {2003-03-18}, + journal = {Statistics in Medicine}, + pages = {1217--1233}, + volume = {22}, + number = {8}, + doi = {10.1002/sim.1213}, + url = {http://dx.doi.org/10.1002/sim.1213}, + langid = {en} +} + +@article{nam2002, + title = {Analysis of the ratio of marginal probabilities in a matched{-}pair setting}, + author = {Nam, {Jun{-}mo} and Blackwelder, William C.}, + year = {2002}, + month = {02}, + date = {2002-02-19}, + journal = {Statistics in Medicine}, + pages = {689--699}, + volume = {21}, + number = {5}, + doi = {10.1002/sim.1017}, + url = {http://dx.doi.org/10.1002/sim.1017}, + langid = {en} +} + +@article{laud2018, + title = {Equal{-}tailed confidence intervals for comparison of rates}, + author = {Laud, Peter J.}, + year = {2018}, + month = {05}, + date = {2018-05}, + journal = {Pharmaceutical Statistics}, + pages = {290--293}, + volume = {17}, + number = {3}, + doi = {10.1002/pst.1855}, + url = {http://dx.doi.org/10.1002/pst.1855}, + langid = {en} +} + +@article{mehrotra2000, + title = {Minimum risk weights for comparing treatments in stratified binomial trials}, + author = {Mehrotra, Devan V. and Railkar, Radha}, + year = {2000}, + month = {03}, + date = {2000-03-30}, + journal = {Statistics in Medicine}, + pages = {811--825}, + volume = {19}, + number = {6}, + doi = {10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + url = {http://dx.doi.org/10.1002/(sici)1097-0258(20000330)19:6<811::aid-sim390>3.0.co;2-z}, + langid = {en} +} + +@article{tonycai2005, + title = {One-sided confidence intervals in discrete distributions}, + author = {Tony Cai, T.}, + year = {2005}, + month = {04}, + date = {2005-04}, + journal = {Journal of Statistical Planning and Inference}, + pages = {63--88}, + volume = {131}, + number = {1}, + doi = {10.1016/j.jspi.2004.01.005}, + url = {http://dx.doi.org/10.1016/j.jspi.2004.01.005}, + langid = {en} +} + +@article{laud2014, + title = {Confidence intervals for the difference between independent binomial proportions: comparison using a graphical approach and moving averages}, + author = {Laud, Peter J. and Dane, Aaron}, + year = {2014}, + month = {08}, + date = {2014-08-27}, + journal = {Pharmaceutical Statistics}, + pages = {294--308}, + volume = {13}, + number = {5}, + doi = {10.1002/pst.1631}, + url = {http://dx.doi.org/10.1002/pst.1631}, + langid = {en} +} + +@article{bai2021, + title = {Confidence interval of risk difference by different statistical methods and its impact on the study conclusion in antibiotic non-inferiority trials}, + author = {Bai, Anthony D. and Komorowski, Adam S. and Lo, Carson K. L. and Tandon, Pranav and Li, Xena X. and Mokashi, Vaibhav and Cvetkovic, Anna and Findlater, Aidan and Liang, Laurel and Tomlinson, George and Loeb, Mark and Mertz, Dominik and , }, + year = {2021}, + month = {10}, + date = {2021-10-16}, + journal = {Trials}, + volume = {22}, + number = {1}, + doi = {10.1186/s13063-021-05686-8}, + url = {http://dx.doi.org/10.1186/s13063-021-05686-8}, + langid = {en} +} From affd708b772c70b3901823f49c82ff619ebb4a22 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:25:42 +0000 Subject: [PATCH 36/58] expand introduction, and add link to method summary page --- SAS/ci_for_2indep_prop.qmd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 8abfd49a0..88810e82b 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -6,7 +6,9 @@ csl: nature.csl ## Introduction -\[See separate page for general introductory information on confidence intervals for proportions.\] +This page covers confidence intervals for comparisons of two independent proportions in SAS, including the contrast parameters for risk difference $\theta_{RD} = p_1 - p_2$, relative risk $\theta_{RR} = p_1 / p_2$, and odds ratio $\theta_{OR} = p_1(1-p_2) / (p_2(1-p_1))$. + +See the [summary page](../method_summary/ci_for_prop_intro.html) for general introductory information on confidence intervals for proportions, including the principles underlying the most common methods. ## Data used From 521a9e401a0887c75b78b81c71ff150241d5eec7 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:26:55 +0000 Subject: [PATCH 37/58] add SAS code for importing the csv file --- SAS/ci_for_2indep_prop.qmd | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 88810e82b..eb583c15b 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -16,6 +16,14 @@ The adcibc data stored [here](../data/adcibc.csv) was used in this example, crea ```{sas} #| eval: false +proc import datafile = 'data/adcibc.csv' + out = adcibc + dbms = csv + replace; + getnames = yes; + guessingrows = max; +run; + data adcibc2 (keep=trt resp) ; set adcibc; if aval gt 4 then resp="Yes"; From 785c827ec31de755998e69721d902ed5f868dde8 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:28:38 +0000 Subject: [PATCH 38/58] add some references --- SAS/ci_for_2indep_prop.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index eb583c15b..b0fbd65f0 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -51,7 +51,7 @@ knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The performance of many of the same methods are compared graphically here\[add ref\]. According to a recent paper\[\], the most commonly reported method in non-inferiority clinical trials is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). SAS PROC FREQ is able to calculate CIs using the following methods: Exact, Agresti/Caffo (AC), Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, Miettinen and Nurminen (MN or SCORE), Mee (MN(correct=Mee)), and Hauck-Anderson (HA). The SCAS method is not available in PROC FREQ, but a SAS macro is available in a GitHub repository\[\], which also provides an optional continuity correction for both SCAS and MN. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. From 1463683cefa6f81c3107cb5259b800d2fe5573ae Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:33:34 +0000 Subject: [PATCH 39/58] tidy up description of available methods in PROC FREQ, and add example code for all methods --- SAS/ci_for_2indep_prop.qmd | 58 ++++++++++++++++++++++++++++++++++---- 1 file changed, 53 insertions(+), 5 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index b0fbd65f0..84bd1ce71 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -53,19 +53,67 @@ knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). -SAS PROC FREQ is able to calculate CIs using the following methods: Exact, Agresti/Caffo (AC), Wald, Wald with Continuity correction, Newcombe Score, Newcombe score with continuity correction, Miettinen and Nurminen (MN or SCORE), Mee (MN(correct=Mee)), and Hauck-Anderson (HA). The SCAS method is not available in PROC FREQ, but a SAS macro is available in a GitHub repository\[\], which also provides an optional continuity correction for both SCAS and MN. Example code is shown below, before provision of a much more detailed analysis using Wald and Wilson with and without continuity correction. +SAS PROC FREQ is able to calculate CIs using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. Example code is shown below. + +The SCAS method is not available in PROC FREQ, but can be produced using a SAS macro (%SCORECI) which can be downloaded from . ```{sas} #| eval: false -# Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods +*** Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods; proc freq data=adcibc2 order=data; - table trt*resp /riskdiff(CL=(wald newcombe ac ha mn)); + table trt*resp /riskdiff(CL=(wald newcombe ac mn); run; -# exact method + +*** Mee score method; +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(mn(mee)); +run; + +*** exact (Chan-Zhang) and continuity-adjusted methods for conservative coverage; proc freq data=adcibc2 order=data; exact riskdiff (method=noscore); - table trt*resp/riskdiff(CL=(exact)); + table trt*resp/riskdiff(CL=(exact ha wald(correct) newcombe(correct))); run; + +*** exact (Santner-Snell) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=exact); +run; + +*** 2-sided exact (Agresti-Min) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=score2); + table trt*resp/riskdiff(CL=exact); +run; + +*** MN and SCAS methods from %SCORECI macro; +proc tabulate data=adcibc2 out=tab2; + class trt resp; + table (resp all),trt; +run; + +data ds(keep = n1 n0 e1 e0); + set tab2; + by _page_; + retain n1 n0 e1 e0; + if trt = "ACT" then do; + if _type_ = "11" then e1 = n; + if _type_ = "10" then n1 = n; + end; + else if trt = "PBO" then do; + if _type_ = "11" then e0 = n; + if _type_ = "10" then n0 = n; + end; + if last._page_ then output; +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, skew=FALSE); + +*** SCAS CI; +%scoreci(ds); + ``` ### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) From bfd528194f4f9f506f3a098fb378e37a392115f2 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:35:30 +0000 Subject: [PATCH 40/58] improve description of Wald and Newcome methods, and add brief descriptions of the other methods --- SAS/ci_for_2indep_prop.qmd | 48 ++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 84bd1ce71..cede7bf24 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -118,43 +118,51 @@ run; ### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) -In large random samples from independent trials, the sampling distribution of the difference between two proportions approximately follows the normal distribution. +The difference between two independent sample proportions is calculated as: $\hat \theta_{RD} = \hat p_1 - \hat p_2 = x_1 / n_1 - x_2 / n_2$ -The difference between two independent sample proportions is calculated as: $D= \hat p_1-\hat p_2$ +The Wald CI for $\theta_{RD}$ is calculated using: -A confidence interval for the difference between two independent proportions $D$ can be calculated using: +$\hat \theta_{RD} \pm z_{\alpha/2} \times SE(\hat \theta_{RD})$, -$D\approx \hat D \pm z_\alpha * SE(\hat p)$, +where $SE (\hat \theta_{RD}) = \sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$, and +With continuity correction, the equation becomes -$SE (\hat p) = sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$ +$\hat \theta_{RD} \pm (CC + z_{\alpha/2} \times SE(\hat \theta_{RD}))$, -With continuity correction, the equation becomes +where $CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ + +### Newcombe Method (Also known as the Hybrid Score method, Square-and-Add, or the Method of Variance Estimates Recovery (MOVER) ) + +Derive the confidence intervals for the separate proportions in each group, $p_1$ and $p_2$, using the Wilson Score Method equations as described [here](ci_for_prop.html). + +Let $l_1$ = Lower CI for sample 1, and $u_1$ be the upper CI for sample 1. + +Let $l_2$ = Lower CI for sample 2, and $u_2$ be the upper CI for sample 2. + +Let D = $\hat p_1 - \hat p_2$ (the difference between the observed proportions) -$D\approx \hat D \pm (CC + z_\alpha * SE(\hat p))$, +The CI for $\theta_{RD}$ the difference between two proportions is: $$ D - sqrt((\hat p_1 - l_1)^2+(u_2 - \hat p_2)^2) \quad, \quad D + sqrt((\hat p_2 - l_2)^2 + (u_1 - \hat p_1)^2 ) $$ -where $z_\alpha$ is the $1-\alpha/2$ quantile of a standard normal distribution corresponding to level $\alpha$,\ -and $SE (\hat p)$ = $sqrt{( \frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2})}$\ -and +Note that earlier versions of SAS PROC FREQ (before implementation of the MN method) allowed the option CL=WILSON or CL=SCORE to produce this method, but it is not really a score method. As of SAS/STAT 15.4, the CL=WILSON option (undocumented) still gives the Newcombe interval, but CL=SCORE gives the Miettinen-Nurminen score method. -$CC = \frac{1}{2} (\frac{1}{n_1} + \frac{1}{n_2})$ +### Miettinen-Nurminen, Mee and other Asymptotic Score Methods -### Newcombe Method (Also known as the Hybrid Score method or the Method of Variance Estimates Recovery (MOVER) ) +These truly are score methods, as they are based on an extension of the score methodology applied to two independent proportions, using the contrast function $S(\theta) = \hat p_1 - \hat p_2 - \theta$, which for any given value of $\theta$ has expectation zero, and variance $\{\tilde p_1 (1 - \tilde p_1)/n_1 + \tilde p_2 (1 - \tilde p_2)/n_2\} \times N/(N+1)$ where $N = n_1 + n_2$. -Derive the confidence intervals for the separate proportions in each group, p1 and p2, using the Wilson Method equations \***above**. +The Mee variant of the method omits the $N/(N-1)$ variance bias correction factor. -Let l1 = Lower CI for sample 1, and u1 be the upper CI for sample 1. +Gart and Nam derived a similar method, arrived at from a different underlying 'efficient score' methodology, so the formulae look different but are essentially equivalent to the Mee interval. They added a correction for skewness to improve one-sided coverage. The skewness correction was applied to the Miettinen-Nurminen formula for all contrast parameters by Laud, to give the SCAS method[@laud2017]. The SCAS method is available for SAS via the %SCORECI macro. -Let l2 = Lower CI for sample 2, and u2 be the upper CI for sample 2. +### Agresti-Caffo Method -Let D = p1-p2 (the difference between the observed proportions) +Similar to the Agresti-Coull method for a single proportion, the Agresti-Caffo interval is designed to be an easily taught method to enable the CI to be calculated by hand. The formula involves simply adding one success and one failure to each sample, and then using the Wald formula. -The CI for the difference between two proportions is: $$ D - sqrt((p_1-l_1)^2+(u_2-p_2)^2) \quad to\quad D + sqrt((p_2-l_2)^2+(u_1-p_1)^2 ) $$ +### 'Exact' Methods -### Miettinen-Nurminen Asymptotic Score Method +The current default 'exact' method produced by PROC FREQ is the Chan-Zhang variant. This is undoubtedly an improvement on the extremely over-conservative Santner-Snell method, but is more computationally intensive, and can result in a Warning in the SAS log about long computation times. -\[Details to be added, noting options for skewness correction offered by the %SCORECI macro\] +The third alternative 'exact' method by Agresti & Min is less conservative, but is only guaranteed to achieve strictly conservative two-sided coverage, so is not appropriate for use in one-sided hypothesis testing. ## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples From fa586bd419b3cb4b6badd0457bd6ca313bf46b63 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:36:29 +0000 Subject: [PATCH 41/58] expand section on continuity adjustments --- SAS/ci_for_2indep_prop.qmd | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index cede7bf24..946545721 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -176,9 +176,11 @@ SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nu ## Continuity Adjusted Methods -SAS provides an option (CORRECT) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage. Note however that the Wald-cc method fails to achieve strictly conservative coverage \[See Laud & Dane 2014\]. +SAS provides an option (CORRECT) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage. The Hauck-Anderson (HA) method for RD is a slightly less conservative variation of a continuity adjustment. Note however that all of these methods fail to achieve strictly conservative coverage, although the adjusted Newcombe method comes close[@laud2014]. -It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, CORRECT=NO removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and an equivalent un-corrected score method for RR and OR). No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described \[Laud & Dane 2014\] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. +It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, the CORRECT=NO option removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and an equivalent un-corrected score method for RR and OR). + +No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described[@laud2014] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. ## Consistency with Hypothesis Tests From f84a0fc22c2af9fe414729ef714c6c41c687cb1b Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 10:56:16 +0000 Subject: [PATCH 42/58] tidy up references --- SAS/ci_for_2indep_prop.qmd | 28 ++-------------------------- SAS/ci_for_prop.qmd | 4 +--- method_summary/ci_for_prop_intro.qmd | 2 ++ 3 files changed, 5 insertions(+), 29 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 946545721..da9336028 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -51,7 +51,7 @@ knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf)^12^ describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf) describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). SAS PROC FREQ is able to calculate CIs using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. Example code is shown below. @@ -229,28 +229,4 @@ run; knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") ``` -## Reference - -1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) - -2. [Five Confidence Intervals for Proportions That You Should Know about](https://towardsdatascience.com/five-confidence-intervals-for-proportions-that-you-should-know-about-7ff5484c024f) - -3. [Confidence intervals for Binomial Proportion Using SAS](https://www.lexjansen.com/sesug/2015/103_Final_PDF.pdf) - -4. Brown LD, Cai TT, DasGupta A (2001). "Interval estimation for a binomial proportion", Statistical Science 16(2):101-133 - -5. Newcombe RG (1998). "Two-sided confidence intervals for the single proportion: comparison of seven methods", Statistics in Medicine 17(8):857-872 - -6. Clopper,C.J.,and Pearson,E.S.(1934),"The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial", Biometrika 26, 404--413. - -7. D. Altman, D. Machin, T. Bryant, M. Gardner (eds). Statistics with Confidence: Confidence Intervals and Statistical Guidelines, 2nd edition. John Wiley and Sons 2000. - -8. Laud PJ (2017) Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 16: 334-348 - -9. Laud PJ (2018) Corrigendum: Equal-tailed confidence intervals for comparison of rates. Pharmaceutical Statistics 17: 290-293 - -10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 - -11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. - -12. +## References diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 6de961560..0afbead0c 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -236,7 +236,7 @@ knitr::include_graphics("../images/ci_for_prop/binomial_prop_cc_act.png") SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. `ods output binomialcls=bcl;` and then using the bcl dataset, in a data step to put the variable out to the number of decimal places we require.\ 10 decimal places shown here ! `lowercl2=put(lowercl,12.10);` -## Reference +## References 1. [SAS PROC FREQ here](https://support.sas.com/documentation/cdl/en/procstat/63104/HTML/default/viewer.htm#procstat_freq_sect010.htm) and [here](https://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_freq_sect028.htm) @@ -259,5 +259,3 @@ SAS output often rounds to 3 or 4 decimal places in the output window, however t 10. Blaker, H. (2000). Confidence curves and improved exact confidence intervals for discrete distributions, Canadian Journal of Statistics 28 (4), 783--798 11. Klaschka, J. and Reiczigel, J. (2021). "On matching confidence intervals and tests for some discrete distributions: Methodological and computational aspects," Computational Statistics, 36, 1775--1790. - -12. diff --git a/method_summary/ci_for_prop_intro.qmd b/method_summary/ci_for_prop_intro.qmd index 62f655bf8..d79b98db0 100644 --- a/method_summary/ci_for_prop_intro.qmd +++ b/method_summary/ci_for_prop_intro.qmd @@ -87,3 +87,5 @@ Several asymptotic methods offer a continuity adjustment, which aims to emulate 'Exact' methods may be adapted to be less conservative (i.e. to achieve proximate coverage) by applying a mid-P adjustment, essentially including half the probability of the observed data in the calculations. This is like a reversal of the continuity adjustment for asymptotic methods. Although theoretically possible for any contrast, the mid-P method is only implemented in SAS for CIs for the single proportion and the odds ratio contrast. + +## References From 2d75571011382048385cb96d6b36acb681a15549 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:40:19 +0000 Subject: [PATCH 43/58] expand intro with warning for situations when SAS fails to produce an estimate. --- SAS/ci_for_2indep_prop.qmd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index da9336028..91f622dc3 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -6,10 +6,12 @@ csl: nature.csl ## Introduction -This page covers confidence intervals for comparisons of two independent proportions in SAS, including the contrast parameters for risk difference $\theta_{RD} = p_1 - p_2$, relative risk $\theta_{RR} = p_1 / p_2$, and odds ratio $\theta_{OR} = p_1(1-p_2) / (p_2(1-p_1))$. +This page covers confidence intervals for comparisons of two independent proportions in SAS, including the contrast parameters for risk difference (RD) $\theta_{RD} = p_1 - p_2$, relative risk (RR) $\theta_{RR} = p_1 / p_2$, and odds ratio (OR) $\theta_{OR} = p_1(1-p_2) / (p_2(1-p_1))$. See the [summary page](../method_summary/ci_for_prop_intro.html) for general introductory information on confidence intervals for proportions, including the principles underlying the most common methods. +Caution is required if there are no responders (or all responders) in both groups, which might happen in a subgroup analysis for example. PROC FREQ (as of v9.4) does not output any confidence intervals in this case, when valid CIs can (and should) be reported for the RD contrast, since the dataset provides an estimate of zero for RD (and the confidence in the estimate is proportional to the sample size). Similarly, if $\hat p_1 = \hat p_2 = 1$ then an estimate and confidence interval can be obtained for RR, but not from PROC FREQ. + ## Data used The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. From f512e98d24893fb6e342cbc052b17379b0fdfdac Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:40:58 +0000 Subject: [PATCH 44/58] add sort step in code to facilitate later examples --- SAS/ci_for_2indep_prop.qmd | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 91f622dc3..d1f090977 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -33,9 +33,15 @@ data adcibc2 (keep=trt resp) ; if trtp="Placebo" then trt="PBO"; else trt="Act"; run; + +* Sort to ensure that the outcome of interest ("Yes" in this example) is first; +* when using default COLUMN=1 option in the TABLES statement; +proc sort data=adcibc2; +by trt descending resp; +run; ``` -The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558. +The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558, giving a risk difference of 0.0779, relative risk 1.50, and odds ratio 1.6525. ```{sas} #| eval: false From 7e34beb76e66449bc69c992c4b032701771100f1 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:42:54 +0000 Subject: [PATCH 45/58] move example code to end section --- SAS/ci_for_2indep_prop.qmd | 203 +++++++++++++++++++++++-------------- 1 file changed, 128 insertions(+), 75 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index d1f090977..5fa2f17a2 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -59,70 +59,10 @@ knitr::include_graphics("../images/ci_for_prop/2by2crosstab.png") ## Methods for Calculating Confidence Intervals for Proportion Difference from 2 independent samples -This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf) describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast). +This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf) describes many methods for the calculation of confidence intervals for 2 independent proportions. The 2-sided and 1-sided performance of many of the same methods have been compared graphically[@laud2014]. According to a recent paper[@bai2021], the most commonly reported method in non-inferiority clinical trials for antibiotics is the Wald asymptotic normal approximation (despite its well-documented poor performance), followed by the Miettinen-Nurminen (asymptotic score) method. More recently, an improved variant of the Miettinen-Nurminen method (SCAS) was developed, by including a skewness correction designed to optimise the performance in terms of one-sided coverage for NI testing. SCAS corrects the slightly asymmetrical coverage of the Miettinen-Nurminen interval (note the skewness is more pronounced when analysing the RR contrast, and/or when group sizes are imbalanced). -SAS PROC FREQ is able to calculate CIs using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. Example code is shown below. +SAS PROC FREQ is able to calculate CIs for RD using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. -The SCAS method is not available in PROC FREQ, but can be produced using a SAS macro (%SCORECI) which can be downloaded from . - -```{sas} -#| eval: false -*** Wald, Wilson, Agresti/Caffo, Hauck-Anderson, and Miettinen and Nurminen methods; -proc freq data=adcibc2 order=data; - table trt*resp /riskdiff(CL=(wald newcombe ac mn); -run; - -*** Mee score method; -proc freq data=adcibc2 order=data; - table trt*resp /riskdiff(CL=(mn(mee)); -run; - -*** exact (Chan-Zhang) and continuity-adjusted methods for conservative coverage; -proc freq data=adcibc2 order=data; - exact riskdiff (method=noscore); - table trt*resp/riskdiff(CL=(exact ha wald(correct) newcombe(correct))); -run; - -*** exact (Santner-Snell) method; -proc freq data=adcibc2 order=data; - exact riskdiff (method=noscore); - table trt*resp/riskdiff(CL=exact); -run; - -*** 2-sided exact (Agresti-Min) method; -proc freq data=adcibc2 order=data; - exact riskdiff (method=score2); - table trt*resp/riskdiff(CL=exact); -run; - -*** MN and SCAS methods from %SCORECI macro; -proc tabulate data=adcibc2 out=tab2; - class trt resp; - table (resp all),trt; -run; - -data ds(keep = n1 n0 e1 e0); - set tab2; - by _page_; - retain n1 n0 e1 e0; - if trt = "ACT" then do; - if _type_ = "11" then e1 = n; - if _type_ = "10" then n1 = n; - end; - else if trt = "PBO" then do; - if _type_ = "11" then e0 = n; - if _type_ = "10" then n0 = n; - end; - if last._page_ then output; -run; - -*** Miettinen-Nurminen CI; -%scoreci(ds, skew=FALSE); - -*** SCAS CI; -%scoreci(ds); - -``` ### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) @@ -208,26 +148,139 @@ Options for riskdiff(CL=XXX) consist of AC: Agresti-Caffo, EXACT: exact, HA: Hau ```{sas} #| eval: false -proc sort data=adcibc2; -by trt descending resp; + +****************************; +*** Risk Difference examples; +****************************; + +*** Wald, Newcombe, Agresti/Caffo, and Miettinen-Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(wald newcombe ac mn); run; -# without continuity correction -proc freq data=adcibc2 order=data; -table trt*resp/riskdiff(CL=(wald wilson)); +*** Mee score method; +proc freq data=adcibc2 order=data; + table trt*resp /riskdiff(CL=(mn(mee)); run; -# with continuity correction +*** exact (Chan-Zhang) and continuity-adjusted methods for conservative coverage; proc freq data=adcibc2 order=data; -table trt*resp/riskdiff(CORRECT CL=(wald wilson)); + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=(exact ha wald(correct) newcombe(correct))); run; -``` -```{r} -#| echo: false -#| fig-align: center -#| out-width: 50% - knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") +*** exact (Santner-Snell) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=noscore); + table trt*resp/riskdiff(CL=exact); +run; + +*** 2-sided exact (Agresti-Min) method; +proc freq data=adcibc2 order=data; + exact riskdiff (method=score2); + table trt*resp/riskdiff(CL=exact); +run; + +*** MN and SCAS methods from %SCORECI macro; +*** First manipulate the data to a form for input to the macro; +proc tabulate data=adcibc2 out=tab2; + class trt resp; + table (resp all),trt; +run; +data ds(keep = n1 n0 e1 e0); + set tab2; + by _page_; + retain n1 n0 e1 e0; + * Initialise counts in case there are none in the observed data; + if first._page_ then do; + e1 = 0; + e0 = 0; + end; + if upcase(trt) = "ACT" then do; + if _type_ = "11" and resp = "Yes" then e1 = n; + if _type_ = "10" then n1 = n; + end; + else if upcase(trt) = "PBO" then do; + if _type_ = "11" and resp = "Yes" then e0 = n; + if _type_ = "10" then n0 = n; + end; + if last._page_ then output; +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE); + +*** SCAS CI; +%scoreci(ds, stratify=FALSE); + +*** Farrington-Manning NI test (p=0.0248) contradicts MN interval (LCL < -0.036); +*** (Arbitrary NI margin of -0.036 used for illustration) +proc freq data=adcibc2 order=data; + table trt*resp / riskdiff(CL=mn noninf margin=0.036 method=score); +run; + +*** Miettinen-Nurminen CI with consistent NI test: PVAL_R > 0.025; +%scoreci(ds, stratify=FALSE, skew=FALSE, delta=-0.036); + + +****************************; +*** Relative risk examples; +****************************; + +*** Wald, LR, and Miettinen and Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /relrisk(CL=(wald waldmodified lr score)); +run; +*** MN without the correction; +proc freq data=adcibc2 order=data; + table trt*resp /relrisk(CL=(score(correct=no))); +run; +*** exact method; +proc freq data=adcibc2 order=data; + exact relrisk; + table trt*resp/relrisk(CL=(exact)); +run; +*** santner-snell; +proc freq data=adcibc2 order=data; + exact relrisk (method=noscore); + table trt*resp / relrisk(CL=exact); +run; + +*** Agresti-Min; +proc freq data=adcibc2 order=data; + exact relrisk (method=score2); + table trt*resp / relrisk(CL=exact); +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE, contrast=RR); +*** SCAS CI; +%scoreci(ds, stratify=FALSE, contrast=RR); + + +****************************; +*** Odds Ratio examples; +****************************; + + +* Wald, LR, mid-P and Miettinen and Nurminen methods; +proc freq data=adcibc2 order=data; + table trt*resp /or(CL=(wald waldmodified lr score midp)); +run; +* MN without the correction; +proc freq data=adcibc2 order=data; + table trt*resp /or(CL=(score(correct=no))); +run; +*** 'exact' method; +proc freq data=adcibc2 order=data; + exact or; + table trt*resp/or(CL=(exact)); +run; + +*** Miettinen-Nurminen CI; +%scoreci(ds, stratify=FALSE, skew=FALSE, orbias=FALSE, contrast=OR); +*** SCAS CI; +%scoreci(ds, stratify=FALSE, contrast=OR); ``` ```{r} From b2412bda52b5dfe595fdbb403529bb739a7626b6 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:44:57 +0000 Subject: [PATCH 46/58] add further description for score methods --- SAS/ci_for_2indep_prop.qmd | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 5fa2f17a2..24e525e59 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -63,6 +63,7 @@ This [paper](https://www.lexjansen.com/wuss/2016/127_Final_Paper_PDF.pdf) descri SAS PROC FREQ is able to calculate CIs for RD using the following methods: Agresti/Caffo (AC), Miettinen and Nurminen (MN or SCORE), Mee (MN(Mee)), Newcombe Hybrid Score, and Wald. For conservative coverage, there is the 'Exact' method, or continuity-adjusted versions of the Wald and Newcombe methods, and also the Hauck-Anderson (HA) continuity-adjustment. +The SCAS method is not available in PROC FREQ, but can be produced using a SAS macro (`%SCORECI`) which can be downloaded from . ### Normal Approximation Method (Also known as the Wald or asymptotic CI Method) @@ -96,11 +97,11 @@ Note that earlier versions of SAS PROC FREQ (before implementation of the MN met ### Miettinen-Nurminen, Mee and other Asymptotic Score Methods -These truly are score methods, as they are based on an extension of the score methodology applied to two independent proportions, using the contrast function $S(\theta) = \hat p_1 - \hat p_2 - \theta$, which for any given value of $\theta$ has expectation zero, and variance $\{\tilde p_1 (1 - \tilde p_1)/n_1 + \tilde p_2 (1 - \tilde p_2)/n_2\} \times N/(N+1)$ where $N = n_1 + n_2$. +These truly are score methods, as they are based on an extension of the score methodology applied to two independent proportions, using the contrast function $S(\theta) = \hat p_1 - \hat p_2 - \theta$, which for any given value of $\theta$ has expectation zero, and variance $\{\tilde p_1 (1 - \tilde p_1)/n_1 + \tilde p_2 (1 - \tilde p_2)/n_2\} \times N/(N-1)$ where $N = n_1 + n_2$. -The Mee variant of the method omits the $N/(N-1)$ variance bias correction factor. +The Mee variant of the method omits the $N/(N-1)$ variance bias correction factor (sometimes referred to as an $'N-1'$ correction). -Gart and Nam derived a similar method, arrived at from a different underlying 'efficient score' methodology, so the formulae look different but are essentially equivalent to the Mee interval. They added a correction for skewness to improve one-sided coverage. The skewness correction was applied to the Miettinen-Nurminen formula for all contrast parameters by Laud, to give the SCAS method[@laud2017]. The SCAS method is available for SAS via the %SCORECI macro. +Gart and Nam derived a similar method, arrived at from a different underlying 'efficient score' methodology, so the formulae look different but are essentially equivalent to the Mee interval. They added a correction for skewness to improve one-sided coverage. The skewness correction was applied to the Miettinen-Nurminen formula for all contrast parameters by Laud, to give the SCAS method[@laud2017]. The SCAS method is available for SAS via the `%SCORECI` macro. ### Agresti-Caffo Method From f82e2ac11d1dbb4e8fe75a2867847c0289b92f33 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:45:44 +0000 Subject: [PATCH 47/58] improve description of 'exact' methods --- SAS/ci_for_2indep_prop.qmd | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 24e525e59..97901019d 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -109,9 +109,9 @@ Similar to the Agresti-Coull method for a single proportion, the Agresti-Caffo i ### 'Exact' Methods -The current default 'exact' method produced by PROC FREQ is the Chan-Zhang variant. This is undoubtedly an improvement on the extremely over-conservative Santner-Snell method, but is more computationally intensive, and can result in a Warning in the SAS log about long computation times. +To obtain 'exact' confidence intervals, you need to add an `EXACT RISKDIFF` statement to the PROC FREQ. -The third alternative 'exact' method by Agresti & Min is less conservative, but is only guaranteed to achieve strictly conservative two-sided coverage, so is not appropriate for use in one-sided hypothesis testing. +The current default method produced by PROC FREQ (`METHOD=SCORE`) is the Chan-Zhang variant. This is undoubtedly an improvement on the extremely over-conservative Santner-Snell method (obtained with the `RISKDIFF(METHOD=NOSCORE)` option in the EXACT statement), but is more computationally intensive, and can result in a Warning in the SAS log about long computation times. ## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples From 3926be66547f74411582c9e760ec8301039d61fa Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:46:40 +0000 Subject: [PATCH 48/58] combine sections for RR and RR, since SAS provides a similar set of methods for each --- SAS/ci_for_2indep_prop.qmd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 97901019d..81437407d 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -113,15 +113,15 @@ To obtain 'exact' confidence intervals, you need to add an `EXACT RISKDIFF` stat The current default method produced by PROC FREQ (`METHOD=SCORE`) is the Chan-Zhang variant. This is undoubtedly an improvement on the extremely over-conservative Santner-Snell method (obtained with the `RISKDIFF(METHOD=NOSCORE)` option in the EXACT statement), but is more computationally intensive, and can result in a Warning in the SAS log about long computation times. -## Methods for Calculating Confidence Intervals for Relative Risk from 2 independent samples +The third alternative 'exact' method by Agresti & Min (`RISKDIFF(METHOD=SCORE2)`) is less conservative, but is only guaranteed to achieve strictly conservative two-sided coverage, so is not appropriate for use in one-sided hypothesis testing. -SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nurminen (SCORE), Exact, Likelihood Ratio (LR), Wald, Haldane Modified Wald. The SCAS method (which addresses asymmetric one-sided coverage of the MN method which is particularly pronounced for RR) is not available in PROC FREQ, but is in the SAS macro %SCORECI. +## Methods for Calculating Confidence Intervals for Relative Risk or Odds Ratio from 2 independent samples -## Methods for Calculating Confidence Intervals for Odds Ratio from 2 independent samples +SAS PROC FREQ is able to calculate CIs for RR or OR using the following methods: Miettinen-Nurminen (SCORE), Likelihood Ratio (LR), Wald, Haldane Modified Wald. There is also a modified version of the MN method which omits the 'N-1' variance bias correction, using the `CL=(SCORE(CORRECT=NO))` option - note this is NOT referring to a 'continuity correction' (and also note that no indication is given in the output that this option has been applied). For OR, an additional `CL=MIDP` option is also available. -SAS PROC FREQ is able to calculate CIs using the following methods: Miettinen-Nurminen (SCORE), Exact, Likelihood Ratio (LR), mid-P (MIDP), Wald, Haldane Modified Wald. The SCAS method is not available in PROC FREQ, but is in the SAS macro %SCORECI. +For conservative coverage, the 'Exact' methods are provided. For RR this includes the 3 variants described above for RD, while only one version is given for OR. Run times for the `METHOD=SCORE` version for RR may be lengthy (but the faster alternative Santner-Snell version produces a fairly uninterpretable interval of \[0.03, 23698\] for the example dataset used in this article). Continuity adjustments are not implemented for any method for RR or OR. -### +The SCAS method (which addresses asymmetric one-sided coverage of the MN Score method which is particularly pronounced for RR, and also adds a further bias correction for OR[@laud2018]) is not available in PROC FREQ, but is given by the SAS macro `%SCORECI`. An optional continuity adjustment[@laud2017] for both MN and SCAS is provided in the R package, but not yet implemented in the macro. ## Continuity Adjusted Methods From a16922be605e259b938cb264127e306eec724fd6 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:47:46 +0000 Subject: [PATCH 49/58] minor updates to content and formatting of continuity adjusted section --- SAS/ci_for_2indep_prop.qmd | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 81437407d..78e8d8490 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -125,11 +125,11 @@ The SCAS method (which addresses asymmetric one-sided coverage of the MN Score m ## Continuity Adjusted Methods -SAS provides an option (CORRECT) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage. The Hauck-Anderson (HA) method for RD is a slightly less conservative variation of a continuity adjustment. Note however that all of these methods fail to achieve strictly conservative coverage, although the adjusted Newcombe method comes close[@laud2014]. +SAS provides an option (`CORRECT`) to apply continuity adjustment to the Wald or Newcombe methods for more conservative coverage, but this only applies for the RISKDIFF contrast, not RELRISK or ODDSRATIO. The Hauck-Anderson (HA) method for RD is a slightly less conservative variation of a continuity adjustment. Note however that all of these methods fail to achieve strictly conservative coverage, although the adjusted Newcombe method comes close[@laud2014]. -It is important to note that the CORRECT sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, the CORRECT=NO option removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and an equivalent un-corrected score method for RR and OR). +It is important to note that the `CORRECT` sub-option for the MN/Score method serves an entirely different purpose. The Miettinen-Nurminen method is **not** a 'continuity-corrected' version of the Mee interval. Rather, the `CORRECT=NO` option removes the variance bias correction factor N/(N-1) from the Miettinen-Nurminen formula in order to produce the Mee version of the score method for RD (and equivalent un-corrected score methods for RR and OR). -No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described[@laud2014] and implemented in the ratesci package for R, but not yet added to the %SCORECI macro. +No continuity adjustment is currently available for the score methods in SAS. A 'sliding scale' adjustment has been described[@laud2014] and implemented for MN and SCAS in the ratesci package for R, but not yet added to the %SCORECI macro. ## Consistency with Hypothesis Tests From e6279a8346a3c60cae53f90a14d89e0286c1cd69 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:48:51 +0000 Subject: [PATCH 50/58] updates to section on consistency with hypothesis tests --- SAS/ci_for_2indep_prop.qmd | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 78e8d8490..8dd93a1b7 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -133,13 +133,16 @@ No continuity adjustment is currently available for the score methods in SAS. A ## Consistency with Hypothesis Tests -Within SAS PROC FREQ for the asymptotic methods for RD, consistency with a traditional Chi-squared test for superiority (Karl Pearson version as produced by PROC FREQ), and the Farrington-Manning test for non-inferiority, is provided by the Mee CI (CL=SCORE(CORRECT=MEE)), which is similar to MN but omitting the N/(N-1) correction factor. Note that the MN method (including the correction factor) is consistent with the Egon Pearson 'N-1' version of the chi-squared test. SAS PROC FREQ does not produce that, nor does it offer the option to include the 'N-1' adjustment when requesting a non-inferiority test with the NONINF option. Consequently, there is a risk of contradictory results if using PROC FREQ to obtain a MN CI with a corresponding non-inferiority test. +Within SAS PROC FREQ for the asymptotic methods for RD, consistency with a traditional Chi-squared test for association (**Karl** Pearson version, as produced by PROC FREQ), and the Farrington-Manning test for non-inferiority, is provided by the Mee CI (`CL=SCORE(CORRECT=MEE)`). That method is similar to MN but without the 'N-1' correction factor (the omission of which produces coverage that is slightly below the nominal confidence level on average). Note that the MN method (including the correction factor) is consistent with the **Egon** Pearson 'N-1' version of the chi-squared test[@campbell2007]. SAS PROC FREQ does not produce that test, nor does it offer the option to include the 'N-1' adjustment when requesting a non-inferiority test with the `NONINF` option. Consequently, there is a risk of contradictory results if using PROC FREQ to obtain a MN CI with a corresponding non-inferiority test. -For example... +For the SCAS or MN method, the %SCORECI macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. -For the SCAS method, the macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. +```{sas} +#| eval: false + +``` -If an EXACT statement is used... +If an EXACT statement is used to produce CIs, SAS does not offer any matching hypothesis tests for NI or equivalence testing. ## Example Code using PROC FREQ From 9cde348458a7212100f4c508ead90751776e7e47 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:50:39 +0000 Subject: [PATCH 51/58] update text around example code --- SAS/ci_for_2indep_prop.qmd | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 8dd93a1b7..91126754b 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -146,9 +146,9 @@ If an EXACT statement is used to produce CIs, SAS does not offer any matching hy ## Example Code using PROC FREQ -It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN= Option` on riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". Note how in the SAS output, it calls the requested 'wilson' method 'Newcombe' in the output. +It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN=` Option in riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". -Options for riskdiff(CL=XXX) consist of AC: Agresti-Caffo, EXACT: exact, HA: Hauck-Anderson, MN or SCORE: Miettinen-Nurminen (another type of Score CI), SCORE(CORRECT=MEE): Mee (Score CI omitting variance bias correction), WILSON or NEWCOMBE: Newcombe hybrid score method described above, and WALD: normal approximation Wald method described above. Examples using Wald and Wilson are shown below with and without continuity correction. +Similarly for relrisk, although the output does not state that the RR is calculated as (Row 1) / (Row 2). If treatment groups are labelled differently, you might need to sort by descending trt to obtain the correct contrast (note that unlike for RD, the same result cannot be obtained by setting `COLUMN=2`) ```{sas} #| eval: false @@ -287,11 +287,13 @@ run; %scoreci(ds, stratify=FALSE, contrast=OR); ``` +\[Selected output screenshots to be added to the below\] + ```{r} #| echo: false #| fig-align: center #| out-width: 50% -knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_CC.png") +knitr::include_graphics("../images/ci_for_prop/binomial_2sampleCI_noCC.png") ``` ## References From 0fce38a3a7890abfdb0a379c460a5ee3dd0eb88e Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 14:55:28 +0000 Subject: [PATCH 52/58] update ci_for_prop table entries --- data/stat_method_tbl.csv | 138 +++++++++++++++++++-------------------- 1 file changed, 69 insertions(+), 69 deletions(-) diff --git a/data/stat_method_tbl.csv b/data/stat_method_tbl.csv index 2c1f0a8e0..070a281b6 100644 --- a/data/stat_method_tbl.csv +++ b/data/stat_method_tbl.csv @@ -1,69 +1,69 @@ -method_grp,method_subgrp,r_links,sas_links,python_links,comparison_links -Summary Statistics,Rounding,[R](R/rounding),[SAS](SAS/rounding),[Python](python/Rounding),[R vs SAS](Comp/r-sas_rounding) -Summary Statistics,Summary statistics,[R](R/summary-stats),[SAS](SAS/summary-stats),[Python](python/Summary_statistics),[R vs SAS](Comp/r-sas-summary-stats) -Summary Statistics,Skewness/Kurtosis,[R](R/summary_skew_kurt),[SAS](SAS/summary_skew_kurt),[Python](python/skewness_kurtosis),[R vs SAS](Comp/r-sas_summary_skew_kurt) -General Linear Models,One-sample t-test,[R](R/ttest_1Sample),[SAS](SAS/ttest_1Sample),[Python](python/one_sample_t_test),[R vs SAS](Comp/r-sas_ttest_1Sample) -General Linear Models,Paired t-test,[R](R/ttest_Paired),[SAS](SAS/ttest_Paired),[Python](python/paired_t_test),[R vs SAS](Comp/r-sas_ttest_Paired) -General Linear Models,Two-sample t-test,[R](R/ttest_2Sample),[SAS](SAS/ttest_2Sample),[Python](python/two_samples_t_test),[R vs SAS](Comp/r-sas_ttest_2Sample) -General Linear Models,ANOVA,[R](R/anova),[SAS](SAS/anova),[Python](python/anova),[R vs SAS](Comp/r-sas_anova) -General Linear Models,ANCOVA,[R](R/ancova),[SAS](SAS/ancova),[Python](python/ancova),[R vs SAS](Comp/r-sas_ancova) -General Linear Models,MANOVA,[R](R/manova),[SAS](SAS/manova),[Python](python/MANOVA),[R vs SAS](Comp/r-sas_manova) -General Linear Models,Linear regression,[R](R/linear-regression),[SAS](SAS/linear-regression),[Python](python/linear_regression),[R vs SAS](Comp/r-sas_linear-regression) -Generalized Linear Models,Logistic regression,[R](R/logistic_regr),[SAS](SAS/logistic-regr),[Python](python/logistic_regression),[R vs SAS](Comp/r-sas_logistic-regr) -Generalized Linear Models,Poisson/negative binomial regression,[R](R/count_data_regression),[SAS](SAS/count_data_regression),,[R vs SAS](Comp/r-sas_negbin) -Non-Parametric Analysis,Wilcoxon signed-rank test,[R](R/wilcoxonsr_hodges_lehman),[SAS/ StatXact](SAS/wilcoxonsr_HL),,[R vs SAS](Comp/r-sas-wilcoxonsr_HL) -Non-Parametric Analysis,Mann-Whitney U/Wilcoxon rank-sum test,[R](R/nonpara_wilcoxon_ranksum),[SAS](SAS/ranksum),,[R vs SAS](Comp/r-sas-wilcoxon-ranksum_hl) -Non-Parametric Analysis,Kolmogorov-Smirnov test,[R](R/kolmogorov-smirnov_test),,, -Non-Parametric Analysis,Kruskal-Wallis test,[R](R/kruskal_wallis),[SAS](SAS/kruskal_wallis),[Python](python/kruskal_wallis),[R vs SAS](Comp/r-sas_kruskalwallis) -Non-Parametric Analysis,Friedman test,[R](R/friedman_test),[SAS](SAS/SAS_Friedmantest),,[R vs SAS](Comp/r-sas_friedman) -Non-Parametric Analysis,Jonckheere-Terpstra test,[R](R/jonckheere),[SAS](SAS/jonchkheere_terpstra),,[R vs SAS](Comp/r-sas_jonckheere) -Non-Parametric Analysis,Hodges-Lehman estimator,[R](R/nparestimate),[SAS](SAS/nparestimate),, -Categorical Data Analysis,Single proportion - Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), -Categorical Data Analysis,Two paired proportions - McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) -Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, -Categorical Data Analysis,Two independent proportions - Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) -Categorical Data Analysis,Stratified tables - Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) -Categorical Data Analysis,Intro to CIs for proportions,,,,[Summary](method_summary/ci_for_prop_intro) -Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Two Independent Proportions ,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Two Paired Proportions,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) -Categorical Data Analysis,CIs - Proportions in Stratified Designs,,,, -Categorical Data Analysis,CIs - Poisson Exposure Adjusted Incidence Rates,,,, -Correlated Data Analysis,Linear Mixed Model Repeated Measure (MMRM),[R](R/mmrm),[SAS](SAS/mmrm),,[R vs SAS](Comp/r-sas_mmrm) -Correlated Data Analysis,Linear Mixed Model Random Effect,,,,[R vs SAS](Comp/Sas_vs_R_Mixed_Model) -Correlated Data Analysis,Generalized Linear Mixed Model (GLMM),,,, -Correlated Data Analysis,Generalized Estimating Equation (GEE),[R](R/gee),[SAS](SAS/gee),,[R vs SAS](Comp/r-sas_gee) -Multiple Imputation - Continuous Data MAR,Linear regression imputation,[R](R/mi_mar_regression),[SAS](SAS/mi_mar_regression),, -Multiple Imputation - Continuous Data MAR,Predictive mean matching,[R](R/mi_mar_predictive_mean_match),,, -Multiple Imputation - Continuous Data MNAR,Tipping point analysis (delta adjustment),[R](R/tipping_point),[SAS](SAS/tipping_point),,[R vs SAS](Comp/r-sas_tipping_point) -Multiple Imputation - Continuous Data MNAR,Reference-based imputation/joint modeling,[R](R/rbmi_continuous_joint),[SAS](SAS/rbmi_continuous_joint_SAS),,[R vs SAS](Comp/r-sas_rbmi_continuous_joint) -Correlation,Pearson/Spearman/Kendall's Rank,[R](R/correlation),[SAS](SAS/correlation),[Python](python/correlation),[R vs SAS](Comp/r-sas_correlation) -Survival Models,Kaplan-Meier/log-rank test/Cox proportional hazards,[R](R/survival),[SAS](SAS/survival),,[R vs SAS](Comp/r-sas_survival) -Survival Models,Cause-specific hazards,[R](R/survival_csh),[SAS](SAS/survival_csh),,[R vs SAS](Comp/r-sas_survival_csh) -Survival Models,Accelerated failure time,[R](R/Accelerated_Failure_time_model),,, -Survival Models,Weighted log-rank test,[R](R/Weighted-log-rank),,, -Survival Models,Recurrent events,[R](R/recurrent_events),[SAS](SAS/recurrent_events),,[R vs SAS](Comp/r-sas_recurrent_events) -Survival Models,Cumulative incidence functions,[R](R/survival_cif),[SAS](SAS/survival_cif),,[R vs SAS](Comp/r-sas_survival_cif) -Survival Models,Tobit regression,[R](R/tobit regression),[SAS](SAS/tobit regression SAS),,[R vs SAS](Comp/r-sas_tobit) -Survival Models,Restricted Mean Survival Time (RMST),,[SAS](SAS/rmst),, -Bayesian Methods,Intro to Bayesian Analysis,,,, -Bayesian Methods,Repeated Measures MMRM,,,, -Sample size/ Power calculations,Intro to Sample Size,,,,[Summary](method_summary/sample_s_theory) -Sample size/ Power calculations,Superiority Single timepoint,[R](R/sample_s_superiority),[SAS](SAS/sample_s_superiority),, -Sample size/ Power calculations,Equivalence Single timepoint,[R](R/sample_s_equivalence),[SAS](SAS/sample_s_equivalence),, -Sample size/ Power calculations,Non-Inferiority Single timepoint,[R](R/sample_size_non-inferiority),[SAS](SAS/sample_s_noninferiority),, -Sample size/ Power calculations,Average BioEquivalence,[R](R/sample_size_average_bioequivalence),,, -Sample size/ Power calculations,Cochran-Armitage Test For Trend,[R](R/sample_s_test_of_trends),[SAS/ StatXact](SAS/sample_s_StatXact_test_of_trends),, -Sample size/ Power calculations,Group sequential designs-Survival,[R](R/gsd-tte),[SAS](SAS/gsd-tte),[East](East/gsd-tte),[R vs SAS vs East](Comp/r-east_gsd_tte) -Sample size/ Power calculations,Group sequential designs-Binary Endpoint,,,, -Causal inference/ Machine learning,Intro to Machine Learning,,,,[Summary](method_summary/stat_ml_methods) -Causal inference/ Machine learning,Propensity Score Matching,[R](R/causal_ps_matching),,,[R vs SAS](Comp/r-sas_psmatch) -Causal inference/ Machine learning,Propensity Score Weighting,,,,[R vs SAS](Comp/r-sas_psweight) -Causal inference/ Machine learning,Clustering,[R](R/Clustering_Knowhow),,, -Causal inference/ Machine learning,Principal Components Analysis (PCA),[R](R/PCA_analysis),,, -Causal inference/ Machine learning,Lasso,,,, -Causal inference/ Machine learning,Ridge Regression,,,, -Causal inference/ Machine learning,xgboost,[R](R/xgboost),,, -Other Methods,Survey statistics,[R](R/survey-stats-summary),[SAS](SAS/survey-stats-summary),[Python](python/survey-stats-summary),[R vs SAS vs Python](Comp/r-sas-python_survey-stats-summary) - - +method_grp,method_subgrp,r_links,sas_links,python_links,comparison_links +Summary Statistics,Rounding,[R](R/rounding),[SAS](SAS/rounding),[Python](python/Rounding),[R vs SAS](Comp/r-sas_rounding) +Summary Statistics,Summary statistics,[R](R/summary-stats),[SAS](SAS/summary-stats),[Python](python/Summary_statistics),[R vs SAS](Comp/r-sas-summary-stats) +Summary Statistics,Skewness/Kurtosis,[R](R/summary_skew_kurt),[SAS](SAS/summary_skew_kurt),[Python](python/skewness_kurtosis),[R vs SAS](Comp/r-sas_summary_skew_kurt) +General Linear Models,One-sample t-test,[R](R/ttest_1Sample),[SAS](SAS/ttest_1Sample),[Python](python/one_sample_t_test),[R vs SAS](Comp/r-sas_ttest_1Sample) +General Linear Models,Paired t-test,[R](R/ttest_Paired),[SAS](SAS/ttest_Paired),[Python](python/paired_t_test),[R vs SAS](Comp/r-sas_ttest_Paired) +General Linear Models,Two-sample t-test,[R](R/ttest_2Sample),[SAS](SAS/ttest_2Sample),[Python](python/two_samples_t_test),[R vs SAS](Comp/r-sas_ttest_2Sample) +General Linear Models,ANOVA,[R](R/anova),[SAS](SAS/anova),[Python](python/anova),[R vs SAS](Comp/r-sas_anova) +General Linear Models,ANCOVA,[R](R/ancova),[SAS](SAS/ancova),[Python](python/ancova),[R vs SAS](Comp/r-sas_ancova) +General Linear Models,MANOVA,[R](R/manova),[SAS](SAS/manova),[Python](python/MANOVA),[R vs SAS](Comp/r-sas_manova) +General Linear Models,Linear regression,[R](R/linear-regression),[SAS](SAS/linear-regression),[Python](python/linear_regression),[R vs SAS](Comp/r-sas_linear-regression) +Generalized Linear Models,Logistic regression,[R](R/logistic_regr),[SAS](SAS/logistic-regr),[Python](python/logistic_regression),[R vs SAS](Comp/r-sas_logistic-regr) +Generalized Linear Models,Poisson/negative binomial regression,[R](R/count_data_regression),[SAS](SAS/count_data_regression),,[R vs SAS](Comp/r-sas_negbin) +Non-Parametric Analysis,Wilcoxon signed-rank test,[R](R/wilcoxonsr_hodges_lehman),[SAS/ StatXact](SAS/wilcoxonsr_HL),,[R vs SAS](Comp/r-sas-wilcoxonsr_HL) +Non-Parametric Analysis,Mann-Whitney U/Wilcoxon rank-sum test,[R](R/nonpara_wilcoxon_ranksum),[SAS](SAS/ranksum),,[R vs SAS](Comp/r-sas-wilcoxon-ranksum_hl) +Non-Parametric Analysis,Kolmogorov-Smirnov test,[R](R/kolmogorov-smirnov_test),,, +Non-Parametric Analysis,Kruskal-Wallis test,[R](R/kruskal_wallis),[SAS](SAS/kruskal_wallis),[Python](python/kruskal_wallis),[R vs SAS](Comp/r-sas_kruskalwallis) +Non-Parametric Analysis,Friedman test,[R](R/friedman_test),[SAS](SAS/SAS_Friedmantest),,[R vs SAS](Comp/r-sas_friedman) +Non-Parametric Analysis,Jonckheere-Terpstra test,[R](R/jonckheere),[SAS](SAS/jonchkheere_terpstra),,[R vs SAS](Comp/r-sas_jonckheere) +Non-Parametric Analysis,Hodges-Lehman estimator,[R](R/nparestimate),[SAS](SAS/nparestimate),, +Categorical Data Analysis,Single proportion - Binomial test,[R](R/binomial_test),[SAS](SAS/binomial_test),[Python](python/binomial_test), +Categorical Data Analysis,Two paired proportions - McNemar's test,[R](R/mcnemar),[SAS](SAS/mcnemar),,[R vs SAS](Comp/r-sas_mcnemar) +Categorical Data Analysis,Marginal homogeneity tests,[R](R/marginal_homogeneity_tests),,, +Categorical Data Analysis,Two independent proportions - Chi-square test/Fisher's exact test,[R](R/association),[SAS](SAS/association),[Python](python/chi-square),[R vs SAS](Comp/r-sas_chi-sq) +Categorical Data Analysis,Stratified tables - Cochran-Mantel-Haenszel test,[R](R/cmh),[SAS](SAS/cmh),,[R vs SAS](Comp/r-sas_cmh) +Categorical Data Analysis,Intro to CIs for proportions,,,,[Summary](method_summary/ci_for_prop_intro) +Categorical Data Analysis,CIs - Single Proportion,[R](R/ci_for_prop),[SAS](SAS/ci_for_prop),,[R vs SAS](Comp/r-sas_ci_for_prop) +Categorical Data Analysis,CIs - Two Independent Proportions ,,[SAS](SAS/ci_for_2indep_prop),, +Categorical Data Analysis,CIs - Two Paired Proportions,,,, +Categorical Data Analysis,CIs - Proportions in Stratified Designs,,,, +Categorical Data Analysis,CIs - Poisson Exposure Adjusted Incidence Rates,,,, +Correlated Data Analysis,Linear Mixed Model Repeated Measure (MMRM),[R](R/mmrm),[SAS](SAS/mmrm),,[R vs SAS](Comp/r-sas_mmrm) +Correlated Data Analysis,Linear Mixed Model Random Effect,,,,[R vs SAS](Comp/Sas_vs_R_Mixed_Model) +Correlated Data Analysis,Generalized Linear Mixed Model (GLMM),,,, +Correlated Data Analysis,Generalized Estimating Equation (GEE),[R](R/gee),[SAS](SAS/gee),,[R vs SAS](Comp/r-sas_gee) +Multiple Imputation - Continuous Data MAR,Linear regression imputation,[R](R/mi_mar_regression),[SAS](SAS/mi_mar_regression),, +Multiple Imputation - Continuous Data MAR,Predictive mean matching,[R](R/mi_mar_predictive_mean_match),,, +Multiple Imputation - Continuous Data MNAR,Tipping point analysis (delta adjustment),[R](R/tipping_point),[SAS](SAS/tipping_point),,[R vs SAS](Comp/r-sas_tipping_point) +Multiple Imputation - Continuous Data MNAR,Reference-based imputation/joint modeling,[R](R/rbmi_continuous_joint),[SAS](SAS/rbmi_continuous_joint_SAS),,[R vs SAS](Comp/r-sas_rbmi_continuous_joint) +Correlation,Pearson/Spearman/Kendall's Rank,[R](R/correlation),[SAS](SAS/correlation),[Python](python/correlation),[R vs SAS](Comp/r-sas_correlation) +Survival Models,Kaplan-Meier/log-rank test/Cox proportional hazards,[R](R/survival),[SAS](SAS/survival),,[R vs SAS](Comp/r-sas_survival) +Survival Models,Cause-specific hazards,[R](R/survival_csh),[SAS](SAS/survival_csh),,[R vs SAS](Comp/r-sas_survival_csh) +Survival Models,Accelerated failure time,[R](R/Accelerated_Failure_time_model),,, +Survival Models,Weighted log-rank test,[R](R/Weighted-log-rank),,, +Survival Models,Recurrent events,[R](R/recurrent_events),[SAS](SAS/recurrent_events),,[R vs SAS](Comp/r-sas_recurrent_events) +Survival Models,Cumulative incidence functions,[R](R/survival_cif),[SAS](SAS/survival_cif),,[R vs SAS](Comp/r-sas_survival_cif) +Survival Models,Tobit regression,[R](R/tobit regression),[SAS](SAS/tobit regression SAS),,[R vs SAS](Comp/r-sas_tobit) +Survival Models,Restricted Mean Survival Time (RMST),,[SAS](SAS/rmst),, +Bayesian Methods,Intro to Bayesian Analysis,,,, +Bayesian Methods,Repeated Measures MMRM,,,, +Sample size/ Power calculations,Intro to Sample Size,,,,[Summary](method_summary/sample_s_theory) +Sample size/ Power calculations,Superiority Single timepoint,[R](R/sample_s_superiority),[SAS](SAS/sample_s_superiority),, +Sample size/ Power calculations,Equivalence Single timepoint,[R](R/sample_s_equivalence),[SAS](SAS/sample_s_equivalence),, +Sample size/ Power calculations,Non-Inferiority Single timepoint,[R](R/sample_size_non-inferiority),[SAS](SAS/sample_s_noninferiority),, +Sample size/ Power calculations,Average BioEquivalence,[R](R/sample_size_average_bioequivalence),,, +Sample size/ Power calculations,Cochran-Armitage Test For Trend,[R](R/sample_s_test_of_trends),[SAS/ StatXact](SAS/sample_s_StatXact_test_of_trends),, +Sample size/ Power calculations,Group sequential designs-Survival,[R](R/gsd-tte),[SAS](SAS/gsd-tte),[East](East/gsd-tte),[R vs SAS vs East](Comp/r-east_gsd_tte) +Sample size/ Power calculations,Group sequential designs-Binary Endpoint,,,, +Causal inference/ Machine learning,Intro to Machine Learning,,,,[Summary](method_summary/stat_ml_methods) +Causal inference/ Machine learning,Propensity Score Matching,[R](R/causal_ps_matching),,,[R vs SAS](Comp/r-sas_psmatch) +Causal inference/ Machine learning,Propensity Score Weighting,,,,[R vs SAS](Comp/r-sas_psweight) +Causal inference/ Machine learning,Clustering,[R](R/Clustering_Knowhow),,, +Causal inference/ Machine learning,Principal Components Analysis (PCA),[R](R/PCA_analysis),,, +Causal inference/ Machine learning,Lasso,,,, +Causal inference/ Machine learning,Ridge Regression,,,, +Causal inference/ Machine learning,xgboost,[R](R/xgboost),,, +Other Methods,Survey statistics,[R](R/survey-stats-summary),[SAS](SAS/survey-stats-summary),[Python](python/survey-stats-summary),[R vs SAS vs Python](Comp/r-sas-python_survey-stats-summary) + + From a93b208681fb62a8342736cd627c09ec961f5cf0 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 15:12:49 +0000 Subject: [PATCH 53/58] minor edits --- SAS/ci_for_2indep_prop.qmd | 4 +++- SAS/ci_for_prop.qmd | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 91126754b..6d76ab32b 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -148,7 +148,9 @@ If an EXACT statement is used to produce CIs, SAS does not offer any matching hy It is important to check the output to ensure that you are modelling Active - Placebo, and response = Yes (not Response=No). By default SAS sorts alphabetically and calculates CI's for the first column. You can change this by using the `COLUMN=` Option in riskdiff or by sorting the dataset (here by trt, then descending resp), and then using `order=data` in the proc freq. This tells SAS to use the order you have sorted the data by. SAS confirms this by saying "Difference is (Row 1 - Row 2)" and "Column 1 (resp=Yes)". -Similarly for relrisk, although the output does not state that the RR is calculated as (Row 1) / (Row 2). If treatment groups are labelled differently, you might need to sort by descending trt to obtain the correct contrast (note that unlike for RD, the same result cannot be obtained by setting `COLUMN=2`) +Similarly for relrisk, although the output does not state that the RR is calculated as (Row 1) / (Row 2). If treatment groups are labelled differently, you might need to sort by descending trt to obtain the correct contrast (note that unlike for RD, the same result cannot be obtained by setting `COLUMN=2`). + +SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. ```{sas} #| eval: false diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index 0afbead0c..81237078f 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -163,7 +163,7 @@ By adding the option `BINOMIAL(LEVEL="Yes")` to your 'PROC FREQ' TABLES statemen **It is very important to ensure you are calculating the CI for the correct level! Check your output to confirm, you will see below it states `resp=Yes` !** -Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can (and should) be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. +Caution is required if there are no responders in a group (aside from any issues with the choice of confidence interval method), as SAS PROC FREQ (as of v9.4) does not output any confidence intervals in this case. If the `LEVEL` option has been specified, an error is produced, otherwise the procedure by default generates CIs for the proportion of non-responders. Note that valid CIs can (and should) be obtained for both p = 0/n and p = n/n. If needed, the interval for 0/n can be derived as 1 minus the transposed interval for n/n. The output consists of the proportion of resp=Yes, the Asymptotic SE, 95% CIs using normal-approximation method, 95% CI using the Clopper-Pearson method, and then a Binomial test statistic and p-value for the null hypothesis of H0: Proportion = 0.5. From 80da58f63b0c10bb2d77c23546df39695cf59336 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 15:38:52 +0000 Subject: [PATCH 54/58] add .csl file for citations --- SAS/nature.csl | 189 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 SAS/nature.csl diff --git a/SAS/nature.csl b/SAS/nature.csl new file mode 100644 index 000000000..3b1e3d7a2 --- /dev/null +++ b/SAS/nature.csl @@ -0,0 +1,189 @@ + + From 74891222ec1dce04787e4177451dad7c9fac2708 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 19:36:48 +0000 Subject: [PATCH 55/58] remove #| eval: false from SAS chunks --- SAS/ci_for_2indep_prop.qmd | 3 --- SAS/ci_for_paired_prop.qmd | 23 ++++++++++------------- 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 6d76ab32b..135b1e99e 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -17,7 +17,6 @@ Caution is required if there are no responders (or all responders) in both group The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. ```{sas} -#| eval: false proc import datafile = 'data/adcibc.csv' out = adcibc dbms = csv @@ -44,7 +43,6 @@ run; The below shows that for the Active Treatment, there are 36 responders out of 154 subjects, p1 = 0.2338 (23.38% responders), while for the placebo treatment p2 = 12/77 = 0.1558, giving a risk difference of 0.0779, relative risk 1.50, and odds ratio 1.6525. ```{sas} -#| eval: false proc freq data=adcibc2; table trt*resp/ nopct nocol; run; @@ -153,7 +151,6 @@ Similarly for relrisk, although the output does not state that the RR is calcula SAS output often rounds to 3 or 4 decimal places in the output window, however the full values can be obtained using SAS ODS statements. ```{sas} -#| eval: false ****************************; *** Risk Difference examples; diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index 5c3762b25..a9f17f001 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -11,7 +11,6 @@ title: "Confidence intervals for Paired Proportions in SAS" The adcibc data stored [here](../data/adcibc.csv) was used in this example, creating a binary treatment variable `trt` taking the values of `Act` or `PBO` and a binary response variable `resp` taking the values of `Yes` or `No`. For this example, a response is defined as a score greater than 4. ```{sas} -#| eval: false data adcibc2 (keep=trt resp) ; set adcibc; if aval gt 4 then resp="Yes"; @@ -24,7 +23,6 @@ run; The below shows that for the Active Treatment, there are 36 responders out of 154 subjects = 0.2338 (23.38% responders). ```{sas} -#| eval: false proc freq data=adcibc2; table trt*resp/ nopct nocol; run; @@ -51,16 +49,16 @@ In all these cases, the calculated proportions for the 2 groups are not independ Using a cross over study as our example, a 2 x 2 table can be formed as follows: -+-----------------------+---------------+---------------+--------------+ -| | Placebo\ | Placebo\ | Total | -| | Response= Yes | Response = No | | -+=======================+===============+===============+==============+ -| Active Response = Yes | r | s | r+s | -+-----------------------+---------------+---------------+--------------+ -| Active Response = No | t | u | t+u | -+-----------------------+---------------+---------------+--------------+ -| Total | r+t | s+u | N = r+s+t+u | -+-----------------------+---------------+---------------+--------------+ ++-----------------------+---------------+---------------+---------------+ +| | Placebo\ | Placebo\ | Total | +| | Response= Yes | Response = No | | ++=======================+===============+===============+===============+ +| Active Response = Yes | r | s | r+s | ++-----------------------+---------------+---------------+---------------+ +| Active Response = No | t | u | t+u | ++-----------------------+---------------+---------------+---------------+ +| Total | r+t | s+u | N = r+s+t+u | ++-----------------------+---------------+---------------+---------------+ The proportions of subjects responding on each treatment are: @@ -178,7 +176,6 @@ The default method is Mantel-Haenszel confidence limits. SAS can also Score (Mie As you can see below, this is not a CI for difference in proportions of 0.196, it is a CI for the risk difference of 0.0260. So must be interpreted with much consideration. ```{sas} -#| eval: false data dat_used; input ID$ PBO ACT @@; datalines; From 2670cd2eb252010253e4d27b4798290765f642f4 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Thu, 5 Mar 2026 19:45:29 +0000 Subject: [PATCH 56/58] delete spurious SAS chunk --- SAS/ci_for_2indep_prop.qmd | 5 ----- 1 file changed, 5 deletions(-) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 135b1e99e..3387c7695 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -135,11 +135,6 @@ Within SAS PROC FREQ for the asymptotic methods for RD, consistency with a tradi For the SCAS or MN method, the %SCORECI macro provides the p-value for a specified NI margin, with guaranteed consistency with the CI. -```{sas} -#| eval: false - -``` - If an EXACT statement is used to produce CIs, SAS does not offer any matching hypothesis tests for NI or equivalence testing. ## Example Code using PROC FREQ From 2191c0dba450e42be1750a13403b3c91bbeaddf0 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Fri, 6 Mar 2026 12:05:23 +0000 Subject: [PATCH 57/58] add eval: false in header --- SAS/ci_for_2indep_prop.qmd | 2 ++ 1 file changed, 2 insertions(+) diff --git a/SAS/ci_for_2indep_prop.qmd b/SAS/ci_for_2indep_prop.qmd index 3387c7695..368378dc1 100644 --- a/SAS/ci_for_2indep_prop.qmd +++ b/SAS/ci_for_2indep_prop.qmd @@ -2,6 +2,8 @@ title: "Confidence Intervals for Independent Proportions in SAS" bibliography: references.bib csl: nature.csl +execute: + eval: false --- ## Introduction From c9c76d7d1182dfd86377776c8c123d58d3fc9db4 Mon Sep 17 00:00:00 2001 From: Pete Laud Date: Fri, 6 Mar 2026 12:16:49 +0000 Subject: [PATCH 58/58] add eval: false to header --- SAS/ci_for_paired_prop.qmd | 2 ++ SAS/ci_for_prop.qmd | 2 ++ 2 files changed, 4 insertions(+) diff --git a/SAS/ci_for_paired_prop.qmd b/SAS/ci_for_paired_prop.qmd index a9f17f001..4adcad917 100644 --- a/SAS/ci_for_paired_prop.qmd +++ b/SAS/ci_for_paired_prop.qmd @@ -1,5 +1,7 @@ --- title: "Confidence intervals for Paired Proportions in SAS" +execute: + eval: false --- ## Introduction diff --git a/SAS/ci_for_prop.qmd b/SAS/ci_for_prop.qmd index a01f33af0..60d0166e3 100644 --- a/SAS/ci_for_prop.qmd +++ b/SAS/ci_for_prop.qmd @@ -1,5 +1,7 @@ --- title: "Confidence intervals for a Proportion in SAS" +execute: + eval: false --- ## Introduction