Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
e0a00dc
Merge pull request #33 from jepusto/main
jepusto Sep 5, 2025
3c358e5
Merge pull request #36 from jepusto/main
jepusto Sep 20, 2025
2f0bd55
Comment on power conventions.
jepusto Sep 29, 2025
e7d7fa0
Working on Chapter 9 (performance metrics).
jepusto Oct 22, 2025
700e967
Added Monte Carlo standard error for median bias.
jepusto Oct 23, 2025
4d51cab
Added MCSE for trimmed mean. Added equation references to the summary…
jepusto Oct 23, 2025
0a6bf84
Copy edits.
jepusto Oct 28, 2025
2556fb6
Performance measures instead of metrics. Finished revisions to chapte…
jepusto Oct 29, 2025
52a97c2
Working on new chapter on simulation across multiple scenarios.
jepusto Nov 1, 2025
fd4261c
Fixed bug in generate_ANOVA_new().
jepusto Nov 2, 2025
a64d7ba
Corrected bug in generate_ANOVA_new().
jepusto Nov 3, 2025
132de3f
Merge pull request #39 from jepusto/main
jepusto Nov 11, 2025
0c5aace
Merge pull request #40 from jepusto/JEP-Chp-9
jepusto Nov 11, 2025
3d2e19d
small todo
lmiratrix Dec 18, 2025
a234f93
Finished edits to chapter on simulating across multiple scenarios.
jepusto Dec 18, 2025
96f9697
Fixed cross-references.
jepusto Dec 18, 2025
f8c4a0a
Cut simhelpers from the list of further resources.
jepusto Dec 18, 2025
8d90f5c
compiled book.
lmiratrix Jan 6, 2026
76e931a
Merge pull request #37 from jepusto/JEP-Chp-9
jepusto Jan 6, 2026
ba132d3
Merge branch 'main' of https://github.com/jepusto/Designing-Simulatio…
lmiratrix Jan 6, 2026
d653aca
Merge branch 'JEP-Chp-9'
lmiratrix Jan 6, 2026
cf35e27
Fixed merge conflicts.
jepusto Jan 6, 2026
26f44f7
Merge pull request #41 from jepusto/JEP-Chp-10
jepusto Jan 6, 2026
8cb3626
Updated chapter 7 with some minor edits and cleanup of the safely sec…
lmiratrix Jan 6, 2026
b92bccc
Tidied up intro to Chapter 7.
jepusto Jan 9, 2026
253e00c
Merge pull request #42 from jepusto/luke-edit-chap7
jepusto Jan 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions 005-initial-t-test-simulation.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ prop.test( sum(covered), length(covered), p = 0.95)
Our coverage is _too low_; the confidence interval based on the $t$-test misses the the true value more often than it should.
We have learned that the $t$-test can fail when applied to non-normal (skewed) data.

## Simulating across different scenarios
## Simulating across different scenarios {#simulating-across-different-scenarios}

So far, we have looked at coverage rates of the confidence interval under a single, specific scenario, with a sample size of 10, a population mean of 4, and a geometrically distributed variable.
We know from statistical theory (specifically, the central limit theorem) that the confidence interval should work better if the sample size is big enough.
Expand Down Expand Up @@ -194,7 +194,6 @@ However, this will quickly get cumbersome if we want to evaluate many different
A better approach is to use a mapping function from the `purrr` package.[^apply]
The `map_dbl()` function takes a list of values and calls a function for each value in the list.
This accomplishes the same thing as using a `for` loop to iterate through a list of items (if you happen to be familiar with these), but is more succinct.
See Appendix chapter \@ref(repeating-oneself) for more on `map()`.[^mapping]

To proceed, we first create a list of sample sizes to test out:
```{r}
Expand All @@ -206,8 +205,7 @@ coverage_est <- map_dbl( ns, ttest_CI_experiment)
```
This code will run our experiment for each value in `ns`, and then return a vector of the estimated coverage rates for each of the sample sizes.

[^apply]: Alternately, readers familiar with the `*apply()` family of functions from Base R might prefer to use `lapply()` or `sapply()`, which do essentially the same thing as `purrr::map_dbl()`.
[^mapping]: You can also check out [Section 21.5 of R for Data Science (1st edition)](https://r4ds.had.co.nz/iteration.html#the-map-functions), which provides an introduction to mapping.
[^apply]: See [Section 21.5 of R for Data Science (1st edition)](https://r4ds.had.co.nz/iteration.html#the-map-functions), which provides an introduction to mapping. Alternately, readers familiar with the `*apply()` family of functions from Base R might prefer to use `lapply()` or `sapply()`, which do essentially the same thing as `purrr::map_dbl()`.

We advocate for depicting simulation results graphically.
To do so, we store the simulation results in a dataset and then create a line plot using a log scale for the horizontal axis:
Expand Down
6 changes: 3 additions & 3 deletions 010-Simulation-structure.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ A well-written estimation method should, in principle, work not only on a simula
Because of this, the inputs of the `analyze()` function should not typically include any information about the parameters of the data-generating process.
To be realistic, the code for our simulated data-analysis procedure should not make use of anything that the analyst could not know when analyzing a real dataset.
Thus, `analyze()` has an argument for the sample dataset but not for `model_params`.
We discuss the form and content of the data analysis function further in Chapter \@ref(data-analysis-procedures).
We discuss the form and content of the data analysis function further in Chapter \@ref(estimation-procedures).

### Repetition

Expand Down Expand Up @@ -261,7 +261,7 @@ For example, we might want to know how close an estimator gets to the target par
We might want to know if a confidence interval captures the true parameter the right proportion of the time, as in the simulation from Chapter \@ref(t-test-simulation).
Performance is defined in terms of the sampling distribution of estimators or analysis results, across an infinite number of replications of the data-generating process.
In practice, we use many replications of the process, but still only a finite number. Consequently, we actually _estimate_ the performance measures and need to attend to the Monte Carlo error in the estimates.
We discuss the specifics of different performance measures and assessment of Monte Carlo error in Chapter \@ref(performance-criteria).
We discuss the specifics of different performance measures and assessment of Monte Carlo error in Chapter \@ref(performance-measures).

### Multifactor simulations

Expand All @@ -277,7 +277,7 @@ To implement a multifactor simulation, we will follows the same principles of mo
In particular, we will take the code developed for simulating a single context and bundle it into a function that can be evaluated for any and all scenarios of interest.
Simulation studies often follow a full factorial design, in which each level of a factor (something we vary, such as sample size, true treatment effect, or residual variance) is crossed with every other level.
The experimental design then consists of sets of parameter values (including design parameters, such as sample sizes), and these too can be represented in an object, distinct from the other components of the simulation.
We will discuss multiple-scenario simulations in Part III (starting with Chapter \@ref(exp-design)), after we more fully develop the core concepts and techniques involved in simulating a single context.
We will discuss multiple-scenario simulations in Part III (starting with Chapter \@ref(simulating-multiple-scenarios)), after we more fully develop the core concepts and techniques involved in simulating a single context.

## Exercises

Expand Down
4 changes: 2 additions & 2 deletions 015-Case-study-ANOVA.Rmd
Original file line number Diff line number Diff line change
Expand Up @@ -288,7 +288,7 @@ Once you have built a function, one way to check that it is working properly is
If things still work, then you can be somewhat confident that you have successfully bundled your code into the function.
Once you bundle your code, you can also do a search and replace to change the variable names inside your function to something more generic, to better clarify the distinction betwen object names and argument names.

## The hypothesis testing procedures
## The hypothesis testing procedures {#ANOVA-hypothesis-testing-function}

Brown and Forsythe considered four different hypothesis testing procedures for heteroskedastic ANOVA, but we will focus on just two of the tests for now.
We start with the conventional one-way ANOVA that mistakenly assumes homoskedasticity.
Expand Down Expand Up @@ -409,7 +409,7 @@ mean(p_vals$Welch < 0.05)
The Welch test does much better, although it appears to be a little bit in excess of 0.05.

Note that these two numbers are quite close (though not quite identical) to the corresponding entries in Table 1 of Brown and Forsythe (1974). The difference is due to the fact that both Table 1 and are results are actually _estimated_ rejection rates, because we have not actually simulated an infinite number of replications. The estimation error arising from using a finite number of replications is called _simulation error_ (or _Monte Carlo error_).
In Chapter \@ref(performance-criteria), we will look more at how to estimate and control the Monte Carlo simulation error in performance measures.
In Chapter \@ref(performance-measures), we will look more at how to estimate and control the Monte Carlo simulation error in performance measures.

So there you have it! Each part of the simulation is a distinct block of code, and together we have a modular simulation that can be easily extended to other scenarios or other tests.
The exercises at the end of this chapter ask you to extend the framework further.
Expand Down
Loading
Loading