ABC Routine #1413

phumtutum · 2021-11-08T01:32:30Z

Added an ABC base class + controller
Added an implementation of the ABC rejection algorithm
Added tests for those classes
Added a jupyter notebook to highlight the rejection algorithm, here is the resulting plot:

codecov · 2021-11-08T01:46:21Z

Codecov Report

Merging #1413 (95c910e) into master (ba6d5ce) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##            master     #1413    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           92        94     +2     
  Lines         8968      9148   +180     
==========================================
+ Hits          8968      9148   +180

Impacted Files	Coverage Δ
pints/__init__.py	`100.00% <100.00%> (ø)`
pints/_abc/__init__.py	`100.00% <100.00%> (ø)`
pints/_abc/_abc_rejection.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba6d5ce...95c910e. Read the comment docs.

MichaelClerx

Haven't checked the tests or notebook, but this is looking quite good!

Some comments attached

pints/_abc/__init__.py

pints/tests/test_abc_controller.py

pints/tests/test_abc_rejection.py

MichaelClerx · 2021-11-15T15:03:50Z

pints/tests/test_abc_rejection.py

+    """
+    Tests the basic methods of the ABC Rejection routine.
+    """
+# Set up toy model, parameter values, problem, error measure


chonlei · 2021-11-26T08:19:11Z

@phumtutum @ben18785 Just while you are still working on it, have you come across this pyabc package before? They tried applying ABC to ODE problems (and SDE too of course). Thought might be good to have a look, in particular their choice of summary statistics, if you haven't done so already.

P.S.: If you think that's relevant enough, might be good to open a separate issue for it! Not saying you have to finish/include it in this PR. For some reason I couldn't find any relevant issue, so I thought I'd put a note here first.

MichaelClerx · 2021-11-29T12:05:15Z

pints/_abc/_abc_rejection.py

+    def threshold(self):
+        """
+        Returns threshold error distance that determines if a sample is
+        accepted (is ``error < threshold``).


Suggested change

accepted (is ``error < threshold``).

accepted (if ``error < threshold``).

MichaelClerx · 2021-11-29T12:08:49Z

Thanks for the changes @phumtutum , looks good to me now!

@ben18785 @fcooper8472 @martinjrobins are we happy to merge this, or should we have a https://github.com/pints-team/method-merge-tests first?

MichaelClerx · 2021-11-30T12:50:27Z

Discussion in meeting on 2021-12-30

Victor & Ben say: decent pre-merge tests require more stochastic problems. this is work in progress (#1417), so can't do this just yet

Conclusion: Let's merge this before adding pre-merge tests

MichaelClerx

Some final suggestions:

This needs a changelog.md update
The notebook could do with a bit more text, i.e. start by showing output from the model, explain how a threshold is set, show a final posterior prediction (i.e. the range of model predictions deemed acceptable)

rccreswell

This looks great, thanks @phumtutum , I'm very excited to see pints branching into these topics

Suggestions on the notebook:

Add a plot of the data
Add a random seed. On the current problem, the performance can vary significantly across realizations of the data generation
The rejection ABC seems a highly simplistic inference method. I ran some simulations and when the prior has a significantly higher variance than the posterior, the method is hopelessly inefficient. This rather worries me as in Bayesian inference it is not uncommon to choose a diffuse prior with a large variance. While I admit that this is a deeper topic than ought to be comprehensively addressed in an example notebook, somewhere in the notebook I think we need to give users some warning that this method does not always work, and tell them to monitor the acceptance rate to see if it is working for their problem.

docs/source/abc_samplers/index.rst

pints/_abc/__init__.py

pints/_abc/_abc_rejection.py

pints/_abc/__init__.py

phumtutum · 2021-12-01T16:17:43Z

I have addressed the comments and also fixed a small bug that was making the sampler less efficient.

CHANGELOG.md

docs/source/abc_samplers/index.rst

MichaelClerx

It's looking really good, but the notebook can still use more text. Imagine this is someone's first attempt at running an ABC algorithm.

I'd want a bit more explanation, e.g.

This example shows you how to perform Rejection ABC on a time series from the stochastic degradation model. This model describes the .... over time. It differs from most other models in PINTS, in that it ... Then explain why ABC can deal with these kind of models.

Since it's stochastic, I'd also plot a few runs with the same parameters, and then show how that contrasts with a few runs with a 2nd set of parameters. Not because we are describing the model (that's for the model's notebook), but to show the problem faced by the sampling routine.

Now the Rejection ABC algorithm can be applied to sample parameter values. To do this, we... explain about using an error measure instead of a likelihood. Explain how you choose it (ok if this is a bit hand wavy!).

Plotting the approximate posterior compared to the actual parameter value.
-->
Finally, to see why are we plotting this?, we plot the approximate...

Then show the graph. Then explain (A) what we see in the graph (it's never as obvious to the audience, so always explain what they're looking at. also add x & y labels); (B) explain what that tells us about the problem, method, and choices we made (e.g. error)

MichaelClerx · 2021-12-07T14:52:15Z

You could also consider keeping this notebook short, but adding a "first-example" notebook for ABC with more background and then refer to that in the abc rejection notebook. This would be in line with the notebooks for optimisation and mcmc sampling

phumtutum · 2021-12-12T21:39:29Z

I have decided to go with extending the current jupyter notebook instead of creating a new one. Please let me know what you think about this version of the ipynb.

phumtutum · 2022-02-03T22:09:58Z

This PR is ready for a review, I have added the ipynb changes so that now pints.toy.stochastic.Degradation model is used. However, some set up for the unit test seems not to work. I am not sure why or if that problem is related to this PR.

ben18785

I've made a few cosmetic changes to the notebook, but otherwise I'm happy for this to go in.

ben18785 · 2022-03-15T18:53:21Z

@MichaelClerx @rcw5890 are you happy for this to go in now? I've had a run through and think all's good from my end.

MichaelClerx · 2022-03-15T19:04:16Z

@ben18785 Yeah I'd gotten to the point where I was happy if you were. Have not re-checked but safe to assume I still am :D

ben18785 · 2022-03-15T19:12:23Z

Great, thanks @MichaelClerx. @rcw5890 are you happy with this one to go in?

MichaelClerx · 2022-03-17T20:33:56Z

Fixed the merge conflicts (got a bit messy 😓 ). @rcw5890 what do you think?

phumtutum requested a review from ben18785 November 8, 2021 13:29

MichaelClerx requested changes Nov 15, 2021

View reviewed changes

phumtutum requested a review from MichaelClerx November 18, 2021 21:58

MichaelClerx reviewed Nov 29, 2021

View reviewed changes

MichaelClerx approved these changes Nov 29, 2021

View reviewed changes

MichaelClerx requested review from MichaelClerx and rccreswell and removed request for ben18785 November 30, 2021 12:50

MichaelClerx requested changes Nov 30, 2021

View reviewed changes

MichaelClerx added the new method label Nov 30, 2021

rccreswell requested changes Nov 30, 2021

View reviewed changes

phumtutum requested review from MichaelClerx and rccreswell December 1, 2021 16:17

MichaelClerx reviewed Dec 2, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

MichaelClerx reviewed Dec 2, 2021

View reviewed changes

docs/source/abc_samplers/index.rst Outdated Show resolved Hide resolved

MichaelClerx requested changes Dec 7, 2021

View reviewed changes

phumtutum requested a review from MichaelClerx December 12, 2021 21:38

phumtutum mentioned this pull request Dec 12, 2021

Generic stochastic model #1417

Merged

phumtutum mentioned this pull request Feb 3, 2022

Added additional stochastic models[finished] #1432

Merged

phumtutum and others added 2 commits February 11, 2022 13:42

fixed scalability bug + redone ipynb to assure proper functionality

0222bbf

small changes to rejection abc notebook

e992673

ben18785 self-requested a review March 15, 2022 18:52

ben18785 approved these changes Mar 15, 2022

View reviewed changes

MichaelClerx approved these changes Mar 15, 2022

View reviewed changes

MichaelClerx force-pushed the i-96-abc-routine branch from 4082204 to e992673 Compare March 17, 2022 19:55

MichaelClerx added 3 commits March 17, 2022 20:19

Merge branch 'master' into i-96-abc-routine

64bd474

Merge leftovers

a5e026c

Test tweak

95c910e

rccreswell approved these changes Mar 22, 2022

View reviewed changes

rccreswell merged commit 7380440 into master Mar 22, 2022

MichaelClerx deleted the i-96-abc-routine branch March 22, 2022 14:26

	accepted (is ``error < threshold``).
	accepted (if ``error < threshold``).

ABC Routine #1413

ABC Routine #1413

Uh oh!

Conversation

phumtutum commented Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MichaelClerx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MichaelClerx Nov 15, 2021

Choose a reason for hiding this comment

Uh oh!

chonlei commented Nov 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelClerx Nov 29, 2021

Choose a reason for hiding this comment

Uh oh!

MichaelClerx commented Nov 29, 2021

Uh oh!

MichaelClerx commented Nov 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelClerx left a comment • edited by phumtutum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rccreswell left a comment • edited by phumtutum Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phumtutum commented Dec 1, 2021

Uh oh!

Uh oh!

Uh oh!

MichaelClerx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaelClerx commented Dec 7, 2021

Uh oh!

phumtutum commented Dec 12, 2021

Uh oh!

phumtutum commented Feb 3, 2022

Uh oh!

ben18785 left a comment

Choose a reason for hiding this comment

Uh oh!

ben18785 commented Mar 15, 2022

Uh oh!

MichaelClerx commented Mar 15, 2022

Uh oh!

ben18785 commented Mar 15, 2022

Uh oh!

MichaelClerx commented Mar 17, 2022

Uh oh!

Reviewers

phumtutum commented Nov 8, 2021 •

edited

Loading

codecov bot commented Nov 8, 2021 •

edited

Loading

chonlei commented Nov 26, 2021 •

edited

Loading

MichaelClerx commented Nov 30, 2021 •

edited

Loading

MichaelClerx left a comment •

edited by phumtutum

Loading

rccreswell left a comment •

edited by phumtutum

Loading

MichaelClerx left a comment •

edited

Loading