Building blocks for PEBBLE#625

Open

dan-pandori wants to merge 55 commits intomasterfrom

dpandori_wellford

Commits on Dec 1, 2022

Welfords alg and test

dan-pandori
authored and
Jan Michelfeit
committed
Next func

dan-pandori
authored and
Jan Michelfeit
committed
Test update

dan-pandori
authored and
Jan Michelfeit
committed
compute_state_entropy and test

dan-pandori
authored and
Jan Michelfeit
committed
Sketch of the entropy reward replay buffer

dan-pandori
authored and
Jan Michelfeit
committed
Batchify state entropy func

dan-pandori
authored and
Jan Michelfeit
committed
Final sketch of replay entropy buffer.

dan-pandori
authored and
Jan Michelfeit
committed
First test

dan-pandori
authored and
Jan Michelfeit
committed
Test cleanup

dan-pandori
authored and
Jan Michelfeit
committed
Update

dan-pandori
authored and
Jan Michelfeit
committed
Commit for diff

dan-pandori
authored and
Jan Michelfeit
committed
Push final-ish state

dan-pandori
authored and
Jan Michelfeit
committed
#625 refactor RunningMeanAndVar
Jan Michelfeit
committed
#625 use RunningNorm instead of RunningMeanAndVar
Jan Michelfeit
committed
#625 make copy of train_preference_comparisons.py for pebble
Jan Michelfeit
committed
#625 use an OffPolicy for pebble
Jan Michelfeit
committed
#625 fix assumptions about shapes in ReplayBufferEntropyRewardWrapper
Jan Michelfeit
committed
#625 entropy reward as a function
Jan Michelfeit
committed
#625 make entropy reward serializable with pickle
Jan Michelfeit
committed
#625 revert change of compute_state_entropy() from tensors to numpy
Jan Michelfeit
committed
#625 extract _preference_feedback_schedule()
Jan Michelfeit
committed
#625 introduce parameter for pretraining steps
Jan Michelfeit
committed
#625 add initialized callback to ReplayBufferRewardWrapper
Jan Michelfeit
committed
#625 fix entropy_reward.py
Jan Michelfeit
committed
#625 remove ReplayBufferEntropyRewardWrapper
Jan Michelfeit
committed
#625 introduce ReplayBufferAwareRewardFn
Jan Michelfeit
committed
#625 rename PebbleStateEntropyReward
Jan Michelfeit
committed
#625 PebbleStateEntropyReward can switch from unsupervised pretraining
Jan Michelfeit
committed
#625 add optional pretraining to PreferenceComparisons
Jan Michelfeit
committed
#625 PebbleStateEntropyReward supports the initial phase before replay buffer is filled
Jan Michelfeit
committed
#625 entropy_reward can automatically detect if enough observations are present
Jan Michelfeit
committed
#625 fix entropy shape
Jan Michelfeit
committed
#625 rename unsupervised_agent_pretrain_frac parameter
Jan Michelfeit
committed
#625 specialized PebbleAgentTrainer to distinguish from old preference comparison trainer
Jan Michelfeit
committed
#625 merge pebble to train_preference_comparisons.py and configure only through sacred
Jan Michelfeit
committed
#625 plug in pebble according to parameters
Jan Michelfeit
committed
#625 fix pre-commit errors
Jan Michelfeit
committed
#625 add test for pebble agent trainer
Jan Michelfeit
committed
#625 fix more pre-commit errors
Jan Michelfeit
committed

Commits on Dec 2, 2022

#625 fix even more pre-commit errors
Jan Michelfeit
committed
code review - Update src/imitation/policies/replay_buffer_wrapper.py

mifeet
and
AdamGleave
authored
#625 code review
Jan Michelfeit
committed
#625 code review: do not allocate timesteps for pretraining if there is no pretraining
Jan Michelfeit
committed
Update src/imitation/algorithms/preference_comparisons.py

mifeet
and
AdamGleave
authored
#625 code review: remove ignore
Jan Michelfeit
committed
#625 code review - skip pretrainining if zero timesteps
Jan Michelfeit
committed
#625 code review: separate pebble and environment configuration
Jan Michelfeit
committed
#625 fix even even more pre-commit errors
Jan Michelfeit
committed
#625 fix even even more pre-commit errors
Jan Michelfeit
committed

Commits on Dec 9, 2022

#641 code review: remove set_replay_buffer
Jan Michelfeit
committed
#641 code review: fix comment
Jan Michelfeit
committed

Commits on Dec 10, 2022

#641 code review: replace RunningNorm with NormalizedRewardNet
Jan Michelfeit
committed
#641 code review: refactor PebbleStateEntropyReward so that inner RewardNets can be injected from the outside
Jan Michelfeit
committed
#641 fix static analysis and tests
Jan Michelfeit
committed

Commits on Dec 13, 2022

#641 increase coverage
Jan Michelfeit
committed