Memory consumption and learning issues

[Phantomb]

Tested as well with the unaltered codebase and the training example given in the readme. Over time, memory consumed grows to using up all available ram (nearly 16GBs), at which point either: - Python throws an error and stops the execution, or - Malmo becomes unreachable, throwing a specific error message every step (I can't recall which one, will edit when it occurs again).

I believe this wasn't the case for earlier versions of the codebase as I was previously able to run for days on end without issue.

I'm curious if you recognise this behaviour, and what part of the code might be responsible.

[tesslerc]

This could be due to the replay memory, as this is the only section in which we store data (which isn't thrown away).
The following params could be the cause:

1. retain rgb - each state is x3 larger
2. success replay memory - an additional replay memory which stores only success trajectories
3. replay memory size - the size itself of the replay memory

it could be this commit: ffe029e

[Phantomb]

Hmm I'll give it a try learning with a smaller replay memory size. The increase from 5K to 100K seems pretty significant. Do you reckon setting it back to something like 5K, and enabling both the success replay memory & the prioritized experience replay might be a worthwhile setting?

Thanks for the quick answer!

[tesslerc]

I think most problems in Minecraft are rather "simple" meaning short trajectories in a small domain - so this should be fine. Just remember that you need a large enough replay memory so the samples will be as diverse as possible.

[Phantomb]

Allright I did some testing and by playing with the parameters (most importantly keeping the memory smaller) the memory is no longer an issue, thanks!

However, as I was testing this with the single_room domain, it has not managed to achieve any reliable success percentage above 5%, even after 400k steps. Image:

This uses the unchanged master branch code. I ran two versions, one with prioritized replay and success replay:
python main.py qr_dqn single_room --number_of_atoms 200 --number_of_agents 1 --malmo_ports 10000 --retain_rgb --save_name qr-dqn-test_extra-replay --replay_memory_size 5000 --success_replay_memory --prioritized_experience_replay
And one without:
python main.py qr_dqn single_room --number_of_atoms 200 --number_of_agents 1 --malmo_ports 10000 --retain_rgb --save_name qr-dqn-test --replay_memory_size 5000
Both ran for over 460k steps, and both achieving similar results as the image shown.

However, when I tested the single_room domain with the unmodified codebase and the parameters from the readme a while back, it performed much better, reaching a near 100% success rate after only 150k steps. Image:

I am wondering what kind of reliable performance you get out of the single_room domain and with what kind of parameters.

[tesslerc]

From what commit did the test run properly? I'll try and find what I broke in the process.

[Phantomb]

Unfortunately I am not sure of the exact commit, but it was at least shortly before 050800e

I'm running a single-room test with the parameters from the readme now on commit 90e2eae to check the performance there (currently at 18k steps)

[tesslerc]

OK, thanks.
I'll take a look at this on Sunday. If you have any updates by then regarding your current test, please share :)

[Phantomb]

Thanks, Will do!
Out of curiosity, have you run the single-room (or any other domain I can test as well) any time lately? What kind of performance do you get with it?

[tesslerc]

I think the issue is due to the success replay memory probability decay. Default parameter is decay steps = 0 and when decay steps is 0, it falls back to the final probability (defaults to 0%).

Making sure this is the issue, then I'll push a quick fix.

[tesslerc]

Pushed a fix. I see on my end that it solves within 50k steps.
The commandline I used:
python3.6 main.py qr_dqn single_room --number_of_agents 1 --malmo_ports 10000 --ms_per_tick 75 --save_name qr_single_room --number_of_atoms 200 --double_dqn --epsilon_decay 20000 --success_replay_memory --replay_memory_size 10000

[Phantomb]

Oh that sounds like great performance! I am still running on commit 90e2eae, and it's unly just now starting to pick up success percentage after >350k steps..

I'll give it a try.

[Phantomb]

Pushed a fix. I see on my end that it solves within 50k steps.
The commandline I used:
python3.6 main.py qr_dqn single_room --number_of_agents 1 --malmo_ports 10000 --ms_per_tick 75 --save_name qr_single_room --number_of_atoms 200 --double_dqn --epsilon_decay 20000 --success_replay_memory --replay_memory_size 10000

Interestingly enough, using the exact same command line on the latest commit, after 210k steps mine still hadn't finished... :

I am going to run it some more times, but I'm wondering what might cause this discrepancy with your results.

[Phantomb]

Ran two more times for over >150k steps on two different machines, I get very similar results to my first try.

@tesslerc What do you think might cause this difference in performance between you and me?

[tesslerc]

This is really weird behavior...
Can you also try with the flag '--normalize_reward'
While in theory, this shouldn't change anything... it might help convergence due to the network learning values on a lower scale.

[Phantomb]

I'll try that now and will report back.