Terminal states logic for sub-policies

[ysaibhargav]

mlsh/mlsh_code/rollouts.py

Line 123 in 58f527a

nonterminal = 1-new[t+1]

Hi, shouldn't the logic for determining terminal states for sub-policies consider the case where the master action changes? If the action changes, shouldn't we designate the current state as terminal? It seems that the current implementation can bootstrap from a different sub-policy network when such a case arises.

[AaronHavens]

I agree with you. We probably shouldn't be rolling in the expectation of a different policy into a sub-policy advantage. They are practically different MDP's.