SQL algorithm is not working

[ivan-ji-walmart]

Seems there are some error with SQL algorithm. I can run SAC but not SQL

ERROR trial_runner.py:567 -- Trial id=a3651_00000-seed=1911: Error processing event.
Traceback (most recent call last):
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 515, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 488, in fetch_result
result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/worker.py", line 1428, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::ExperimentRunner.train() (pid=24394, ip=10.22.134.202)
File "python/ray/_raylet.pyx", line 484, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 438, in ray._raylet.execute_task.function_executor
File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/ray/tune/trainable.py", line 336, in train
result = self.step()
File "/home/app/softlearning/examples/development/main.py", line 101, in step
diagnostics = next(self.train_generator)
File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 182, in _train
timestep=self._total_timestep)
File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 346, in _do_training_repeats
for i in range(self._n_train_repeat)
File "/home/app/softlearning/softlearning/algorithms/rl_algorithm.py", line 346, in
for i in range(self._n_train_repeat)
File "/home/app/softlearning/softlearning/algorithms/sql.py", line 356, in _do_training
training_diagnostics = self._do_updates(batch)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in call
result = self._call(*args, **kwds)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize
*args, **kwds))
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected
graph_function, _, _ = self._maybe_define_function(args, kwargs)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn
return weak_wrapped_fn().wrapped(*args, **kwds)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 3735, in bound_method_wrapper
return wrapped_fn(*args, **kwargs)
File "/opt/rh/rh-python36/root/usr/lib64/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper
raise e.ag_error_metadata.to_exception(e)
TypeError: in user code:

/home/app/softlearning/softlearning/algorithms/sql.py:338 _do_updates  *
    Qs_values, Qs_losses = self._update_Q(
/home/app/softlearning/softlearning/algorithms/sql.py:205 _update_Q  *
    Q_observations = {

TypeError: 'NoneType' object is not iterable

[hartikainen]

Hey @ivan-ji-walmart, thanks for opening this issue! Sorry about this. Some of my recent changes must have broken the SQL implementation and has left unnoticed since I haven't been using SQL in a long time. I'll try to get this fixed in the upcoming weeks, but in the meanwhile, I'd highly recommend using the official reference implementation found at https://github.com/haarnoja/softqlearning.