Build not showing agent

[NicolasKuske]

Dear all,

when I run your obsta.exe build (windows), I can play the game and see everything. However, when the build window opens after I instantiated it in a UnityEnvironment it remains blank for all times except for the one frame when I do an img.show call. Is that normal and is there any way to see the agent in real time doing its thing?

Best wishes

Nicolas

[MarcoMeter]

Hi @NicolasKuske

can you share the code that you use to reproduce this issue?

[NicolasKuske]

Hi Marco,

the code is the very beginning from the basic_usage notebook you provide.

from obstacle_tower_env import ObstacleTowerEnv
%matplotlib inline
from matplotlib import pyplot as plt

env = ObstacleTowerEnv(retro=False, realtime_mode=True)

Outputs:
INFO:mlagents_envs.environment:Connected to Unity environment with package version 1.2.0-preview and communication version 1.0.0
INFO:mlagents_envs.environment:Connected new brain:
ObstacleTowerAgent-v4.1?team=0
INFO:gym_unity:1 agents within environment.

And opens the pop up window.

The conda environment dependencies are:

• asttokens=2.0.5=pyhd3eb1b0_0
• backcall=0.2.0=pyhd3eb1b0_0
• ca-certificates=2024.3.11=haa95532_0
• colorama=0.4.6=py38haa95532_0
• comm=0.2.1=py38haa95532_0
• debugpy=1.6.7=py38hd77b12b_0
• decorator=5.1.1=pyhd3eb1b0_0
• executing=0.8.3=pyhd3eb1b0_0
• importlib_metadata=7.0.1=hd3eb1b0_0
• ipykernel=6.28.0=py38haa95532_0
• ipython=8.12.2=py38haa95532_0
• jedi=0.18.1=py38haa95532_1
• jupyter_client=8.6.0=py38haa95532_0
• jupyter_core=5.5.0=py38haa95532_0
• libffi=3.4.4=hd77b12b_1
• libsodium=1.0.18=h62dcd97_0
• matplotlib-inline=0.1.6=py38haa95532_0
• nest-asyncio=1.6.0=py38haa95532_0
• openssl=3.0.13=h2bbff1b_1
• packaging=23.2=py38haa95532_0
• parso=0.8.3=pyhd3eb1b0_0
• pickleshare=0.7.5=pyhd3eb1b0_1003
• pip=24.0=py38haa95532_0
• platformdirs=3.10.0=py38haa95532_0
• prompt-toolkit=3.0.43=py38haa95532_0
• psutil=5.9.0=py38h2bbff1b_0
• pure_eval=0.2.2=pyhd3eb1b0_0
• pygments=2.15.1=py38haa95532_1
• python=3.8.19=h1aa4202_0
• python-dateutil=2.9.0post0=py38haa95532_0
• pywin32=305=py38h2bbff1b_0
• pyzmq=25.1.2=py38hd77b12b_0
• setuptools=69.5.1=py38haa95532_0
• six=1.16.0=pyhd3eb1b0_1
• sqlite=3.45.3=h2bbff1b_0
• stack_data=0.2.0=pyhd3eb1b0_0
• tornado=6.3.3=py38h2bbff1b_0
• traitlets=5.7.1=py38haa95532_0
• typing_extensions=4.11.0=py38haa95532_0
• vc=14.2=h21ff451_1
• vs2015_runtime=14.27.29016=h5e58377_2
• wcwidth=0.2.5=pyhd3eb1b0_0
• wheel=0.43.0=py38haa95532_0
• zeromq=4.3.5=hd77b12b_0
• pip:
  • absl-py==2.1.0
  • anyio==3.7.1
  • argon2-cffi==23.1.0
  • argon2-cffi-bindings==21.2.0
  • astunparse==1.6.3
  • attrs==23.2.0
  • beautifulsoup4==4.12.3
  • bleach==6.1.0
  • cachetools==5.3.3
  • cattrs==1.5.0
  • certifi==2024.2.2
  • cffi==1.16.0
  • charset-normalizer==3.3.2
  • cloudpickle==3.0.0
  • contourpy==1.1.1
  • cycler==0.12.1
  • defusedxml==0.7.1
  • exceptiongroup==1.2.1
  • fastjsonschema==2.19.1
  • filelock==3.14.0
  • flatbuffers==2.0.7
  • fonttools==4.51.0
  • gast==0.4.0
  • google-auth==2.29.0
  • google-auth-oauthlib==1.0.0
  • google-pasta==0.2.0
  • grpcio==1.63.0
  • gym==0.26.2
  • gym-notices==0.0.8
  • h5py==3.11.0
  • idna==3.7
  • importlib-metadata==7.1.0
  • importlib-resources==6.4.0
  • jinja2==3.1.4
  • jsonschema==4.22.0
  • jsonschema-specifications==2023.12.1
  • jupyter-server==1.24.0
  • jupyterlab-pygments==0.3.0
  • keras==2.7.0
  • keras-preprocessing==1.1.2
  • kiwisolver==1.4.5
  • libclang==18.1.1
  • markdown==3.6
  • markupsafe==2.1.5
  • matplotlib==3.5.0
  • mistune==3.0.2
  • mlagents==0.18.1
  • mlagents-envs==0.18.1
  • nbclient==0.10.0
  • nbconvert==7.16.4
  • nbformat==5.10.4
  • numpy==1.18.5
  • oauthlib==3.2.2
  • opt-einsum==3.3.0
  • pandocfilters==1.5.1
  • pettingzoo==1.15.0
  • pillow==10.3.0
  • pkgutil-resolve-name==1.3.10
  • prometheus-client==0.20.0
  • protobuf==3.19.6
  • pyasn1==0.6.0
  • pyasn1-modules==0.4.0
  • pycparser==2.22
  • pyparsing==3.1.2
  • pypiwin32==223
  • pywinpty==2.0.13
  • pyyaml==6.0.1
  • referencing==0.35.1
  • requests==2.31.0
  • requests-oauthlib==2.0.0
  • rpds-py==0.18.1
  • rsa==4.9
  • send2trash==1.8.3
  • setuptools-scm==8.1.0
  • sniffio==1.3.1
  • soupsieve==2.5
  • tensorboard==2.14.0
  • tensorboard-data-server==0.7.2
  • tensorflow==2.7.4
  • tensorflow-estimator==2.7.0
  • tensorflow-io-gcs-filesystem==0.31.0
  • termcolor==2.4.0
  • terminado==0.18.1
  • tinycss2==1.3.0
  • tomli==2.0.1
  • urllib3==2.2.1
  • webencodings==0.5.1
  • websocket-client==1.8.0
  • werkzeug==3.0.3
  • wrapt==1.16.0
  • zipp==3.18.1

[NicolasKuske]

Also env.observation_space independent of retro=False or retro=True always returns
Tuple(Box(0.0, 1.0, (84, 84, 3), float32), Discrete(5), Box(0.0, 3.4028235e+38, (1,), float32), Discrete(9999))

Furthermore the cloned repository is only 2.5 MB size. It seems as if the call

env = ObstacleTowerEnv(retro=False, realtime_mode=True)

never initializes the download of the tower.exe build.

I downloaded the file manually and it is still in my downloads folder. My guess is that the cloned repository and the downloaded build never interact...

[MarcoMeter]

The best way to init the environment is to specify the file path:

env = ObstacleTowerEnv(environment_filename="path to ...")

I wouldn't rely on the automatic download as you might need to change permission on the executable on Linux systems.

As long as the returned visual observations of the reset and step functions are correct, then there is nothing to worry about. Don't care about the actual window, which is opened by the Unity build itself.

If the instance of the build and python don't connect, you should receive a timeout on the python side.

[NicolasKuske]

thank you, although I believe in general it should be possible to have the window display what the agent is currently seeing and doing. I mean that seems to be the reason for the realtime_mode option.