micropy stubs create fails after verifying integrity

[romankulev]

Describe the bug
When I try to create stub from my esp32 i get this error (saved all output in error.txt):
KeyError: 'sysname'
error.txt

Steps to Reproduce

1.Plug esp32 to computer, flash micropython binary
2.run micropy stubs create /dev/ttyUSB0

Logs

attached micropy.log file
micropy.log

Context (Environment)

• OS: Endeavour OS
• Micropy Version: 4.2.1
• Python Version: 3.11.3

[skipper85]

Hello @romankulev , did you manage to solve this? I experience the same issue.

[BradenM]

Hi @skipper85, @romankulev

Sorry for the late reply.

Have either of you tried using the rshell backend as an alternative to the upysh backend?

See micropy stubs create --help for details

[skipper85]

Hi @BradenM ,

Thank you for your response.
I tried today with the rshell backend, but then I face a different error.
i use micropy in a virtual environment.

Python version : 3.10.12
rshell : 0.0.31
micropy : 0.6.2
micropython bin : ESP32_GENERIC_S2-20230426-v1.20.0.bin

micropy stubs create /dev/ttyUSB0 --backend rshell

MicroPy  Connecting to Pyboard @ /dev/ttyUSB0
MicroPy  ✔ Connected!
2023-09-08 10:30:12.881 | DEBUG    | stubber.minify:minify_script:253 - Original length : 26905
2023-09-08 10:30:12.881 | INFO     | stubber.minify:minify_script:254 - Minified length : 11631
2023-09-08 10:30:12.881 | INFO     | stubber.minify:minify_script:255 - Reduced by      : 15274 
2023-09-08 10:30:13.040 | DEBUG    | stubber.minify:cross_compile:328 - mpy-cross compiled to    : mpy_cross_mxm05fa9.mpy
MicroPy  Executing stubber on pyboard...
MicroPy  Failed to execute script: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
MicroPy  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/app/stubs.py:179 in │
│ stubs_create                                                                                     │
│                                                                                                  │
│   176 │   dev_path = DevicePath("createstubs.mpy") if compile else DevicePath("createstubs.py"   │
│   177 │   log.info("Executing stubber on pyboard...")                                            │
│   178 │   try:                                                                                   │
│ ❱ 179 │   │   pyb.run_script(create_stubs, DevicePath(dev_path))                                 │
│   180 │   except Exception as e:                                                                 │
│   181 │   │   # TODO: Handle more usage cases                                                    │
│   182 │   │   log.error(f"Failed to execute script: {str(e)}", exception=e)                      │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        _get_desc = <function stubs_create.<locals>._get_desc at 0x7ffbb65c2a70>              │ │
│ │          backend = <CreateBackend.rshell: 'rshell'>                                          │ │
│ │          compile = True                                                                      │ │
│ │     create_stubs = <_io.BytesIO object at 0x7ffbb51186d0>                                    │ │
│ │              ctx = <click.core.Context object at 0x7ffbb65d38e0>                             │ │
│ │         dev_path = 'createstubs.mpy'                                                         │ │
│ │          exclude = []                                                                        │ │
│ │ exclude_defaults = True                                                                      │ │
│ │              log = <micropy.logger.ServiceLog object at 0x7ffbb65d0040>                      │ │
│ │  message_handler = MessageHandlers(                                                          │ │
│ │                    │   on_message=<function stubs_create.<locals>.<lambda> at                │ │
│ │                    0x7ffbb65c3250>,                                                          │ │
│ │                    )                                                                         │ │
│ │           module = []                                                                        │ │
│ │  module_defaults = True                                                                      │ │
│ │               mp = <micropy.main.MicroPy object at 0x7ffbb6547ee0>                           │ │
│ │             port = '/dev/ttyUSB0'                                                            │ │
│ │              pyb = <micropy.pyd.pydevice.PyDevice object at 0x7ffbb65d3700>                  │ │
│ │          pyb_log = <micropy.logger.ServiceLog object at 0x7ffbb65d3d30>                      │ │
│ │          variant = <CreateStubsVariant.BASE: 'base'>                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/pyd/pydevice.py:94  │
│ in run_script                                                                                    │
│                                                                                                  │
│   91 │   │   │   )                                                                               │
│   92 │   │   │   else content.read()                                                             │
│   93 │   │   )                                                                                   │
│ ❱ 94 │   │   return self.pydevice.eval_script(_content, target_path, consumer=self.consumer)     │
│   95 │                                                                                           │
│   96 │   def run(self, content: str) -> str | None:                                              │
│   97 │   │   return self.pydevice.eval(content, consumer=self.consumer)                          │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    _content = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │     content = <_io.BytesIO object at 0x7ffbb51186d0>                                         │ │
│ │        self = <micropy.pyd.pydevice.PyDevice object at 0x7ffbb65d3700>                       │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /home/gerard/Documents/test_esp32s2/env/lib/python3.10/site-packages/micropy/pyd/backend_rshell. │
│ py:225 in eval_script                                                                            │
│                                                                                                  │
│   222 │   ):                                                                                     │
│   223 │   │   _contents: str | bytes = contents                                                  │
│   224 │   │   if isinstance(_contents, bytes):                                                   │
│ ❱ 225 │   │   │   _contents = _contents.decode()                                                 │
│   226 │   │   with self.repl():                                                                  │
│   227 │   │   │   try:                                                                           │
│   228 │   │   │   │   out_bytes = self.eval(_contents, consumer=consumer)                        │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │   _contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │    consumer = <micropy.pyd.consumers.ConsumerDelegate object at 0x7ffbb65d37c0>              │ │
│ │    contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │        self = <micropy.pyd.backend_rshell.RShellPyDeviceBackend object at 0x7ffbb65d3b20>    │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

[VirtualWolf]

Same issue on both counts for me... without --backend rshell I get a whole ton of reboots (which might be some other issue) before it eventually fails with the KeyError: 'sysname' error: micropy.txt

And with --backend rshell I get the same invalid start byte error as above:

$ micropy stubs create /dev/tty.SLAB_USBtoUART --backend rshell

MicroPy  Connecting to Pyboard @ /dev/tty.SLAB_USBtoUART
MicroPy  ✔ Connected!
2024-04-02 21:46:34.866 | DEBUG    | stubber.minify:minify_script:253 - Original length : 26905
2024-04-02 21:46:34.867 | INFO     | stubber.minify:minify_script:254 - Minified length : 11631
2024-04-02 21:46:34.867 | INFO     | stubber.minify:minify_script:255 - Reduced by      : 15274 
2024-04-02 21:46:35.300 | DEBUG    | stubber.minify:cross_compile:328 - mpy-cross compiled to    : mpy_cross_a5xj9gos.mpy
MicroPy  Executing stubber on pyboard...
MicroPy  Failed to execute script: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
MicroPy  UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/app/stubs.py:17 │
│ 9 in stubs_create                                                                                │
│                                                                                                  │
│   176 │   dev_path = DevicePath("createstubs.mpy") if compile else DevicePath("createstubs.py"   │
│   177 │   log.info("Executing stubber on pyboard...")                                            │
│   178 │   try:                                                                                   │
│ ❱ 179 │   │   pyb.run_script(create_stubs, DevicePath(dev_path))                                 │
│   180 │   except Exception as e:                                                                 │
│   181 │   │   # TODO: Handle more usage cases                                                    │
│   182 │   │   log.error(f"Failed to execute script: {str(e)}", exception=e)                      │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │        _get_desc = <function stubs_create.<locals>._get_desc at 0x1185b7eb0>                 │ │
│ │          backend = <CreateBackend.rshell: 'rshell'>                                          │ │
│ │          compile = True                                                                      │ │
│ │     create_stubs = <_io.BytesIO object at 0x118ac4860>                                       │ │
│ │              ctx = <click.core.Context object at 0x11862b6a0>                                │ │
│ │         dev_path = 'createstubs.mpy'                                                         │ │
│ │          exclude = []                                                                        │ │
│ │ exclude_defaults = True                                                                      │ │
│ │              log = <micropy.logger.ServiceLog object at 0x11847b880>                         │ │
│ │  message_handler = MessageHandlers(                                                          │ │
│ │                    │   on_message=<function stubs_create.<locals>.<lambda> at 0x1185fce50>,  │ │
│ │                    )                                                                         │ │
│ │           module = []                                                                        │ │
│ │  module_defaults = True                                                                      │ │
│ │               mp = <micropy.main.MicroPy object at 0x11847b1f0>                              │ │
│ │             port = '/dev/tty.SLAB_USBtoUART'                                                 │ │
│ │              pyb = <micropy.pyd.pydevice.PyDevice object at 0x11862ac20>                     │ │
│ │          pyb_log = <micropy.logger.ServiceLog object at 0x11862abf0>                         │ │
│ │          variant = <CreateStubsVariant.BASE: 'base'>                                         │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/pyd/pydevice.py │
│ :94 in run_script                                                                                │
│                                                                                                  │
│   91 │   │   │   )                                                                               │
│   92 │   │   │   else content.read()                                                             │
│   93 │   │   )                                                                                   │
│ ❱ 94 │   │   return self.pydevice.eval_script(_content, target_path, consumer=self.consumer)     │
│   95 │                                                                                           │
│   96 │   def run(self, content: str) -> str | None:                                              │
│   97 │   │   return self.pydevice.eval(content, consumer=self.consumer)                          │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │    _content = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │     content = <_io.BytesIO object at 0x118ac4860>                                            │ │
│ │        self = <micropy.pyd.pydevice.PyDevice object at 0x11862ac20>                          │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/virtualwolf/Downloads/stubbing/.venv/lib/python3.10/site-packages/micropy/pyd/backend_rsh │
│ ell.py:225 in eval_script                                                                        │
│                                                                                                  │
│   222 │   ):                                                                                     │
│   223 │   │   _contents: str | bytes = contents                                                  │
│   224 │   │   if isinstance(_contents, bytes):                                                   │
│ ❱ 225 │   │   │   _contents = _contents.decode()                                                 │
│   226 │   │   with self.repl():                                                                  │
│   227 │   │   │   try:                                                                           │
│   228 │   │   │   │   out_bytes = self.eval(_contents, consumer=consumer)                        │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │   _contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │    consumer = <micropy.pyd.consumers.ConsumerDelegate object at 0x11862ac80>                 │ │
│ │    contents = b'M\x06\x00\x1f\x83\x19u\x1ccreatestubs.py\x00\x0f\x06pyb\x00\n{}/{}\x00\x0el… │ │
│ │        self = <micropy.pyd.backend_rshell.RShellPyDeviceBackend object at 0x11862b7c0>       │ │
│ │ target_path = 'createstubs.mpy'                                                              │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 4: invalid start byte

Environment

• Board — Adafruit HUZZAH32
• MicroPython version — 1.22.2, ESP32_GENERIC firmware
• OS — macOS 14.4.1 on Apple Silicon
• micropy-cli version — 4.2.2
• Python version — 3.10.13 in a virtualenv

[kheraud]

I have the same issue with a KeyError: 'sysname' . Looking at the stacktrace, it seems that the chip discovery outputs a dict with missing "sysname" key. Is it just required to auto-name the stub ?

~/.asdf/installs/python/3.11.8/lib/python3.11/site-packages/micropy/stubs/stubs.py:3 │
│ 60 in from_stubber                                                                               │
│                                                                                                  │
│   357 │   │   mod_data = json.load(mod_file.open())                                              │
│   358 │   │   dev_fware = mod_data["firmware"]                                                   │
│   359 │   │   fname = dev_fware.get("name", None)                                                │
│ ❱ 360 │   │   out_name = f"{dev_fware['sysname']}"                                               │
│   361 │   │   # TODO: Attempt to Autoresolve Firmware name and add it to info.json               │
│   362 │   │   if fname:                                                                          │
│   363 │   │   │   out_name = f"{out_name}-{fname}"                                               │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │     _path = PosixPath('/tmp/tmpmkabmyj4/stubs')                                              │ │
│ │      dest = PosixPath('/tmp/tmpmkabmyj4')                                                    │ │
│ │ dev_fware = {                                                                                │ │
│ │             │   'version': '1.23.0.',                                                        │ │
│ │             │   'mpy': 'v6.3',                                                               │ │
│ │             │   'port': 'esp32',                                                             │ │
│ │             │   'board': 'Generic_ESP32_module_with_ESP32',                                  │ │
│ │             │   'family': 'micropython',                                                     │ │
│ │             │   'build': '',                                                                 │ │
│ │             │   'arch': 'xtensawin',                                                         │ │
│ │             │   'ver': 'v1.23.0.',                                                           │ │
│ │             │   'cpu': 'ESP32'                                                               │ │
│ │             }                                                                                │ │
│ │     fname = None                                                                             │ │
│ │  mod_data = {                                                                                │ │
│ │             │   'firmware': {                                                                │ │
│ │             │   │   'version': '1.23.0.',                                                    │ │
│ │             │   │   'mpy': 'v6.3',                                                           │ │
│ │             │   │   'port': 'esp32',                                                         │ │
│ │             │   │   'board': 'Generic_ESP32_module_with_ESP32',                              │ │
│ │             │   │   'family': 'micropython',                                                 │ │
│ │             │   │   'build': '',                                                             │ │
│ │             │   │   'arch': 'xtensawin',                                                     │ │
│ │             │   │   'ver': 'v1.23.0.',                                                       │ │
│ │             │   │   'cpu': 'ESP32'                                                           │ │
│ │             │   },                                                                           │ │
│ │             │   'stubber': {'version': 'v1.13.4'},                                           │ │
│ │             │   'stubtype': 'firmware',                                                      │ │
│ │             │   'modules': [                                                                 │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': '_onewire',                                                │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/_onewire.py'  │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': '_thread',                                                 │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/_thread.py'   │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'apa106',                                                  │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/apa106.py'    │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'array',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/array.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'binascii',                                                │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/binascii.py'  │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'bluetooth',                                               │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/bluetooth.py' │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'btree',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/btree.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'cmath',                                                   │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/cmath.py'     │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'collections',                                             │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/collections.… │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   {                                                                        │ │
│ │             │   │   │   'module': 'cryptolib',                                               │ │
│ │             │   │   │   'file':                                                              │ │
│ │             '/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_module_with_ESP32/cryptolib.py' │ │
│ │             │   │   },                                                                       │ │
│ │             │   │   ... +63                                                                  │ │
│ │             │   ]                                                                            │ │
│ │             }                                                                                │ │
│ │  mod_file = PosixPath('/tmp/tmpmkabmyj4/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_modu… │ │
│ │      path = PosixPath('/tmp/tmpmkabmyj4/stubs/micropython-v1_23_0_-esp32-Generic_ESP32_modu… │ │
│ │      self = <micropy.stubs.stubs.StubManager object at 0x7cd704f46b50>                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'sysname'

[BradenM]

Looks like the format changed in one of the recent stubber updates and somehow was missed in the unit tests.

That piece of code specifically (adapting the module format from stubber) is far from ideal and needs some improving. I will try and look into this soon.

[BradenM]

Not ideal of course, but a temporary workaround would be to modify the resulting "modules.json" file on the device by renaming "firmware.family" -> "firmware.name" and "firmware.board" -> "firmware.sysname" (or "firmware.port", either would work)