Hanging after keyboard interrupt during jdk retrieval

[gselzer]

This error is likely pretty rare, but recording it anyways. Not a priority to fix.

We've recently started using cjdk in napari-imagej, and with this change I've been re-evaluating our installation process.

I ran one test with an environment where pyimagej was installed from Github. When napari-imagej launched, I used Ctrl+C to close napari, which shut down during JDK extraction process. I then tried restarting napari (after installing a different pyimagej version as a part of my testing) and now CDK hangs as it waits for that download+extraction to complete, throwing an error after a 5 minute timeout.

Of course, I can wipe the cjdk cache to fix this, but maybe we could do better?

 napari-imagej  napari --with napari-imagej
cjdk: Installing JDK zulu-jre:11.0.28 to C:\Users\gjselzer\AppData\Local\cjdk\cache
Download 100% of  38.8 MiB |#########################################################################################################################################################################################| Elapsed Time: 0:00:02 Time:  0:00:02
Extract  12% (50 of 415) |######################                                                                                                                                                                     | Elapsed Time: 0:00:00 ETA:   0:00:00Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\gjselzer\micromamba\Scripts\napari.exe\__main__.py", line 7, in <module>
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\__main__.py", line 566, in main
    _run()
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\__main__.py", line 385, in _run
    run(gui_exceptions=True)
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\_qt\qt_event_loop.py", line 408, in run
    with (
  File "C:\Users\gjselzer\micromamba\Lib\contextlib.py", line 144, in __exit__
    next(self.gen)
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\_qt\utils.py", line 393, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)
KeyboardInterrupt
Extract  17% (72 of 415) |################################                                                                                                                                                           | Elapsed Time: 0:00:00 ETA:   0:00:00Traceback (most recent call last):
  File "C:\Users\gjselzer\micromamba\Lib\shutil.py", line 633, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\gjselzer\\AppData\\Local\\Temp\\cjdk-1ppwkdx_\\archive.zip'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\gjselzer\micromamba\Lib\weakref.py", line 666, in _exitfunc
    f()
  File "C:\Users\gjselzer\micromamba\Lib\weakref.py", line 590, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gjselzer\micromamba\Lib\tempfile.py", line 935, in _cleanup
    cls._rmtree(name, ignore_errors=ignore_errors)
  File "C:\Users\gjselzer\micromamba\Lib\tempfile.py", line 930, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
  File "C:\Users\gjselzer\micromamba\Lib\shutil.py", line 781, in rmtree
    return _rmtree_unsafe(path, onexc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\gjselzer\micromamba\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.unlink, fullname, err)
  File "C:\Users\gjselzer\micromamba\Lib\tempfile.py", line 905, in onexc
    _os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\gjselzer\\AppData\\Local\\Temp\\cjdk-1ppwkdx_\\archive.zip'
 napari-imagej  ^C
 napari-imagej  pip install pyimagej==1.7.0
Collecting pyimagej==1.7.0
.
.
.
Successfully installed pyimagej-1.7.0
 napari-imagej  napari --with napari-imagej
Already downloading; waiting / |                                                                                                 #                                                                                                | 0 Elapsed Time: 0:05:00
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\gjselzer\micromamba\Scripts\napari.exe\__main__.py", line 7, in <module>
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\__main__.py", line 566, in main
    _run()
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\__main__.py", line 385, in _run
    run(gui_exceptions=True)
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\_qt\qt_event_loop.py", line 408, in run
    with (
  File "C:\Users\gjselzer\micromamba\Lib\contextlib.py", line 144, in __exit__
    next(self.gen)
  File "C:\Users\gjselzer\micromamba\Lib\site-packages\napari\_qt\utils.py", line 393, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)
KeyboardInterrupt

[marktsuchida]

Thanks. Do I understand right that stopping cjdk with KeyboardInterrupt during extraction causes it to hang the next time it is run with the same request? Is this calling cjdk via the CLI or the Python API? Does it reproduce with just cjdk (in which case those tracebacks would be more useful)?

[marktsuchida]

The proximal cause is that downloads use URL-keyed temporary directories as destination. If you try to start a second download of the same URL (in a separate process or thread), it will first wait (with the timeout) for that temp dir to vanish, so that multiple downloads for the same item won't occur. (Directory creation and deletion is globally atomic.)

Even on Windows (where open files cannot be deleted), there is (usually) no problem caused by CTRL-C during a download. We correctly clean up the temporary directory when KeyboardInterrupt is raised.

But here the file deletion fails, despite being correctly closed. It turns out that Windows Antivirus can cause this, because it triggers off of written files being closed and starts scanning them. Even though Antivirus opens the file with the FILE_SHARE_DELETE flag, the stanard way to delete a file (used by os.unlink(), called by shutil.rmtree()), which is Win32 DeleteFile() does not use FILE_SHARE_DELETE, and the sharing check is bidirectional so it fails.

The heavy file writing load could be why the behavior gets triggered in the context of napari-imagej; I've never seen it with the cjdk CLI. That's complete speculation, though.

Working on a workaround.

[marktsuchida]

As mentioned in #35, the Antivirus thing is a red herring and the cause is that QThread threads are abruptly terminated on program exit. The correct fix is for (1) cjdk to implement some API to allow cancellation of an in-progress install and (2) for user code to cancel the operation and wait for the QThread to exit before the program exits (aboutToQuit signal in the case of Qt). Keeping this open until we have (1).