Refine find_nexus_module search locations and error handling#5767
Refine find_nexus_module search locations and error handling#5767prckent merged 10 commits intoQMCPACK:developfrom
Conversation
|
Test this please |
brockdyer03
left a comment
There was a problem hiding this comment.
As best I can tell, while this code does unify the find_nexus_modules function, it doesn't do anything to stop enforcement of the location of the executables being with the rest of Nexus. Perhaps a separate bit of code that checks if you're in a QMCPACK install directory would fix it?
nexus/bin/nexus_modules.py
Outdated
| nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..')) | ||
|
|
||
| # ensure closely coupled nexus module is in use. | ||
| if nexus_lib != nexus_lib2: |
There was a problem hiding this comment.
This is the line that is probably going to continue failing if the executables get moved. The enforcement of location of the current file with the location of Nexus means that this will likely still result in a failure if the executables are moved.
There was a problem hiding this comment.
Unfortunately that is a current requirement.
There was a problem hiding this comment.
We should keep the strict association in this PR. We can remove it in a following PR once a decision is made.
|
The introduced Line 116 in 01c9846 |
149eedb to
5d9f62e
Compare
|
To be honest I forgot about I think since @jtkrogel said he was fine with most of the executables not actually enforcing association with the rest of Nexus (except for |
…y nexus exectuables.
5d9f62e to
0441e82
Compare
You analysis is correct
simply avoid files starting with underscore. taken care by #5768
I actually prefer all the nexus executable runnable without setting |
|
Need to catch up with the discussion here. From what I understand of our prior discussion, no decisions had been made about enforcing strict association between the binaries and nexus modules (though I favor retaining it). |
|
On a closer look, I really don't see the virtue of including a non-binary ( |
I think this could potentially introduce unwanted side effects. For example, |
prckent
left a comment
There was a problem hiding this comment.
nexus/bin is for directly shell executable programs; putting a not directly callable function there is non-standard and will confuse users (etc.). Could it be made a verify_nexus_installation utility that can also be called directly? If this route works, we could expand it subsequently to include a reporting of the various nexus required and optional package imports.
4657b08 to
139fac4
Compare
|
@prckent |
139fac4 to
5eec27e
Compare
5eec27e to
646ad70
Compare
|
It is the job of |
nexus/bin/nexus_modules.py
Outdated
| nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..')) | ||
|
|
||
| # ensure closely coupled nexus module is in use. | ||
| if nexus_lib != nexus_lib2: |
There was a problem hiding this comment.
We should keep the strict association in this PR. We can remove it in a following PR once a decision is made.
| from time import time # used to get user wall clock rather than cpu time | ||
| tstart_all = time() | ||
|
|
||
| testing_wrong_nexus_install = False |
There was a problem hiding this comment.
Please reply. This should not be removed.
There was a problem hiding this comment.
The replaced find_nexus_module did the work already.
There was a problem hiding this comment.
Due to the current requirement of nxs-test that exception should not be raise by itself. The gold implementation can not be used and kept by nxs-test. I reverted most of the changes in nxs-test.
nexus/bin/verify_nexus_installation
Outdated
| @@ -0,0 +1,38 @@ | |||
| #! /usr/bin/env python3 | |||
There was a problem hiding this comment.
Should be removed. See comment in the conversation/chat.
Currently nxs-test needs extra work to have a proper installation. Consider the added tool a stopgap utility. |
|
The implementation of Here is an updated version that solves these problems: def find_nexus_modules():
import os
import sys
# Prepend the assumed path of a folder containing the closely coupled nexus module to module search path
# It points to the top Nexus directory not necessarily the top QMCPACK directory.
nexus_lib = os.path.realpath(os.path.join(__file__,'..','..'))
# "import nexus" will fail due to missing module files if nexus/__init__.py does not exists
nexus_init = os.path.join(nexus_lib,'nexus','__init__.py')
if not os.path.exists(nexus_init):
print('\nNexus modules are required but cannot be imported!'
'\nMake sure Nexus modules are available at:'
'\n '+nexus_lib+'\n')
sys.exit(1)
# check import of nexus modules
sys.path.insert(0,nexus_lib)
try:
import nexus
except Exception as err:
print('\nImporting Nexus modules failed for an unknown reason.'
'\nThis is likely a developer error.'
'\nFor more information, the original exception is reproduced below.'
'\nPlease report this error to the Nexus developers.'+2*'\n')
import traceback
exc_type, exc_value, exc_traceback = sys.exc_info()
exc_lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
for line in exc_lines:
print(line.rstrip())
print()
sys.exit(1)
# ensure closely coupled nexus module is in use.
nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..'))
if nexus_lib != nexus_lib2:
exe = sys.argv[0].split('/')[-1]
exe_full_path = os.path.abspath(exe)
print('\nBroken Nexus installation!'
+'\n {} is required to reside in the Nexus module directory.'.format(exe)
+'\n {} location {} : {}'.format(exe,' '*max(0,8-len(exe)),exe_full_path)
+'\n Nexus module directory : ' + nexus_lib2+'\n')
sys.exit(1)
#end def find_nexus_modules |
|
Here is a comparison of the error messages the user will see based on the original and updated Nexus modules aren't located where qmca needs them Original: Updated: |
|
Nexus modules have a bad import internally Original: Updated: |
|
Nexus modules execute bad code during import: Original: Updated: |
|
Broken install Original: Updated: |
|
Above are comparisons of original and updated code for
The error messages are more concise and tell the user what to do. With minor modifications, this version might be sufficient for |
b13e284 to
0df99b5
Compare
|
@prckent ping |
|
I will test this "by hand". |
There was a problem hiding this comment.
In my testing, running qmca directly without any PATH or PYTHONPATH set works correctly, as does using a link to qmca. Copying qmca elsewhere and using it correctly gives an error. This clearly a nice improvement. However, I think we can use this opportunity to further improve the error messages for users:
qmcuser@9abc81ade268:~/test_simple_H2O$ $HOME/qmcpack/nexus/bin/qmca -q ev -e 200 --sac LiH.s002.dmc.dat
LocalEnergy Variance ratio
LiH series 2 -8.541570 +/- 0.005672 16.4 0.142099 +/- 0.002624 6.3 0.0166
qmcuser@9abc81ade268:~/test_simple_H2O$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/qe/bin
qmcuser@9abc81ade268:~/test_simple_H2O$ echo $PYTHONPATH
/opt/pyscf:/home/qmcuser/qmcpack/src/QMCTools
qmcuser@9abc81ade268:~/test_simple_H2O$ ln -s $HOME/qmcpack/nexus/bin/qmca .
qmcuser@9abc81ade268:~/test_simple_H2O$ ./qmca -q ev -e 200 --sac LiH.s002.dmc.dat
LocalEnergy Variance ratio
LiH series 2 -8.541570 +/- 0.005672 16.4 0.142099 +/- 0.002624 6.3 0.0166
qmcuser@9abc81ade268:~/test_simple_H2O$ rm -f qmca
qmcuser@9abc81ade268:~/test_simple_H2O$ cp ~/qmcpack/nexus/bin/qmca .
qmcuser@9abc81ade268:~/test_simple_H2O$ ./qmca -q ev -e 200 --sac LiH.s002.dmc.dat
File /home/qmcuser/nexus/__init__.py does not exist!
Broken Nexus installation!
Make sure Nexus modules are available at:
/home/qmcuser
I suggest the error be expanded to something like the following. We can not expect users to be familiar with file layout in modules, so providing more info should help.
Expected file /home/qmcuser/nexus/__init__.py does not exist!
Broken Nexus installation! All expected files not found.
Make sure Nexus modules are available at:
/home/qmcuser
Expecting to find files including
/home/qmcuser/nexus/__init__.py
/home/qmcuser/nexus/basis.py
|
@prckent I agree that we can not expect users to be familiar with file layout in modules. can potentially encourage users copying two mentioned files. It won't work well. How about the following? The first line must be kept reflecting the exact failed check and also indicating a correct installation should provide that file. |
|
Good improvement. https://nexus-workflows.readthedocs.io/en/latest/installation.html seems sufficiently up to date. Could be expanded e.g. when the installer gets updated. |
|
@prckent updated. |
|
Test this please |
That part of the manual will be almost completely rewritten when the packaging PR (#5742) gets merged, all in reStructuredText. I'd advise waiting for a rewrite until after that PR is merged if at all possible. |
Proposed changes
Currently, nexus executables require a closely coupled nexus modules, namely
find_nexus_modulehonors this layout and verifiesnexusmodule import.qmca has the gold implementation and other scripts follow.
I also corrected the following discovery path.
nexus internal lookup should not go beyond
<nexus_root>. Simply useWhat type(s) of changes does this code introduce?
Does this introduce a breaking change?
What systems has this change been tested on?
laptop
Checklist