Skip to content

Refine find_nexus_module search locations and error handling#5767

Merged
prckent merged 10 commits intoQMCPACK:developfrom
ye-luo:consolidate-nexus-executables
Feb 6, 2026
Merged

Refine find_nexus_module search locations and error handling#5767
prckent merged 10 commits intoQMCPACK:developfrom
ye-luo:consolidate-nexus-executables

Conversation

@ye-luo
Copy link
Contributor

@ye-luo ye-luo commented Jan 28, 2026

Proposed changes

Currently, nexus executables require a closely coupled nexus modules, namely

<nexus_root>/bin # contains nexus exectuables
<nexus_root>/nexus #contains nexus module files

find_nexus_module honors this layout and verifies nexus module import.
qmca has the gold implementation and other scripts follow.

I also corrected the following discovery path.

nexus_lib = os.path.realpath(os.path.join(__file__,'..','..','..','nexus'))

nexus internal lookup should not go beyond <nexus_root>. Simply use

nexus_lib = os.path.realpath(os.path.join(__file__,'..','..'))

What type(s) of changes does this code introduce?

  • Refactoring (no functional changes, no api changes)
  • Other (please describe): more error traps

Does this introduce a breaking change?

  • No

What systems has this change been tested on?

laptop

Checklist

    • I have read the pull request guidance and develop docs
    • This PR is up to date with the current state of 'develop'
    • Code added or changed in the PR has been clang-formatted
    • This PR adds tests to cover any new code, or to catch a bug that is being fixed
    • Documentation has been added (if appropriate)

@ye-luo
Copy link
Contributor Author

ye-luo commented Jan 28, 2026

Test this please

Copy link
Contributor

@brockdyer03 brockdyer03 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As best I can tell, while this code does unify the find_nexus_modules function, it doesn't do anything to stop enforcement of the location of the executables being with the rest of Nexus. Perhaps a separate bit of code that checks if you're in a QMCPACK install directory would fix it?

nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..'))

# ensure closely coupled nexus module is in use.
if nexus_lib != nexus_lib2:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the line that is probably going to continue failing if the executables get moved. The enforcement of location of the current file with the location of Nexus means that this will likely still result in a failure if the executables are moved.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately that is a current requirement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep the strict association in this PR. We can remove it in a following PR once a decision is made.

@ye-luo ye-luo marked this pull request as draft January 29, 2026 00:45
@ye-luo
Copy link
Contributor Author

ye-luo commented Jan 29, 2026

The introduced nexus_modules.py generate __pycache__ and prevents executing nexus/install.

execute('cp {0}/bin/* {1}'.format(source_dir,install_dir))

err: b"cp: -r not specified; omitting directory '/home/yeluo/opt/qmcpack/nexus/bin/__pycache__'\n"

@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch 2 times, most recently from 149eedb to 5d9f62e Compare January 29, 2026 01:18
@brockdyer03
Copy link
Contributor

To be honest I forgot about __pycache__. That gets created whenever you import Python modules that are not your current module. Usually that only gets made in qmcpack/nexus/nexus, but since there was a .py file added to qmcpack/nexus/bin it'll make one if you run anything before doing the copy. I don't think that creating a separate .py file is a viable option.

I think since @jtkrogel said he was fine with most of the executables not actually enforcing association with the rest of Nexus (except for nxs-test) it would probably be fine to just delete find_nexus_modules from all of the files (again, except for nxs-test) and put the onus on the user to ensure their Python path is set properly.

@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch from 5d9f62e to 0441e82 Compare January 29, 2026 04:17
@ye-luo
Copy link
Contributor Author

ye-luo commented Jan 29, 2026

To be honest I forgot about __pycache__. That gets created whenever you import Python modules that are not your current module. Usually that only gets made in qmcpack/nexus/nexus, but since there was a .py file added to qmcpack/nexus/bin it'll make one if you run anything before doing the copy.

You analysis is correct

I don't think that creating a separate .py file is a viable option.

simply avoid files starting with underscore. taken care by #5768

I think since @jtkrogel said he was fine with most of the executables not actually enforcing association with the rest of Nexus (except for nxs-test) it would probably be fine to just delete find_nexus_modules from all of the files (again, except for nxs-test) and put the onus on the user to ensure their Python path is set properly.

I actually prefer all the nexus executable runnable without setting PYTHONPATH by the user or by a virtual environment. They must be exactly matched to the nexus module in the same git commit. The current strict coupling requirement made nexus executables invulnerable to hidden PYTHONPATH changes.

@ye-luo ye-luo marked this pull request as ready for review January 29, 2026 04:41
@jtkrogel
Copy link
Contributor

Need to catch up with the discussion here. From what I understand of our prior discussion, no decisions had been made about enforcing strict association between the binaries and nexus modules (though I favor retaining it).

@jtkrogel
Copy link
Contributor

On a closer look, I really don't see the virtue of including a non-binary (find_nexus_modules.py) in the binaries directory. I can understand the desire to remove all instances of code duplication, but I think the unwieldiness of this change is worse.

@brockdyer03
Copy link
Contributor

simply avoid files starting with underscore. taken care by #5768

I think this could potentially introduce unwanted side effects. For example, qmcpack/nexus/nexus has a file __init__.py which was renamed from nexus.py to be compliant with packaging guidelines. Without that file, the entire package will stop working, so if that copy script ever gets duplicated it will cause significant errors that will be hard to trace.

Copy link
Contributor

@prckent prckent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nexus/bin is for directly shell executable programs; putting a not directly callable function there is non-standard and will confuse users (etc.). Could it be made a verify_nexus_installation utility that can also be called directly? If this route works, we could expand it subsequently to include a reporting of the various nexus required and optional package imports.

@ye-luo ye-luo changed the title Introduce nexus_modules to handle consistent nexus module searching by nexus exectuables Refine find_nexus_module search locations and error handling Jan 29, 2026
@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch from 4657b08 to 139fac4 Compare January 29, 2026 18:51
@ye-luo
Copy link
Contributor Author

ye-luo commented Jan 29, 2026

@prckent verify_nexus_installation has been introduced. I couldn't find a clean way to avoid code duplication. Using sub-process to invoke verify_nexus_installation is still very messy. So just copied find_nexus_module into each script noting that the reference copy is from verify_nexus_installation

@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch from 139fac4 to 5eec27e Compare January 29, 2026 18:55
@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch from 5eec27e to 646ad70 Compare January 29, 2026 19:01
@ye-luo ye-luo added this to the v4.2.0 Release milestone Jan 29, 2026
@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 2, 2026

It is the job of nxs-test to validate the installation, so having verify_nexus_installation is redundant and misleading (it verifies nothing but paths while the installation could still be broken). The "gold" implementation of the path checking should reside in nxs-test.

nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..'))

# ensure closely coupled nexus module is in use.
if nexus_lib != nexus_lib2:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should keep the strict association in this PR. We can remove it in a following PR once a decision is made.

from time import time # used to get user wall clock rather than cpu time
tstart_all = time()

testing_wrong_nexus_install = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this deleted?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please reply. This should not be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The replaced find_nexus_module did the work already.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Due to the current requirement of nxs-test that exception should not be raise by itself. The gold implementation can not be used and kept by nxs-test. I reverted most of the changes in nxs-test.

@@ -0,0 +1,38 @@
#! /usr/bin/env python3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be removed. See comment in the conversation/chat.

@ye-luo
Copy link
Contributor Author

ye-luo commented Feb 2, 2026

It is the job of nxs-test to validate the installation, so having verify_nexus_installation is redundant and misleading (it verifies nothing but paths while the installation could still be broken). The "gold" implementation of the path checking should reside in nxs-test.

Currently nxs-test needs extra work to have a proper installation. Consider the added tool a stopgap utility.

@ye-luo
Copy link
Contributor Author

ye-luo commented Feb 3, 2026

@prckent I removed verify_nexus_Installation that @jtkrogel considered redundant and we had a hard time to agree on what exactly it should serve and where it should be hosted. The same check exists in every nexus binary should be sufficient to catch the issue and inform users.

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

The implementation of find_nexus_modules displays error messages unclear to the user. It also masks import errors internal to Nexus modules which misleads users.

Here is an updated version that solves these problems:

def find_nexus_modules():
    import os
    import sys
    # Prepend the assumed path of a folder containing the closely coupled nexus module to module search path
    # It points to the top Nexus directory not necessarily the top QMCPACK directory.
    nexus_lib = os.path.realpath(os.path.join(__file__,'..','..'))

    # "import nexus" will fail due to missing module files if nexus/__init__.py does not exists
    nexus_init = os.path.join(nexus_lib,'nexus','__init__.py')
    if not os.path.exists(nexus_init):
        print('\nNexus modules are required but cannot be imported!'
              '\nMake sure Nexus modules are available at:'
              '\n  '+nexus_lib+'\n')
        sys.exit(1)

    # check import of nexus modules
    sys.path.insert(0,nexus_lib)
    try:
        import nexus
    except Exception as err:
        print('\nImporting Nexus modules failed for an unknown reason.'
              '\nThis is likely a developer error.'
              '\nFor more information, the original exception is reproduced below.'
              '\nPlease report this error to the Nexus developers.'+2*'\n')
        import traceback
        exc_type, exc_value, exc_traceback = sys.exc_info()
        exc_lines = traceback.format_exception(exc_type, exc_value, exc_traceback)
        for line in exc_lines:
            print(line.rstrip())
        print()
        sys.exit(1)

    # ensure closely coupled nexus module is in use.
    nexus_lib2 = os.path.realpath(os.path.join(nexus.__file__,'..','..'))
    if nexus_lib != nexus_lib2:
        exe = sys.argv[0].split('/')[-1]
        exe_full_path = os.path.abspath(exe)
        print('\nBroken Nexus installation!' 
              +'\n  {} is required to reside in the Nexus module directory.'.format(exe)
              +'\n  {} location      {}   : {}'.format(exe,' '*max(0,8-len(exe)),exe_full_path)
              +'\n  Nexus module directory    : ' + nexus_lib2+'\n')
        sys.exit(1)
#end def find_nexus_modules

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

Here is a comparison of the error messages the user will see based on the original and updated find_nexus_modules above:

Nexus modules aren't located where qmca needs them

Original:

>./qmca 
Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 18, in find_nexus_modules
    import nexus
ModuleNotFoundError: No module named 'nexus'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 36, in <module>
    find_nexus_modules()
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 20, in find_nexus_modules
    raise ImportError(
ImportError: Nexus module is required but cannot be imported! Make sure Nexus module available under /home/j1k/repositories/nexus_dev_as_package/nexus

Updated:

>./qmca 

Nexus modules are required but cannot be imported!
Make sure Nexus modules are available at:
  /home/j1k/repositories/nexus_dev_as_package/nexus

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

Nexus modules have a bad import internally

Original:

>./qmca
Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 26, in find_nexus_modules
    import nexus
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/__init__.py", line 43, in <module>
    from .pwscf   import Pwscf  , PwscfInput  , PwscfAnalyzer  , generate_pwscf_input  , generate_pwscf
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/pwscf.py", line 29, in <module>
    import bad_module
ModuleNotFoundError: No module named 'bad_module'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 44, in <module>
    find_nexus_modules()
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 28, in find_nexus_modules
    raise ImportError(
ImportError: Nexus module is required but cannot be imported! Make sure Nexus module available under /home/j1k/repositories/nexus_dev_as_package/nexus

Updated:

>./qmca

Importing Nexus modules failed for an unknown reason.
This is likely a developer error.
For more information, the original exception is reproduced below.
Please report this error to the Nexus developers.


Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 26, in find_nexus_modules
    import nexus
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/__init__.py", line 43, in <module>
    from .pwscf   import Pwscf  , PwscfInput  , PwscfAnalyzer  , generate_pwscf_input  , generate_pwscf
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/pwscf.py", line 29, in <module>
    import bad_module
ModuleNotFoundError: No module named 'bad_module'

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

Nexus modules execute bad code during import:

Original:

>./qmca
Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 44, in <module>
    find_nexus_modules()
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 26, in find_nexus_modules
    import nexus
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/__init__.py", line 43, in <module>
    from .pwscf   import Pwscf  , PwscfInput  , PwscfAnalyzer  , generate_pwscf_input  , generate_pwscf
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/pwscf.py", line 30, in <module>
    bad_int = int('6.66')
ValueError: invalid literal for int() with base 10: '6.66'

Updated:

>./qmca

Importing Nexus modules failed for an unknown reason.
This is likely a developer error.
For more information, the original exception is reproduced below.
Please report this error to the Nexus developers.


Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 26, in find_nexus_modules
    import nexus
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/__init__.py", line 43, in <module>
    from .pwscf   import Pwscf  , PwscfInput  , PwscfAnalyzer  , generate_pwscf_input  , generate_pwscf
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/nexus/pwscf.py", line 30, in <module>
    bad_int = int('6.66')
ValueError: invalid literal for int() with base 10: '6.66'

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

Broken install

Original:

>./qmca
Traceback (most recent call last):
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 60, in <module>
    find_nexus_modules()
  File "/home/j1k/repositories/nexus_dev_as_package/nexus/bin/./qmca", line 44, in find_nexus_modules
    raise Exception('Broken Nexus installation! Nexus Python scripts are required to reside in the directory that contains the Nexus module.'
Exception: Broken Nexus installation! Nexus Python scripts are required to reside in the directory that contains the Nexus module.
Nexus Python script         : /home/j1k/repositories/nexus_dev_as_package/nexus/bin/qmca
Nexus module root directory : /home/j1k/repositories/nexus_dev_as_package/nexus

Updated:

>./qmca

Broken Nexus installation!
  qmca is required to reside in the Nexus module directory.
  qmca location             : /home/j1k/repositories/nexus_dev_as_package/nexus/bin/qmca
  Nexus module directory    : /home/j1k/repositories/nexus_dev_as_package/nexus

@jtkrogel
Copy link
Contributor

jtkrogel commented Feb 4, 2026

Above are comparisons of original and updated code for find_nexus_modules for these failure cases:

  • Nexus modules aren't located where qmca needs them
  • Nexus modules have a bad import internally
  • Nexus modules execute bad code during import
  • Broken install

The error messages are more concise and tell the user what to do.

With minor modifications, this version might be sufficient for nxs-test.

@ye-luo ye-luo force-pushed the consolidate-nexus-executables branch from b13e284 to 0df99b5 Compare February 4, 2026 19:32
jtkrogel
jtkrogel previously approved these changes Feb 4, 2026
Copy link
Contributor

@jtkrogel jtkrogel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks solid now.

@ye-luo
Copy link
Contributor Author

ye-luo commented Feb 5, 2026

@prckent ping

@prckent
Copy link
Contributor

prckent commented Feb 5, 2026

I will test this "by hand".

Copy link
Contributor

@prckent prckent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In my testing, running qmca directly without any PATH or PYTHONPATH set works correctly, as does using a link to qmca. Copying qmca elsewhere and using it correctly gives an error. This clearly a nice improvement. However, I think we can use this opportunity to further improve the error messages for users:

qmcuser@9abc81ade268:~/test_simple_H2O$ $HOME/qmcpack/nexus/bin/qmca -q ev -e 200 --sac LiH.s002.dmc.dat 
                            LocalEnergy               Variance           ratio 
LiH  series 2  -8.541570 +/- 0.005672   16.4   0.142099 +/- 0.002624    6.3   0.0166 
qmcuser@9abc81ade268:~/test_simple_H2O$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/qe/bin
qmcuser@9abc81ade268:~/test_simple_H2O$ echo $PYTHONPATH
/opt/pyscf:/home/qmcuser/qmcpack/src/QMCTools
qmcuser@9abc81ade268:~/test_simple_H2O$ ln -s $HOME/qmcpack/nexus/bin/qmca  .
qmcuser@9abc81ade268:~/test_simple_H2O$ ./qmca -q ev -e 200 --sac LiH.s002.dmc.dat  
                            LocalEnergy               Variance           ratio 
LiH  series 2  -8.541570 +/- 0.005672   16.4   0.142099 +/- 0.002624    6.3   0.0166 
qmcuser@9abc81ade268:~/test_simple_H2O$ rm -f qmca 
qmcuser@9abc81ade268:~/test_simple_H2O$ cp ~/qmcpack/nexus/bin/qmca .
qmcuser@9abc81ade268:~/test_simple_H2O$ ./qmca -q ev -e 200 --sac LiH.s002.dmc.dat

File /home/qmcuser/nexus/__init__.py does not exist!
Broken Nexus installation!
Make sure Nexus modules are available at:
  /home/qmcuser

I suggest the error be expanded to something like the following. We can not expect users to be familiar with file layout in modules, so providing more info should help.

Expected file /home/qmcuser/nexus/__init__.py does not exist!
Broken Nexus installation! All expected files not found.
Make sure Nexus modules are available at:
  /home/qmcuser
Expecting to find files including
/home/qmcuser/nexus/__init__.py
/home/qmcuser/nexus/basis.py

@ye-luo
Copy link
Contributor Author

ye-luo commented Feb 5, 2026

@prckent I agree that we can not expect users to be familiar with file layout in modules.
For that reason, "Broken Nexus installation!" is a strong message to users and they should check Nexus manual about installation. I will emphasize this in the printout.

Expecting to find files including
/home/qmcuser/nexus/__init__.py
/home/qmcuser/nexus/basis.py

can potentially encourage users copying two mentioned files. It won't work well.

How about the following?

Expected file /home/qmcuser/nexus/__init__.py does not exist!
Broken Nexus installation! Please follow Nexus manual regarding installation.

The first line must be kept reflecting the exact failed check and also indicating a correct installation should provide that file.

@prckent
Copy link
Contributor

prckent commented Feb 5, 2026

Good improvement. https://nexus-workflows.readthedocs.io/en/latest/installation.html seems sufficiently up to date. Could be expanded e.g. when the installer gets updated.

@ye-luo
Copy link
Contributor Author

ye-luo commented Feb 5, 2026

@prckent updated.

Copy link
Contributor

@prckent prckent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Ye and all.

@prckent
Copy link
Contributor

prckent commented Feb 5, 2026

Test this please

@brockdyer03
Copy link
Contributor

Good improvement. https://nexus-workflows.readthedocs.io/en/latest/installation.html seems sufficiently up to date. Could be expanded e.g. when the installer gets updated.

That part of the manual will be almost completely rewritten when the packaging PR (#5742) gets merged, all in reStructuredText. I'd advise waiting for a rewrite until after that PR is merged if at all possible.

@prckent prckent merged commit 7b6468e into QMCPACK:develop Feb 6, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants