POC: Remote module inclusion

[jorgee]

This pull request introduces a new Service Provider Interface (SPI) for resolving remote Nextflow modules using the @scope/name syntax. The changes provide an implementation that can work with Syntax parser V1 and V2 without coupling Nextflow runtime classes to nf-lang. It also includes a fallback resolver which check if the module files exists in case any resolver is loaded.

Remote Module Resolution SPI Integration:

• Added the RemoteModuleResolver SPI interface in nextflow.module.spi, defining methods for resolving remote modules and specifying resolver priority.
• Implemented the RemoteModuleResolverProvider class, which uses Java's ServiceLoader to discover and select the highest-priority resolver, with a fallback if none are found.
• Provided a fallback implementation (FallbackRemoteModuleResolver) that throws an informative error if no SPI resolver is available.
• Registered the default resolver implementation (DefaultRemoteModuleResolver) via the SPI mechanism in META-INF/services/nextflow.module.spi.RemoteModuleResolver.

Default Resolver Implementation and Testing:

• Added DefaultRemoteModuleResolver as the default SPI implementation, handling local and remote module resolution, version constraints, and registry integration.
• Added a test suite for DefaultRemoteModuleResolver to verify SPI loading and priority behavior.

Module Inclusion Logic Updates:

• Updated Groovy and Java module inclusion logic (IncludeDef.groovy, ModuleResolver.java, ResolveIncludeVisitor.java) to use the SPI resolver for @scope/name references and ensure correct path resolution. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]

[jorgee]

Updated the PoC for remote module inclusion, working with V2 and V1 syntax parsers and nextflow lint. @bentsherman, not sure if it will work with the language server.

[pditommaso]

@jorgee can you please include a super short "how to test" in a comment?

[jorgee]

To test this branch you need to do the following:

1. Clone the registry repo and clone branch 251215-module-binary-registry-fixes

2. Build and publish the npr-api version 0.20.1 in Maven local

$ make build
$ ./gradlew :npr-api:publishToMavenLocal 

3. Deploy a local docker registry to store modules
   $ docker run -d --name docker-registry -p 5010:5000 registry:latest

4. Run the Nextflow registry
   $ bash run.sh

5. Clone Nextflow repo and checkout this branch

6. Include the Maven local in the build.gradle file

         mavenCentral()
         maven { url 'https://repo.eclipse.org/content/groups/releases' }
         maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
+        mavenLocal()
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/releases" }
         maven { url = "https://s3-eu-west-1.amazonaws.com/maven.seqera.io/snapshots" }

8. Build Nextflow as usual

[pditommaso]

Trying this I get

» ./launch.sh -h
java.lang.NullPointerException: Cannot invoke "com.beust.jcommander.JCommander.getUnknownOptions()" because the return value of "java.util.Map.get(Object)" is null
        at nextflow.cli.Launcher.parseMainArgs(Launcher.groovy:162)
        at nextflow.cli.Launcher.command(Launcher.groovy:454)
        at nextflow.cli.Launcher.main(Launcher.groovy:682)

[jorgee]

I miss to check it. Fixed in the last commit. I have also added the fix in #6768

[pditommaso]

This looks going into the right direction, first comments:

• nextflow module should print the help (now crashes)
• The search command should likely report only "name" and "description"
• likely it should be added a info command, that dump the module meta data and include a template how to run the module itself e.g. nextflow run modules/nf-core/shasta --meta.id <ID> --reads <FASTQ_FILE> --outdir results

[jorgee]

Updated from source branch:

• Add module info command
• Update to registry API to v1
• /api path expected in registry config URL
• search only prints name and description (full for -json)

To test, you must run with registry branch fix-module-release-metadata

[pditommaso]

To test, you must run with registry branch fix-module-release-metadata

This has been merged, isn't it?

[jorgee]

Yes, it has been merged to 251215-module-binary-registry.

[pditommaso]

Push a small change, to fix compilation on my side

[pditommaso]

Nice the info command, suggest command can be improved:

This is what mcp is suggesting

  nextflow run modules/nf-core/shasta/main.nf \
      --meta.id <ID> \
      --reads <FASTQ_FILE> \
      -resume \
      --outdir results

[pditommaso]

Run supports nextflow.config, great

[pditommaso]

Likely this error message is not needed because there's already the dump of the failing command

ERROR ~ Failed to run module


Module run failed: null

[pditommaso]

Regarding the RegistryConfig i'd like to align with with one used by the gradle plugin

registry {
    url = 'https://registry.nextflow.io/api'
    apiKey = 'your-api-key'
}

it should be fairly simple declare url as List<String> (or URL?) and allow the seting as plain string (url) or a list of strings (url)

as the auth it should be assumed the first is the primary, and publish (auth) would only be allowed for it.

wdyt?

[pditommaso]

I'd also keep RegistryConfig into nf-commons

[jorgee]

Update:

• module info command behaves as mcp for nf-core modules (e.g --meta.id, bio files, and --outdir). For others modules, displays maps as --name.<KEY><NAME_KEY_VALUE> adds the --name <NAME_PATH> for files and path.
• Remove double print when error in module run
• Change RegistryConfig to match with gradle plugin and moved to nf-commons
• Fix compilation ( npr-api version 0.21.4-SNAPSOT)
• Fix Splitter tests (guava update generates a failure serializing JimFs used in tests, adding a default serializer for JimFsPath at tests)

Missing:

• When using registry-dev downloads return a WARN message because the docker-content-digest or X-checksum returned.