multiple separate applications in module build

[celinakalus]

First off: Don't worry about answering this before or during the holidays.

I am currently trying to implement a module build where the top level is a Zephyr module, and there are multiple UFW/Zephyr applications in different subfolders which act as samples or tests for the module. Defining it in module.yaml seems to be possible by doing:

targets:
  - application: application1/
    # ...
  - application: application2/
    # ...

When I do this, the concerning directory needs to also be included in CMake, because the module build will be started from the parent directory of the module.yaml:

add_subdirectory(application1)
add_subdirectory(application1)

This, however, leads to a problem, because each of these applications has a target app, and they will all try to put their sources into the same build output.

Instead I could do this:

if(DEFINED UFW_ZEPHYR_APPLICATION)
        message("Building application ${UFW_ZEPHYR_APPLICATION}")
        add_subdirectory(${UFW_ZEPHYR_APPLICATION})
endif()

This works as long as this is built as a toplevel module build (so mmh in that folder), but as soon as I try to add this repository as a dependency to other system.yaml or module.yaml, it obviously breaks, because suddenly, UFW_ZEPHYR_APPLICATION is out of tree.

Not entirely sure what the best strategy for this use case would be. Maybe module builds should be able to specify other base directories for CMake, so CMake can be started from the application directly? Or there should be some way for the CMake infrastructure to tell which application should be built, if any? Maybe you disagree and module builds should not support any of this? Maybe there already is a way, in which case I would be very happy about a hint.

[ft]

Hey Celina! Yeah, it's an interesting problem. The idea with module builds was to exercise the code of one module, be it a library or an application. You basically found most of what you can currently do. And as you found out, this clashes a little with zephyr's build system, which is very much application-centric, which is to say you're not building more than one application in a single go. There may be a way of shoehorning this into Zephyr's new system-build facility, which is a way of building multiple zephyr-applications in one go. I am not sure how difficult this would be, because I have not looked into this Zephyr facility in depth yet. It's somehow implemented in CMake, so I have my doubts that it can run totally independent toolchains, unless it uses a similar hack build on top of CMake's ExternalProject, which can get ugly quickly, and is the reason mmh system exists. And the fact that there are functions like ExternalZephyrProject_Add() make me even more sceptical. To me it sounds a little like being able to build things like a bootloader and an application and generates images in different file formats and possibly have them signed as well, rather than have very isolated builds.

However, it is possible to run mmh with different configuration files:

% mmh --directory build-a --module application-a.yaml
% mmh --directory build-b --module application-b.yaml

So that would be multiple calls to mmh, but should support everything you need.

There are a few downsides, that we could improve upon:

• You have to call mmh more than once, and if you have tooling that assumes all it has to do to build something is a single call, then you suddenly need to adjust your tooling. Not the worst thing in the world, but a wrinkle nevertheless.
• The dependencies in the build-trees do not share any data. So big dependencies like the zephyr kernel will be cloned (completely) n times.

There might be others, but these come to mind immediately. One way to improve this, could be a new module type, like:

name: my-multibuild-thingy
type: multi

# Allow specifying dependencies that can be shared among module builds.
# Module builds can still specify additional dependencies.
dependencies:
  # …

modules:
  - spec: application-a.yaml
    # You don't have to add additional arguments to `mmh`, but you can.
    # But with it, it would be possible to run the same application build with
    # different arguments.
    args: --arguments --for --mmh --call
  - …

# This would also allow to set other common things for sub-builds, like:
buildconfigs:
  - debug
buildtools:
  - ninja

This would call multiple mmh processes, one for each entry in the modules list. There are a couple of pitfalls, like injecting the shared dependencies into the subprocesses, and allowing different modules to have different versions of the same dependencies. None of that is terribly difficult, though, just a little tedious to get right.

I like that separate mmh module builds would isolate the different builds processes from each other completely.

WDYT?