Requesting clarification in documentation

[zflat]

Thanks @ruffsl for this helpful extension! I am using this to speed up CI for feature branches and would like to seek some clarification to what is described in the README. (Also, much thanks to providing a nice readme instead of just a mostly empty file only containing the package name!)

Can you please explain the purpose of the --packages-select-cache-key? For example, where would I look to see what the default key is and why would I need to override it?

Also, the lock workflow is not intuitive to me. In the quick start example, why do I need to lock the workspace before the first colcon build and also after the minor source file changes? When colcon build --packages-skip-cache-valid is run, does it re-hash the package and compare to what is cached? Or does the hashing only happen when colcon lock is called? Also, it appears that lockfiles are also updated/created after the build and test verbs too (as described by the sentence "Upon successful task completion for a package job, as when evoking colcon verbs like build, test, etc, these lockfiles are updated for the evoked verb, thereby delineating the provenance of the job’s results"). To my understanding, the verb lock is a bit strong because it sounds like other actions would not be permitted to make changes to lockfiles, but that is not the case. To me, the verb is closer to hash or maybe update to manually trigger an update to the lock files.

I think that a more detailed explanation (maybe annotating the example) of what creates and or modifies which lock files and what checks against the lock files are performed would be helpful. Also, an example of using --packages-select-cache-key would be helpful.

Overall, I think that understanding that there are some "automatic" updates to lock files depending on using verbs like build and test is an important concept that a new user may not be aware of based on the current example and explanation.

[ruffsl]

Hey @zflat , neat to hear your use of this extension to speed up CI. If you like to see a real world example using this extension, we've been using it rather heavily for the Nav2 project for over a year or so:

• Update CI to use colcon cache ros-navigation/navigation2#2343

Can you please explain the purpose of the --packages-select-cache-key? For example, where would I look to see what the default key is and why would I need to override it?

The purpose of the --packages-select-cache-key is to override the default cache key selected by the verb handler. For example, the verb handler for the "colcon list", it uses the same verb handler for "colcon cache", which defines what lockfiles are to be used for comparison or reference. When used with package selection, this can be helpful when listing what caches are still valid for other colcon verbs (e.g. not just for "colcon list").

• colcon-cache/setup.cfg

  Line 94 in 0a23776

  list = colcon_cache.verb_handler.cache:CacheVerbHandler

• colcon-cache/colcon_cache/package_selection/valid.py

  Lines 56 to 58 in 0a23776

  	verb_name = args.packages_select_cache_key
  	if not verb_name:
  	verb_name = args.verb_name

• colcon-cache/colcon_cache/package_selection/valid.py

  Lines 82 to 85 in 0a23776

  	verb_lockfile = verb_handler_extension\
  	.get_current_lockfile(package_build_base)
  	reference_lockfile = verb_handler_extension\
  	.get_reference_lockfile(package_build_base)

In Nav2 for example, we do this to list packages with invalidated caches for specifically the "colcon build" verb, so that we can determine the package builds that must be subsequently cleaned:

https://github.com/ros-planning/navigation2/blob/5547a20ae711417af90768d9d594a4524064e1ca/.circleci/config.yml#L163-L169

Also, the lock workflow is not intuitive to me.

The level of abstraction with caching results for colcon verbs, verbs that transitively depend on other verbs, makes this difficult to describe. The hashing of source files (or query to detected revision control) is only invoked with the "colcon cache lock" subverb. I suppose this could have been abbreviated to the root verb "colcon cache", but I suspected I may want to add more extension subverbs in the future, so I pushed that hashing funatilty into a "lock" subverb, as "colcon cache cache" seemed too peculiar.

The updating of lockfiles is currently simply a copy operation upon a successful task for the package, where the invoked colcon verb triggers an event handler that copies the lockfile from the verb it transitively depends upon. E.g. the build verb depends upon the cache verb, and the test verb depends upon the build verb. This pattern could be extended for any other verbs via more registered event handlers.

I think that a more detailed explanation (maybe annotating the example) of what creates and or modifies which lock files and what checks against the lock files are performed would be helpful.

I can add a "How does it work" section to the readme, but I'd first like to finish refactoring where the lockfiles are written and read from in the colcon workspace, as to store the lockfiles closer to the artifacts they claim to represent:

• Migrate verb lockfiles into verb's base path #42

[zflat]

Thanks @ruffsl for the explanation! The example of using a different --packages-select-cache-key makes it more clear to me now.

It is helpful to know that the build verb lock file is just copied from the cache verb. And I think that a "how it works" section would be very helpful.