Skip to content

Conversation

@dudejas
Copy link

@dudejas dudejas commented Jan 9, 2026

Fixes #199

Copy link
Member

@aramprice aramprice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the changes, this look good code wise. I would appreciate some other folks looking at it as well, especially to think over any potential security changes from this.

Before this change we allowed an inside-container-mount of an outside-the-container-symlink to resolve at container runtime, from inside the container to the outside-container-symlink-target.

With this change bpm will pre-resolve, not at container runtime, an outside-the-container-symlink to its outside-the-container-symlink-target (not mediated by the container subsystem), and then an inside-the-container-mount will be created which points directly to the outside-the-container-symlink-target

I think this change introduces a difference in the time at which a symlink is resolved (container-creation vs. ad hoc during container runtime)... and I'm curious if this is consequential to the security posture of bpm. I don't think this is the case for the issue mentioned in #199...

I'm wondering if this could have implications in arbitrary cases?

@aramprice aramprice requested review from a team, lnguyen and ragaskar and removed request for a team January 20, 2026 19:25
@dudejas
Copy link
Author

dudejas commented Jan 27, 2026

Summary

This fix resolves symlink handling for bind mounts in BPM containers on Noble (Ubuntu 24.04) stemcells. While initially observed on GCP, this affects all major cloud providers when using instance types without ephemeral storage, which is becoming increasingly common due to cost optimization.

The Problem

When BOSH builds stemcells for instances without ephemeral disks (CreatePartitionIfNoEphemeralDisk=true), it creates:

/var/vcap/packages → /var/vcap/data/packages

Noble's stricter mount namespace handling causes BPM bind mounts to fail on symlinked paths, resulting in permission denied errors.

The Fix

Use filepath.EvalSymlinks() to resolve symlinks before creating bind mounts. This safely handles both symlinked and non-symlinked paths with no performance impact.

Multi-Cloud Evidence

Instance Types Without Ephemeral Storage

Cloud Instance Types Evidence Source
GCP n1, n2, e2 (standard) Local SSDs must be explicitly requested, not default Docs
AWS T3, T4g, M6i, M7i, C6i Marked "EBS-Only" - no instance store Docs
Azure Dv5, Dsv5, Ev5, Esv5 "Local Storage: None" Docs
OpenStack Flavors with ephemeral_gb=0 Optional parameter, defaults to 0 Docs
AliCloud g9i, g9a, g8i, g7, g6 Cloud storage only Docs

All Stemcell Builders Use This Pattern

Cloud Stemcell Builder Config
GCP bosh_google_agent_settings/apply.sh#L11
AWS bosh_aws_agent_settings/apply.sh#L13
Azure bosh_azure_agent_settings/apply.sh#L11
OpenStack bosh_openstack_agent_settings/apply.sh#L14
AliCloud bosh_alicloud_agent_settings/apply.sh#L12

Why Merge This

  1. Multi-cloud impact: Affects GCP, AWS, Azure, OpenStack, and AliCloud with commonly-used instance types
  2. Safe: filepath.EvalSymlinks() are returned unchanged for non-symlinks, so no impact on traditional configurations
  3. Preventative: Noble stemcells are rolling out now; this prevents widespread failures
  4. Best practice: Resolving paths before mounting is standard in container runtimes
  5. No cost: Negligible performance impact (one-time at container creation)

Conclusion

This isn't GCP-specific, it happens across all clouds as they move toward storage-optimized instances. The fix is universal for all cloud providers, safe, and prevents failures as we roll out Noble stemcells.

@aramprice aramprice requested a review from rkoster January 29, 2026 16:13
@aramprice aramprice moved this from Inbox to Pending Review | Discussion in Foundational Infrastructure Working Group Jan 29, 2026
@aramprice
Copy link
Member

Possibly look into https://ebpf.io/ changes between Jammy and Noble as a possible root cause for this (new in Noble?) symlink resolution problem. per @rkoster

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Pending Review | Discussion

Development

Successfully merging this pull request may close these issues.

Fix Symlink Resolution in Bind Mounts for Noble Stemcell Compatibility

2 participants