From 7096cd30ed4a982fd4334f42f07bc54e96660499 Mon Sep 17 00:00:00 2001 From: Pranay Sanghvi Date: Fri, 23 Jan 2026 20:22:20 +0530 Subject: [PATCH 1/3] Update staging-scripts.md --- .../customization/staging-scripts.md | 175 +++++++++++++++++- 1 file changed, 173 insertions(+), 2 deletions(-) diff --git a/docs/references/customization/staging-scripts.md b/docs/references/customization/staging-scripts.md index ba1b376e4d..62ed1bfecd 100644 --- a/docs/references/customization/staging-scripts.md +++ b/docs/references/customization/staging-scripts.md @@ -14,7 +14,7 @@ This customization interacts with the [customization of buildpacks](staging.md). ::: -Epinio uses staging scripts to interact with Paketo [Cloud Native Buildpacks](https://buildpacks.io/). +Epinio uses staging scripts to interact with [Cloud Native Buildpacks](https://buildpacks.io/). Epinio automatically selects the set of staging scripts based on the name of the chosen builder image, and the images supported by a specific definition. @@ -29,7 +29,7 @@ It is this last definition which is configured when [customizing buildpacks](sta with a different image. By default this fallback is configured for Jammy-based images in general, and the -[jammy full stack paketo builder image](https://github.com/paketo-buildpacks/builder-jammy-full) +[jammy full stack Cloud Native Buildpacks builder image](https://github.com/paketo-buildpacks/builder-jammy-full) in particular ## Specification @@ -120,3 +120,174 @@ specifications match everything and the last found is used as the fallback if an specification matched. If no specification matched, and no fallback was found, staging fails. + +## Creating New Stage Scripts + +Administrators can add new stage scripts for different build packs using two methods: + +### Method 1: Using Helm Values (Recommended) + +The easiest way to add new stage scripts is through Helm values. This approach allows you to add scripts without modifying the Helm chart templates. + +Add your custom stage scripts to your `values.yaml` file or pass them via `--set` flags: + +```yaml +stagingScripts: + custom: + - name: epinio-stage-scripts-custom-builder + builder: "myorg/my-builder:*" + userID: "1002" + groupID: "1000" + env: | + CNB_PLATFORM_API: "0.12" + base: epinio-stage-scripts # Inherit download/unpack/build scripts from base +``` + +**Using a base ConfigMap (recommended for most cases):** + +When your builder only differs in user/group IDs or environment variables, you can inherit the download, unpack, and build scripts from an existing base ConfigMap: + +```yaml +stagingScripts: + custom: + - name: epinio-stage-scripts-my-builder + builder: "myorg/my-builder:*" + userID: "1002" + groupID: "1000" + env: | + CNB_PLATFORM_API: "0.12" + MY_CUSTOM_VAR: "value" + base: epinio-stage-scripts # Inherits download, unpack, and build scripts +``` + +**Fully custom scripts:** + +If you need completely custom scripts, you can define them inline: + +```yaml +stagingScripts: + custom: + - name: epinio-stage-scripts-fully-custom + builder: "myorg/another-builder:*" + userID: "1003" + groupID: "1000" + env: | + CNB_PLATFORM_API: "0.13" + downloadImage: "ghcr.io/epinio/epinio-awscli:latest" + unpackImage: "ghcr.io/epinio/epinio-unpacker:latest" + download: |- + # Custom download script + echo "Downloading from custom source..." + # Your custom download logic here + unpack: |- + # Custom unpack script + echo "Unpacking with custom logic..." + # Your custom unpack logic here + build: |- + # Custom build script + echo "Building with custom process..." + # Your custom build logic here +``` + +### Example: Adding Support for Paketo Tiny Builder + +Here's a real-world example of adding support for the [Paketo Tiny Builder](https://github.com/paketo-buildpacks/builder-tiny), which is a minimal Cloud Native Buildpacks builder for smaller container images: + +```yaml +stagingScripts: + custom: + - name: epinio-stage-scripts-tiny + # Match all Paketo Tiny builder images + builder: "paketobuildpacks/builder-tiny:*" + # Tiny builder uses the same user/group IDs as Jammy builders + userID: "1001" + groupID: "1000" + env: | + CNB_PLATFORM_API: "0.11" + # Inherit download, unpack, and build scripts from the base + # The build script will use /cnb/lifecycle/creator with the tiny builder + base: epinio-stage-scripts +``` + +After adding this configuration, you can use the Paketo Tiny builder when pushing applications: + +```bash +epinio push --name myapp --builder-image paketobuildpacks/builder-tiny:latest +``` + +**Why use the Tiny builder?** The Tiny builder produces smaller container images by including only essential buildpacks and a minimal base image, making it ideal for applications that don't need the full set of buildpacks available in the full builder. + +**Finding the correct configuration for other CNB builders:** + +1. **Check the builder's documentation** for the Platform API version it supports +2. **Inspect the builder image** to find the `cnb` user IDs: + ```bash + docker run --rm paketobuildpacks/builder-tiny:latest id + ``` +3. **Test with a simple application** to verify the configuration works + +### Method 2: Creating Template Files + +For more complex scenarios or when you need to version control your scripts with the chart, you can create new template files in the Helm chart. + +1. **Create a new template file** in `helm-charts/chart/epinio/templates/` following the naming pattern `stage-scripts-*.yaml`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + labels: + app.kubernetes.io/component: epinio-staging + app.kubernetes.io/part-of: epinio + app.kubernetes.io/version: {{ default .Chart.AppVersion .Values.image.epinio.tag }} + name: epinio-stage-scripts-my-builder + namespace: {{ .Release.Namespace }} +data: + builder: "myorg/my-builder:*" + userID: "1002" + groupID: "1000" + env: |- + CNB_PLATFORM_API: "0.12" + base: epinio-stage-scripts +``` + +2. **Or use the helper template** for consistency: + +```yaml +{{- include "epinio.stage-script" (dict + "name" "epinio-stage-scripts-my-builder" + "builder" "myorg/my-builder:*" + "userID" "1002" + "groupID" "1000" + "env" "CNB_PLATFORM_API: \"0.12\"\n" + "base" "epinio-stage-scripts" + "context" . +) }} +``` + +### Best Practices + +1. **Use the base mechanism**: Most of the time, you only need to override `builder`, `userID`, `groupID`, and `env`. Use the `base` field to inherit the standard download, unpack, and build scripts. + +2. **Naming convention**: Use descriptive names that indicate the builder type, e.g., `epinio-stage-scripts-jammy`, `epinio-stage-scripts-bionic`. + +3. **Builder pattern matching**: Use glob patterns in the `builder` field to match multiple builder images. For example: + - `"paketobuildpacks/builder-jammy-*:*"` matches all jammy-based builders + - `"myorg/my-builder:*"` matches all tags of a specific builder + - `"*"` matches everything (use only for fallback) + +4. **User and Group IDs**: Different builder images may use different user/group IDs. Check your builder image documentation or inspect the image to determine the correct IDs: + ```bash + docker run --rm myorg/my-builder id + ``` + +5. **CNB_PLATFORM_API**: Different builder versions may require different `CNB_PLATFORM_API` values. Check your builder's documentation for the correct version. + +6. **Testing**: After adding a new stage script, test it by pushing an application that uses the matching builder image. + +### Troubleshooting + +- **Script not found**: Ensure your ConfigMap has the label `app.kubernetes.io/component: epinio-staging` +- **Wrong script selected**: Check the builder pattern matches your builder image name and tag. Scripts are matched in lexicographic order by name. +- **Build fails**: Verify the `userID` and `groupID` match the `cnb` user in your builder image +- **Environment issues**: Ensure `CNB_PLATFORM_API` matches your builder's requirements From 6c23197b0f303bd3266a060a54228307d6c1f374 Mon Sep 17 00:00:00 2001 From: Pranay Sanghvi Date: Tue, 27 Jan 2026 13:18:39 +0530 Subject: [PATCH 2/3] Update staging-scripts.md --- .../customization/staging-scripts.md | 148 ++++++++++++++---- 1 file changed, 118 insertions(+), 30 deletions(-) diff --git a/docs/references/customization/staging-scripts.md b/docs/references/customization/staging-scripts.md index 62ed1bfecd..372d92db37 100644 --- a/docs/references/customization/staging-scripts.md +++ b/docs/references/customization/staging-scripts.md @@ -42,9 +42,10 @@ This ConfigMap is expected to have the following keys: |Key |Content | |--- |--- | |`builder` |Glob-pattern to match the image references supported by this spec. | -|`userID` |User ID to run the `build` script with. | -|`groupID` |Group ID to run the `build` script with. | +|`userID` |User ID to run the `build` script with (as string, e.g., "1001"). | +|`groupID` |Group ID to run the `build` script with (as string, e.g., "1000"). | |`env` |Standard environment for the scripts, YAML-formatted string, key/value map. | +|`staging-values.json`|Optional JSON object containing Helm values for staging workload configuration (resources, nodeSelector, tolerations, etc.). | ||| |`base` |Optional redirect to the actual ConfigMap with the scripts. | ||| @@ -127,7 +128,7 @@ Administrators can add new stage scripts for different build packs using two met ### Method 1: Using Helm Values (Recommended) -The easiest way to add new stage scripts is through Helm values. This approach allows you to add scripts without modifying the Helm chart templates. +The easiest way to add new stage scripts is through Helm values. This approach allows you to add scripts without modifying the Helm chart templates. The Helm chart will automatically create ConfigMaps from these values during installation or upgrade. Add your custom stage scripts to your `values.yaml` file or pass them via `--set` flags: @@ -136,59 +137,97 @@ stagingScripts: custom: - name: epinio-stage-scripts-custom-builder builder: "myorg/my-builder:*" - userID: "1002" - groupID: "1000" + userID: 1002 # Can be number or string + groupID: 1000 # Can be number or string env: | CNB_PLATFORM_API: "0.12" base: epinio-stage-scripts # Inherit download/unpack/build scripts from base ``` +:::note +The `name` field is optional. If not provided, a name will be automatically generated from the builder pattern. However, it's recommended to provide a descriptive name for easier identification. +::: + **Using a base ConfigMap (recommended for most cases):** -When your builder only differs in user/group IDs or environment variables, you can inherit the download, unpack, and build scripts from an existing base ConfigMap: +When your builder only differs in user/group IDs or environment variables, you can inherit the download, unpack, and build scripts from an existing base ConfigMap. This is the most common use case: ```yaml stagingScripts: custom: - name: epinio-stage-scripts-my-builder builder: "myorg/my-builder:*" - userID: "1002" - groupID: "1000" + userID: 1002 + groupID: 1000 env: | CNB_PLATFORM_API: "0.12" MY_CUSTOM_VAR: "value" base: epinio-stage-scripts # Inherits download, unpack, and build scripts + # Optional: Custom staging workload configuration + stagingValues: + resources: + requests: + cpu: "500m" + memory: "1Gi" + limits: + cpu: "2" + memory: "4Gi" + nodeSelector: + workload-type: "build" ``` +When using `base`, you only need to specify: +- `builder`: The glob pattern to match your builder image +- `userID` and `groupID`: The user/group IDs for the `cnb` user in your builder +- `env`: Environment variables (especially `CNB_PLATFORM_API`) +- `base`: The name of the base ConfigMap (usually `epinio-stage-scripts`) +- `stagingValues` (optional): Custom workload configuration + **Fully custom scripts:** -If you need completely custom scripts, you can define them inline: +If you need completely custom scripts (not using a base), you must provide all three scripts (`download`, `unpack`, and `build`). You can also optionally override the `downloadImage` and `unpackImage`: ```yaml stagingScripts: custom: - name: epinio-stage-scripts-fully-custom builder: "myorg/another-builder:*" - userID: "1003" - groupID: "1000" + userID: 1003 + groupID: 1000 env: | CNB_PLATFORM_API: "0.13" + # Optional: Override default images downloadImage: "ghcr.io/epinio/epinio-awscli:latest" unpackImage: "ghcr.io/epinio/epinio-unpacker:latest" + # Required when not using base: All three scripts must be provided download: |- - # Custom download script + # Parameters available: PROTOCOL, ENDPOINT, BUCKET, BLOBID echo "Downloading from custom source..." - # Your custom download logic here + aws --endpoint-url "${PROTOCOL}://${ENDPOINT}" s3 cp "s3://${BUCKET}/${BLOBID}" "/workspace/source/${BLOBID}" unpack: |- - # Custom unpack script + # Parameters available: BLOBID, USERID, GROUPID echo "Unpacking with custom logic..." - # Your custom unpack logic here + mkdir -p /workspace/source/app + tar -xzf "/workspace/source/${BLOBID}" -C /workspace/source/app + chown -R ${USERID}:${GROUPID} /workspace build: |- - # Custom build script + # Parameters available: PREIMAGE, APPIMAGE, USERID, GROUPID echo "Building with custom process..." - # Your custom build logic here + /cnb/lifecycle/creator \ + -app=/workspace/source/app \ + -cache-dir=/workspace/cache \ + -uid=${USERID} \ + -gid=${GROUPID} \ + -layers=/layers \ + -platform=/workspace/source \ + "-previous-image=${PREIMAGE}" \ + "${APPIMAGE}" ``` +:::warning +When not using `base`, you must provide all three scripts (`download`, `unpack`, and `build`). The Helm chart will fail to render if any are missing. +::: + ### Example: Adding Support for Paketo Tiny Builder Here's a real-world example of adding support for the [Paketo Tiny Builder](https://github.com/paketo-buildpacks/builder-tiny), which is a minimal Cloud Native Buildpacks builder for smaller container images: @@ -200,8 +239,8 @@ stagingScripts: # Match all Paketo Tiny builder images builder: "paketobuildpacks/builder-tiny:*" # Tiny builder uses the same user/group IDs as Jammy builders - userID: "1001" - groupID: "1000" + userID: 1001 + groupID: 1000 env: | CNB_PLATFORM_API: "0.11" # Inherit download, unpack, and build scripts from the base @@ -267,27 +306,76 @@ data: ### Best Practices -1. **Use the base mechanism**: Most of the time, you only need to override `builder`, `userID`, `groupID`, and `env`. Use the `base` field to inherit the standard download, unpack, and build scripts. +1. **Use the base mechanism**: Most of the time, you only need to override `builder`, `userID`, `groupID`, and `env`. Use the `base` field to inherit the standard download, unpack, and build scripts. This reduces maintenance and ensures consistency. -2. **Naming convention**: Use descriptive names that indicate the builder type, e.g., `epinio-stage-scripts-jammy`, `epinio-stage-scripts-bionic`. +2. **Naming convention**: Use descriptive names that indicate the builder type, e.g., `epinio-stage-scripts-jammy`, `epinio-stage-scripts-bionic`, `epinio-stage-scripts-tiny`. If you don't provide a name, one will be auto-generated, but explicit names are easier to identify in logs and ConfigMap listings. 3. **Builder pattern matching**: Use glob patterns in the `builder` field to match multiple builder images. For example: - - `"paketobuildpacks/builder-jammy-*:*"` matches all jammy-based builders + - `"paketobuildpacks/builder-jammy-*:*"` matches all jammy-based builders (e.g., `builder-jammy-base`, `builder-jammy-full`) - `"myorg/my-builder:*"` matches all tags of a specific builder - - `"*"` matches everything (use only for fallback) + - `"*"` matches everything (use only for fallback ConfigMaps) 4. **User and Group IDs**: Different builder images may use different user/group IDs. Check your builder image documentation or inspect the image to determine the correct IDs: ```bash docker run --rm myorg/my-builder id + # Look for the 'cnb' user output, e.g., uid=1001(cnb) gid=1000(cnb) + ``` + You can specify IDs as numbers (`1001`) or strings (`"1001"`) in the YAML. + +5. **CNB_PLATFORM_API**: Different builder versions may require different `CNB_PLATFORM_API` values. Check your builder's documentation for the correct version. Common values: + - `"0.4"` for older Bionic-based builders + - `"0.11"` for most Jammy-based builders + - `"0.12"` or `"0.13"` for newer builders + +6. **Staging workload configuration**: Use `stagingValues` to customize resource limits, node selectors, tolerations, and other workload settings for specific builders: + ```yaml + stagingValues: + resources: + requests: + cpu: "1" + memory: "2Gi" + limits: + cpu: "4" + memory: "8Gi" + nodeSelector: + workload-type: "build" + tolerations: + - key: "build-workload" + operator: "Equal" + value: "true" + effect: "NoSchedule" ``` -5. **CNB_PLATFORM_API**: Different builder versions may require different `CNB_PLATFORM_API` values. Check your builder's documentation for the correct version. - -6. **Testing**: After adding a new stage script, test it by pushing an application that uses the matching builder image. +7. **Testing**: After adding a new stage script: + - Verify the ConfigMap was created: `kubectl get configmap -n epinio | grep stage-scripts` + - Check the ConfigMap contents: `kubectl get configmap epinio-stage-scripts- -n epinio -o yaml` + - Test by pushing an application with the matching builder: `epinio push --name myapp --builder-image ` ### Troubleshooting -- **Script not found**: Ensure your ConfigMap has the label `app.kubernetes.io/component: epinio-staging` -- **Wrong script selected**: Check the builder pattern matches your builder image name and tag. Scripts are matched in lexicographic order by name. -- **Build fails**: Verify the `userID` and `groupID` match the `cnb` user in your builder image -- **Environment issues**: Ensure `CNB_PLATFORM_API` matches your builder's requirements +- **Script not found**: + - Ensure your ConfigMap has the label `app.kubernetes.io/component: epinio-staging` + - Verify the ConfigMap was created: `kubectl get configmap -n epinio -l app.kubernetes.io/component=epinio-staging` + - Check Helm rendered output: `helm template epinio epinio/epinio --values your-values.yaml | grep -A 20 stage-scripts` + +- **Wrong script selected**: + - Check the builder pattern matches your builder image name and tag exactly (including registry prefix) + - Scripts are matched in lexicographic order by ConfigMap name, so more specific patterns should come first + - Test the glob pattern: `filepath.Match("your-pattern", "your-builder-image:tag")` + +- **Build fails**: + - Verify the `userID` and `groupID` match the `cnb` user in your builder image + - Check staging job logs: `kubectl logs -n epinio -l app.kubernetes.io/component=epinio-staging --tail=100` + +- **Environment issues**: + - Ensure `CNB_PLATFORM_API` matches your builder's requirements + - Check that environment variables are properly formatted as YAML key-value pairs + +- **Helm template errors**: + - If using custom scripts without `base`, ensure all three scripts (`download`, `unpack`, `build`) are provided + - Verify YAML syntax is correct (proper indentation, quoted strings where needed) + - Check for typos in field names (e.g., `userID` not `userId`) + +- **ConfigMap not updating**: + - After modifying `stagingScripts.custom` in values.yaml, run `helm upgrade` to apply changes + - Verify the new ConfigMap exists: `kubectl get configmap -n epinio | grep stage-scripts` From 5addee5660b15546c950df5a5cc3e134c977942b Mon Sep 17 00:00:00 2001 From: Pranay Sanghvi Date: Tue, 27 Jan 2026 14:28:25 +0530 Subject: [PATCH 3/3] Update staging-scripts.md --- .../customization/staging-scripts.md | 39 +++++++++++-------- 1 file changed, 22 insertions(+), 17 deletions(-) diff --git a/docs/references/customization/staging-scripts.md b/docs/references/customization/staging-scripts.md index 372d92db37..d698d30aa0 100644 --- a/docs/references/customization/staging-scripts.md +++ b/docs/references/customization/staging-scripts.md @@ -45,7 +45,7 @@ This ConfigMap is expected to have the following keys: |`userID` |User ID to run the `build` script with (as string, e.g., "1001"). | |`groupID` |Group ID to run the `build` script with (as string, e.g., "1000"). | |`env` |Standard environment for the scripts, YAML-formatted string, key/value map. | -|`staging-values.json`|Optional JSON object containing Helm values for staging workload configuration (resources, nodeSelector, tolerations, etc.). | +|`staging-values.json`|Optional JSON object containing Helm values for staging workload configuration (resources, nodeSelector, tolerations, etc.). When omitted, no custom workload configuration is applied. | ||| |`base` |Optional redirect to the actual ConfigMap with the scripts. | ||| @@ -164,6 +164,7 @@ stagingScripts: MY_CUSTOM_VAR: "value" base: epinio-stage-scripts # Inherits download, unpack, and build scripts # Optional: Custom staging workload configuration + # When omitted, default workload settings from server.stagingWorkloads are used stagingValues: resources: requests: @@ -181,7 +182,7 @@ When using `base`, you only need to specify: - `userID` and `groupID`: The user/group IDs for the `cnb` user in your builder - `env`: Environment variables (especially `CNB_PLATFORM_API`) - `base`: The name of the base ConfigMap (usually `epinio-stage-scripts`) -- `stagingValues` (optional): Custom workload configuration +- `stagingValues` (optional): Custom workload configuration. When omitted, the default workload settings from `server.stagingWorkloads` in values.yaml are used, or system defaults if not specified. **Fully custom scripts:** @@ -269,7 +270,25 @@ epinio push --name myapp --builder-image paketobuildpacks/builder-tiny:latest For more complex scenarios or when you need to version control your scripts with the chart, you can create new template files in the Helm chart. -1. **Create a new template file** in `helm-charts/chart/epinio/templates/` following the naming pattern `stage-scripts-*.yaml`: +**Recommended: Use the helper template** for consistency with other stage script definitions: + +Create a new template file in `helm-charts/chart/epinio/templates/` following the naming pattern `stage-scripts-*.yaml`: + +```yaml +{{- include "epinio.stage-script" (dict + "name" "epinio-stage-scripts-my-builder" + "builder" "myorg/my-builder:*" + "userID" "1002" + "groupID" "1000" + "env" "CNB_PLATFORM_API: \"0.12\"\n" + "base" "epinio-stage-scripts" + "context" . +) }} +``` + +**Alternative: Manual ConfigMap definition** (for advanced use cases where you need full control): + +If you need to define the ConfigMap manually without using the helper template: ```yaml apiVersion: v1 @@ -290,20 +309,6 @@ data: base: epinio-stage-scripts ``` -2. **Or use the helper template** for consistency: - -```yaml -{{- include "epinio.stage-script" (dict - "name" "epinio-stage-scripts-my-builder" - "builder" "myorg/my-builder:*" - "userID" "1002" - "groupID" "1000" - "env" "CNB_PLATFORM_API: \"0.12\"\n" - "base" "epinio-stage-scripts" - "context" . -) }} -``` - ### Best Practices 1. **Use the base mechanism**: Most of the time, you only need to override `builder`, `userID`, `groupID`, and `env`. Use the `base` field to inherit the standard download, unpack, and build scripts. This reduces maintenance and ensures consistency.