Skip to content

Zane/ci agent auto deploy #1587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zanejohnson-azure wants to merge 43 commits into
base: ci_prod
Choose a base branch
from
Open

Zane/ci agent auto deploy #1587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
857cef7
add bebugging
zanejohnson-azure Dec 4, 2025
4ef04b4
debug
zanejohnson-azure Dec 4, 2025
efb8267
fix ifs spliting
zanejohnson-azure Dec 5, 2025
2b6eeb3
more parsing fix
zanejohnson-azure Dec 5, 2025
40511cd
fix image verfication container name
zanejohnson-azure Dec 5, 2025
35215ec
use short wait time temporarily
zanejohnson-azure Dec 5, 2025
9d16e10
simplify
zanejohnson-azure Dec 5, 2025
a2c64ac
refactor
zanejohnson-azure Dec 5, 2025
d2377ad
post check improve
zanejohnson-azure Dec 5, 2025
5a12295
refactor pod verify script
zanejohnson-azure Dec 5, 2025
388b307
move func
zanejohnson-azure Dec 5, 2025
a708ab9
fix
zanejohnson-azure Dec 5, 2025
9235e0a
debug
zanejohnson-azure Dec 5, 2025
3851cb2
add debug
zanejohnson-azure Dec 5, 2025
e68955b
add debug
zanejohnson-azure Dec 5, 2025
1e3c8fc
more debug
zanejohnson-azure Dec 5, 2025
2ecd7ab
more debugging
zanejohnson-azure Dec 5, 2025
1f044be
more debug
zanejohnson-azure Dec 5, 2025
bc988af
refactor..
zanejohnson-azure Dec 5, 2025
8ef284c
add comments
zanejohnson-azure Dec 5, 2025
8d106e9
move container start time capture
zanejohnson-azure Dec 5, 2025
555a4da
add delay
zanejohnson-azure Dec 5, 2025
0f6da99
wait time 15 mins
zanejohnson-azure Dec 5, 2025
354a4b0
minor fix
zanejohnson-azure Dec 5, 2025
dd66985
remove doc
zanejohnson-azure Dec 6, 2025
4b9bd84
pull from ci_prod
zanejohnson-azure Dec 6, 2025
5afb086
rebase
zanejohnson-azure Dec 6, 2025
723b073
wait 5 minutes
zanejohnson-azure Dec 6, 2025
8ffcf9b
remove container start time check wait
zanejohnson-azure Dec 6, 2025
5559943
20 min wait time
zanejohnson-azure Dec 6, 2025
3d2b480
echo utc timing
zanejohnson-azure Dec 6, 2025
d4b985f
fix trigger condition
zanejohnson-azure Dec 8, 2025
555129f
trigger when PR branch contains run-e2e
zanejohnson-azure Dec 8, 2025
b7435f8
fix trigger
zanejohnson-azure Dec 8, 2025
a1f88d9
rename stage name
zanejohnson-azure Dec 9, 2025
f3aa6f3
push image to acr during pr for particiluar branch
zanejohnson-azure Dec 12, 2025
4d2a837
pin fluentd version
zanejohnson-azure Dec 12, 2025
34cff0c
pin cool.io to 1.9.0
zanejohnson-azure Dec 13, 2025
e973a9d
remove cool.io install
zanejohnson-azure Dec 13, 2025
cbdfdcb
cleanup
zanejohnson-azure Jan 22, 2026
a7bfb77
patch ama-logs-prometheus
zanejohnson-azure Jan 22, 2026
0e1bbb9
get changes from main
zanejohnson-azure Jan 22, 2026
35b60fd
pull change from ci_prod
zanejohnson-azure Jan 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 112 additions & 1 deletion .pipelines/azure_pipeline_mergedbranches.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,15 @@ extends:
customBuildTags:
- ES365AIMigrationTooling
stages:
# This stage will be skipped when LinuxImageOverride and WindowsImageOverride are both set
# This feature allows bypassing the build stage when using pre-built images for testing, which saves time and resources.
- stage: stage
displayName: 'Build and Publish Container Images'
condition: |
or(
eq(variables['LinuxImageOverride'], ''),
eq(variables['WindowsImageOverride'], '')
)
jobs:
- job: common
pool:
Expand Down Expand Up @@ -185,7 +193,17 @@ extends:
docker pull mcr.microsoft.com/azuremonitor/containerinsights/cidev/prometheus-collector/images:buildx-stable-1
docker buildx create --name dockerbuilder --driver docker-container --driver-opt image=mcr.microsoft.com/azuremonitor/containerinsights/cidev/prometheus-collector/images:buildx-stable-1 --use
docker buildx inspect --bootstrap
# Determine if we should push to ACR
Copy link
Contributor

@rashmichandrashekar rashmichandrashekar Jan 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Zane, is there a reason why we shouldnt push images to ACR for all builds? generally any build apart from the ones that only run tests would require a new image to be built

Copy link
Contributor Author

@zanejohnson-azure zanejohnson-azure Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is because we do not really do any testing when PR is raised. but when PR is merged, we want to push to acr since it could be a candidate for actual release.

I keep it as what it is now so do not want to introduce any regression. But I believe we should discuss during scum to consider push to acr no matter what.

Copy link
Contributor

@rashmichandrashekar rashmichandrashekar Jan 21, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for development purpose, if I were to make changes to the repo, what would be the steps for me to get a dev image to test out?
If we are expected to update the branch name here, could we make it a variable at the top? something like we do for prom?

Copy link
Contributor Author

@zanejohnson-azure zanejohnson-azure Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before this PR:
for dev purpose, we can run pipeline from portal by specifying a branch name, then pipeline will create an image, then you can deploy to aks clusters.

after this PR:

  1. same as before.
  2. additionally, you can name your branch that includes keyword 'run-e2e', then run pipeline manully which will automatically deploy the new image to all dev clusters.
  3. apparently, any PR meged to ci_prod branch will trigger dev clusters testing too.

# Push when: NOT a PR, OR when PR is from specific branches (zane/ci-agent-auto-deploy or branches containing 'run-e2e')
SHOULD_PUSH="false"
if [ "$(Build.Reason)" != "PullRequest" ]; then
SHOULD_PUSH="true"
elif [[ "$(System.PullRequest.SourceBranch)" == "zane/ci-agent-auto-deploy" ]] || [[ "$(System.PullRequest.SourceBranch)" == *"run-e2e"* ]]; then
SHOULD_PUSH="true"
echo "PR from branch $(System.PullRequest.SourceBranch) - will push image to ACR for E2E testing"
fi

if [ "$SHOULD_PUSH" == "true" ]; then
docker buildx build --platform $(BUILD_PLATFORMS) --tag ${{ variables.repoImageName }}:$(linuxImagetag) -f kubernetes/linux/Dockerfile.multiarch --metadata-file $(Build.ArtifactStagingDirectory)/linux/metadata.json --build-arg IMAGE_TAG=$(linuxTelemetryTag) --build-arg GOLANG_BASE_IMAGE=$(GOLANG_BASE_IMAGE) --build-arg CI_BASE_IMAGE=$(CI_BASE_IMAGE) --push --provenance=false .
echo "##vso[task.logissue type=warning]Linux image built with tag: ${{ variables.repoImageName }}:$(linuxImagetag)"
docker pull ${{ variables.repoImageName }}:$(linuxImagetag)
Expand Down Expand Up @@ -543,7 +561,16 @@ extends:
inputs:
targetType: 'inline'
script: |
# Push when: NOT a PR, OR when PR is from specific branches (zane/ci-agent-auto-deploy or branches containing 'run-e2e')
$shouldPush = $false
if ("$(Build.Reason)" -ne "PullRequest") {
$shouldPush = $true
} elseif ("$(System.PullRequest.SourceBranch)" -eq "zane/ci-agent-auto-deploy" -or "$(System.PullRequest.SourceBranch)" -like "*run-e2e*") {
$shouldPush = $true
Write-Host "PR from branch $(System.PullRequest.SourceBranch) - will push image to ACR for E2E testing"
}

if ($shouldPush) {
docker push ${{ variables.repoImageName }}:$(windowsImageTag)-$(windows2019BaseImageVersion)
}
- task: CodeQL3000Finalize@0
Expand Down Expand Up @@ -751,7 +778,16 @@ extends:
inputs:
targetType: 'inline'
script: |
# Push when: NOT a PR, OR when PR is from specific branches (zane/ci-agent-auto-deploy or branches containing 'run-e2e')
$shouldPush = $false
if ("$(Build.Reason)" -ne "PullRequest") {
$shouldPush = $true
} elseif ("$(System.PullRequest.SourceBranch)" -eq "zane/ci-agent-auto-deploy" -or "$(System.PullRequest.SourceBranch)" -like "*run-e2e*") {
$shouldPush = $true
Write-Host "PR from branch $(System.PullRequest.SourceBranch) - will push image to ACR for E2E testing"
}

if ($shouldPush) {
docker push ${{ variables.repoImageName }}:$(windowsImageTag)-$(windows2022BaseImageVersion)
}
- task: CodeQL3000Finalize@0
Expand Down Expand Up @@ -792,7 +828,16 @@ extends:
az account set -s ${{ variables.subscription }}
az acr login -n ${{ variables.containerRegistry }}
@{"image.name"="${{ variables.repoImageName }}:$(windowsImageTag)"} | ConvertTo-Json -Compress | Out-File -Encoding ascii $(Build.ArtifactStagingDirectory)/windows/metadata.json
# Push when: NOT a PR, OR when PR is from specific branches (zane/ci-agent-auto-deploy or branches containing 'run-e2e')
$shouldPush = $false
if ("$(Build.Reason)" -ne "PullRequest") {
$shouldPush = $true
} elseif ("$(System.PullRequest.SourceBranch)" -eq "zane/ci-agent-auto-deploy" -or "$(System.PullRequest.SourceBranch)" -like "*run-e2e*") {
$shouldPush = $true
Write-Host "PR from branch $(System.PullRequest.SourceBranch) - will push multi-arch image to ACR for E2E testing"
}

if ($shouldPush) {
docker manifest create ${{ variables.repoImageName }}:$(windowsImageTag) ${{ variables.repoImageName }}:$(windowsImageTag)-$(windows2019BaseImageVersion) ${{ variables.repoImageName }}:$(windowsImageTag)-$(windows2022BaseImageVersion)
docker manifest push ${{ variables.repoImageName }}:$(windowsImageTag)
Write-Host "##vso[task.logissue type=warning]Windows image built with tag: ${{ variables.repoImageName }}:$(windowsImageTag)"
Expand Down Expand Up @@ -880,4 +925,70 @@ extends:
ScanType: CustomScan
FileDirPath: '$(Build.ArtifactStagingDirectory)'
DisableRemediation: false
AcceptableOutdatedSignatureInHours: 72
AcceptableOutdatedSignatureInHours: 72

- stage: Deploy_and_Test_Images_In_Dev_Clusters
displayName: Deploy and Test Images in Dev Clusters
lockBehavior: sequential
dependsOn:
- stage
# Deploy runs when Build succeeds OR when Build is skipped with valid overrides
# This stage runs when:
# 1. Direct push to ci_prod or zane/ci-agent-auto-deploy or branches containing 'run-e2e'
# 2. PR from zane/ci-agent-auto-deploy branch OR PR from branch containing 'run-e2e'
condition: |
and(
or(
eq(variables['Build.SourceBranch'], 'refs/heads/ci_prod'),
eq(variables['Build.SourceBranch'], 'refs/heads/zane/ci-agent-auto-deploy'),
contains(variables['Build.SourceBranch'], 'run-e2e'),
and(
eq(variables['Build.Reason'], 'PullRequest'),
or(
eq(variables['System.PullRequest.SourceBranch'], 'zane/ci-agent-auto-deploy'),
contains(variables['System.PullRequest.SourceBranch'], 'run-e2e')
)
)
),
or(
eq(dependencies.stage.result, 'Succeeded'),
and(
eq(dependencies.stage.result, 'Skipped'),
ne(variables['LinuxImageOverride'], ''),
ne(variables['WindowsImageOverride'], '')
)
)
)
variables:
# Use images built from previous build stage by default
# To override: Set pipeline variables 'LinuxImageOverride' and 'WindowsImageOverride' when queuing
linuxImageTagUnderTest: $[coalesce(variables['LinuxImageOverride'], stageDependencies.stage.common.outputs['setup.linuxImagetag'])]
windowsImageTagUnderTest: $[coalesce(variables['WindowsImageOverride'], stageDependencies.stage.common.outputs['setup.windowsImageTag'])]
jobs:
# TODO: gradually add more clusters from test automation framework when the tests are stable
# TODO: TeamsWebhookUri to be added
# Cluster 1: zane-test Cluster
- template: /.pipelines/e2e-test/azure-template-deploy-and-test-ci-image-in-aks-cluster.yml@self
parameters:
clusterName: 'zane-test'
Copy link
Contributor

@rashmichandrashekar rashmichandrashekar Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are the zane-test clusters only for testing or are we planning to keep them in this pipeline?

Copy link
Contributor Author

@zanejohnson-azure zanejohnson-azure Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be gone. I will replace them with whatever clusters we want to add. it is kind of placeholder now, also validates the whole workflow.

resourceGroup: 'zane-test'
azureSubscription: 'ContainerInsights_Build_Subscription_CI'
environmentName: 'CI-Agent-Dev'
linuxImageTag: $(linuxImageTagUnderTest)
windowsImageTag: $(windowsImageTagUnderTest)
azureClientId: $(AksZaneTestClientId)
azureTenantId: $(AzureZaneTestTenantId)
teamsWebhookUri: $(TeamsWebhookUri)
Copy link
Contributor

@rashmichandrashekar rashmichandrashekar Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is the teams webhook uri for?

Copy link
Contributor Author

@zanejohnson-azure zanejohnson-azure Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

plan to use same webhook uri as our existing ci test pipeline. so test results will go to same team channel.

Copy link
Contributor Author

@zanejohnson-azure zanejohnson-azure Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added webhook uri in vars on the build pipeline.


# Cluster 2: zane-test2 Cluster
- template: /.pipelines/e2e-test/azure-template-deploy-and-test-ci-image-in-aks-cluster.yml@self
parameters:
clusterName: 'zane-test2'
resourceGroup: 'zane-test'
azureSubscription: 'ContainerInsights_Build_Subscription_CI'
environmentName: 'CI-Agent-Dev2'
linuxImageTag: $(linuxImageTagUnderTest)
windowsImageTag: $(windowsImageTagUnderTest)
azureClientId: $(AksZaneTest2ClientId)
azureTenantId: $(AzureZaneTestTenantId)
teamsWebhookUri: $(TeamsWebhookUri)
Loading
Loading