🚨 ArgoCD Deployment Failed: demo-payload

[github-actions]

ArgoCD Deployment Failure

Application: demo-payload
Timestamp: 2026-02-11T07:02:03.282Z

Cluster Information

Field	Value
Cluster Name	aks-eastus2
Resource Group	rg-anyscale-demo
Namespace	default

Application Status

Field	Value
Health Status	Degraded
Sync Status	Synced
Revision	def78ff78f604229ea472eb2ecb1a3f001fcec60
Repository	https://github.com/DevExpGbb/agentic-platform-engineering.git

Error Message

successfully synced (all tasks run)

Troubleshooting Commands

# Check application status in ArgoCD
argocd app get demo-payload

# Check pods in namespace
kubectl get pods -n default

# Describe failed pods
kubectl describe pods -n default

# Get pod logs
kubectl logs -n default <pod-name>

# Check events
kubectl get events -n default --sort-by='.lastTimestamp'

Quick Links

• ArgoCD UI
• Source Repository

---

This issue was automatically created by ArgoCD Notifications

[github-actions]

🔍 Diagnostic Summary

Status: Root cause identified and fix PR created
Severity: High - Application deployment failure
Cluster Identity: ✅ Confirmed (aks-eastus2 in rg-anyscale-demo)

---

🎯 Root Cause

Typo in container image name preventing image pull from GitHub Container Registry

The order-service deployment manifest contains a typo in the container image reference:

• Line 196 in Act-3/argocd/apps/broken-aks-store-all-in-one.yaml
• Current (incorrect): ghcr.io/azure-samples/aks-store-demo/order-servi**e**:2.1.0
• Should be: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0

The misspelled repository name "order-servie" does not exist on ghcr.io, causing the kubelet to fail with a 403 Forbidden error when attempting to pull the image.

---

📊 Cluster Evidence

Pod Status

NAME                                READY   STATUS             RESTARTS   AGE
order-service-5c6b567944-v5cfv      0/1     ImagePullBackOff   0          3m
order-service-7585988949-hv6kj      0/1     ImagePullBackOff   0          2m45s

Error from Events

Failed to pull image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0": 
failed to resolve reference "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0": 
failed to authorize: failed to fetch anonymous token: 
unexpected status from GET request to https://ghcr.io/token: 403 Forbidden

Deployment Health

• ✅ RabbitMQ: Running and healthy (init container wait completed successfully)
• ✅ MongoDB: Running and healthy
• ✅ makeline-service: Running (1/1)
• ✅ product-service: Running (1/1)
• ✅ store-front: Running (1/1)
• ✅ store-admin: Running (1/1)
• ✅ virtual-customer: Running (1/1)
• ✅ virtual-worker: Running (1/1)
• ❌ order-service: 0/1 (ImagePullBackOff)

---

🔧 Remediation

Pull Request Created: #45
Branch: fix/cluster-doctor/issue-44-typo-order-service-20260211

Change:

- image: ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0
+ image: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0

---

✅ Verification Steps

After the PR is merged and ArgoCD syncs:

1. Check pod status:

   kubectl get pods -n default -l app=order-service

   Expected: 1/1 Running

2. Verify image pull:

   kubectl describe pod -n default -l app=order-service

   Expected: No ImagePullBackOff errors

3. Check application health:

   kubectl get pods -n default

   Expected: All pods in Running state

4. Test order service endpoint:

   kubectl port-forward -n default svc/order-service 3000:3000
   curl http://localhost:3000/health

   Expected: HTTP 200 response

---

📝 Additional Notes

• No cluster-wide issues detected - other microservices are healthy
• No networking issues - init container successfully reached RabbitMQ
• No RBAC issues - deployment has proper permissions
• No resource constraints - sufficient CPU/memory available
• Issue is isolated to the single typo in the manifest

This is a simple configuration error that can be safely remediated via GitOps PR merge and ArgoCD resync. No manual cluster intervention required.

---

Automated diagnosis completed by Cluster Doctor Agent
Timestamp: 2026-02-11T07:03:58Z

[github-actions]

🔄 Deployment Failed Again

Timestamp: 2026-02-11T07:11:07.500Z
Revision: def78ff78f604229ea472eb2ecb1a3f001fcec60

Error:

successfully synced (all tasks run)