An error log occurs repeatedly in the deployment logs of amazon-cloudwatch-observability-controller-manager.

[wontae913]

eks version : 1.30
add-on for eks: amazon-cloudwatch-observability
version: v2.2.0-eksbuild.1

deployment.apps/amazon-cloudwatch-observability-controller-manager
image: cloudwatch-agent-operator:1.8.0

kubectl -namazon-cloudwatch logs deployment.apps/amazon-cloudwatch-observability-controller-manager

{"level":"error","ts":"2024-11-06T01:27:19Z","logger":"controllers.AmazonCloudWatchAgent","msg":"cannot create liveness probe.","error":"no service available as part of the configuration","stacktrace":"github.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Container\n\t/workspace/internal/manifests/collector/container.go:92\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.DaemonSet\n\t/workspace/internal/manifests/collector/daemonset.go:42\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Build.func5\n\t/workspace/internal/manifests/builder.go:20\ngithub.com/aws/amazon-cloudwatch-agent-operator/internal/manifests/collector.Build\n\t/workspace/internal/manifests/collector/collector.go:51\ngithub.com/aws/amazon-cloudwatch-agent-operator/controllers.BuildCollector\n\t/workspace/controllers/common.go:50\ngithub.com/aws/amazon-cloudwatch-agent-operator/controllers.(*AmazonCloudWatchAgentReconciler).Reconcile\n\t/workspace/controllers/amazoncloudwatchagent_controller.go:107\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

Is it possible to take action by changing the settings? There appears to be no livenessprobe related information in yaml.

[mitali-salvi]

Hey @wontae913 , are you overriding the default add-on configuration ? It seems like the operator did not create a service for the AmazonCloudWatchAgent and hence failed to create the liveness probe as well

[wontae913]

hello @mitali-salvi
As you mentioned, I initially thought the issue was caused by reusing the configuration from the old version when installing the new version of the addon. However, even after creating a new eks cluster and installing the addon again, the logger is still encountering same liveness probe error logs repeatedly.
Regardless of the specific error messages, the running services seem to be functioning normally, and cloudwatch-agent and fluent-bit are also working as expected.

[ovaldi]

@mitali-salvi I'm also facing the same issue.

EKS version is v1.31
amazon-cloudwatch-observability version is v2.3.1-eksbuild

here is my add-on configuration:

{
    "containerLogs": {
        "enabled": false
    },
    "agent": {
        "config": {
            "logs": {
                "metrics_collected": {
                    "kubernetes": {
                        "enhanced_container_insights": true,
                        "accelerated_compute_metrics": false
                    }
                }
            }
        }
    }
}

[ovaldi]

Hi @movence @mitali-salvi Is there anything we can do to assist with this?

[andrewhharmon]

I'm facing the same problem as well. Is this expected? It spams the logs constantly.

[mk2134226]

I am facing same problem eks 1.31 aws-observability addon verison 3.1.0

[suankan]

Having the same problem with EKS 1.32 and amazon-cloudwatch-observability addon verison v3.3.1-eksbuild.1
Not using any custom configuration for the addon.

At the same time, I can see the data in the CloudWatch.

[wormym1011]

I also have the same problem with u guys.
+1 fix it pls

[sky333999]

As mentioned above, the log is noise as we do not expect to have liveness probes on CloudWatchAgent with the current setup. We will look into suppressing this log in the short term.

[NeodymiumFerBore]

We are facing the same issue.

Installed with the EKS amazon-cloudwatch-observability addon, version v3.7.0-eksbuild.1. We use a similar configuration:

cloudwatch-agent-operator image: 2.2.1
cloudwatch-agent image: 1.300055.0b1095
aws-for-fluent-bit image: 2.32.5.20250327

{
    "agent": {
        "config": {
            "logs": {
                "metrics_collected": {
                    "kubernetes": {
                        "enhanced_container_insights": true,
                        "accelerated_compute_metrics": false
                    }
                }
            }
        }
    }
}

The fluent-bit DaemonSet also spams with error messages every minute:

[2025/04/30 13:23:23] [ warn] [net] getaddrinfo(host='cloudwatch-agent.amazon-cloudwatch', err=-2): Name or service not known
[2025/04/30 13:23:23] [error] [net] [filter:kubernetes:kubernetes.1] [kubernetes] no upstream connections available to cloudwatch-agent.amazon-cloudwatch:4311

The Service is, indeed, not created. Maybe this service is related to the service mentioned in the controller manager error message?

When we restore the default configuration, no more error message from both controller manager and fluent-bit DaemonSet: the cloudwatch-agent Service is created in the amazon-cloudwatch Namespace.

We'd expect to not have to customize the fluent-bit configuration if we disable Application Signals and accelerated compute metrics from the controller manager configuration.

(Despite these errors, container logs are correctly shipped to CloudWatch)

[KimYongTTT]

Any Updates? I'm having same issue with eks v1.33 / amazon-cw-observability addon v4.3.1-eksbuild.1

[jscottintegral]

Updates appreciated! Also experiencing with EKS Auto mode, kube v1.33, platform eks.11 and addon v4.3.1-eksbuild.1.

[andromeda306]

Im seeing the same error - clean latest add-on installation with no custom configs. However If I edit the add-on in the aws console and dont change anything then save, the error disappears and status turns to active from degraded.