feat: Migrate fastly-exporter to AWS ECS Express

[amustaque97]

Closes: #820

[marcoieni]

from a first look it doesn't look that much simpler 😅
What do you think?

[amustaque97]

from a first look it doesn't look that much simpler 😅 What do you think?

Umm... TBH, I'm interested in learning about some of the stuff that isn't clearly mentioned in the Terraform docs, or maybe I should dig a little deeper on

• Multi-container tasks (sidecars, service mesh)
• Services requiring EFS mounts
• Complex custom ALB configurations
• Services needing very specific networking setups
• Highly customised autoscaling metrics

With ECS Express we're offloading more infra management from the user side to the vendor (AWS)
Before: Manually managed load balancer target groups, listener rules, Route53 records, ACM certificates (handled by ecs-service module)
After: ECS Express automatically provisions and manages all load-balancing infrastructure

IMO, ECS Express is not suitable for complex/critical setups.

[marcoieni]

So you think aws ecs express is not easily extensible?

Since the terraform code increased I'm wondering if it's suitable even for simple services.

Btw most of our services are quite simple. Just docker container and postgres

[amustaque97]

So you think aws ecs express is not easily extensible?

IMO, yes! For example, Bors uses an EFS mount, and since there’s currently no explicit way to define task definitions, we might lose support for custom configurations. In my experience, sooner or later, a self-managed Postgres instance will also require persistent storage like EFS. And in the future, there may be other containers that need to write to EFS as well.

Along with EFS mounts, we should also consider S3 access, since some services depend on it. For instance, rust-log-analyser uses S3 for storing and retrieving logs.

We can try adding support for the following services:

• fastly-exporter (done)
• monitorbot (inactive — task count is 0 https://github.com/rust-lang/simpleinfra/blob/master/terraform/monitorbot/app.tf#L13)
• crates-io-heroku-metrics
• triagebot

Would love to hear from you!

[jdno]

Since the terraform code increased I'm wondering if it's suitable even for simple services.

I think an interesting aspect here is that the code does not rely on two other Terraform modules anymore. Given that we've been pretty bad historically at maintaining and evolving our code, having a self-contained fastly-exporter environment might actually increase maintainability despite more lines of code.

Having said that, the limitations of ECS Express might still mean that it'd be better for us to improve/update the existing ECS modules... 🤷‍♂️

[marcoieni]

Having said that, the limitations of ECS Express might still mean that it'd be better for us to improve/update the existing ECS modules...

Yeah, I agree, maybe it's better to create our own modules. Thank you for the investigation by the way, Mustaque!

[amustaque97]

Having said that, the limitations of ECS Express might still mean that it'd be better for us to improve/update the existing ECS modules...

Yeah, I agree, maybe it's better to create our own modules. Thank you for the investigation by the way, Mustaque!

Pleasure is all mine! Let me know what the next steps could be and whether community contributions are possible. I would like to participate 🙌🏻

[bryantbiggs]

this is incorrect/invalid - the aws_ecs_express_gateway_service resource was added in v6.23.0 of the AWS provider which means, thats the minimum required version (I haven't checked other resources, but just using that for reference)

[bryantbiggs]

oh goodness - theres a whole bunch more custom modules here https://github.com/rust-lang/simpleinfra/tree/master/terragrunt/modules. again, most of our modules are setup to support Terragrunt natively https://github.com/terraform-aws-modules

[bryantbiggs]

FWIW - there are already a number of well maintained modules for the custom, bespoke modules implemented here https://github.com/rust-lang/simpleinfra/tree/master/terraform/shared/modules

the major downside to rolling your own modules is that you will encounter and have to resolve issues on your own instead of relying on a community. For example, ECS Express Service has a number of issues and the implemenation as its written here is not entirely correct. See warning note we've added to the terrafrom-aws-module
terraform-aws-modules/terraform-aws-ecs#389 (comment)

from a first look it doesn't look that much simpler 😅

Nor cheaper. If you have multiple services that utilize a load balancer, ECS Express Service is not going to be a cheaper solution. It will be cheaper and easier to configure a single, shared load balancer and n-number of ECS services

[bryantbiggs]

this is incorrect

Suggested change

	policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSInfrastructureRolePolicyForVolumes"
	policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSInfrastructureRoleforExpressGatewayServices"

[bryantbiggs]

this is redundant - if you are waiting on the attachment then you are already implicitly waiting on the role.

more importantly, you shouldn't need this time_sleep resource or the explicit depends_on