Section 1.1: Our Unique Use Case

At VTS Engineering, our scenario is distinct, as we operate multiple workflows that necessitate around 50 runners per execution. These workflows are integral to our monolithic application deployment, meaning they are executed frequently throughout the day.

We required a scalable and resilient solution to accommodate approximately 300 CI runners during peak hours, catering to over 150 active repositories and more than 200 engineers.

Subsection 1.1.1: Overview of the Terraform Module

We utilize the terraform-aws-github-runner Terraform module for scalable GitHub Action runners on AWS. Below is a high-level architecture diagram illustrating how the module manages the scaling of runners based on demand or scheduled intervals to optimize Spot instance usage.

Section 1.2: Our Solution

By leveraging the terraform-aws-github-runner module, we successfully deployed a fleet of 300 Spot runners to meet our CI requirements. This setup automatically scales down after work hours according to a cron schedule.

To mitigate the risk of a single point of failure within our runner pool, we also retained the capability to create additional runners on demand if the existing pool lacks idle resources or experiences outages.

Despite the daily need for numerous Spot runners, this self-hosted solution proves to be far more economical compared to previous managed CI solutions or GitHub-hosted runners.

Challenges We Faced

Transitioning from long-lived runners to ephemeral ones posed several difficulties, including maintaining a consistent pool size of around 300 runners, limitations on spinning up multiple runners simultaneously due to throttling, and issues with Spot instance availability, leading to increased service costs.

Challenge #1 — Ephemeral Runners:

In CI, it is crucial for runners to operate in an ephemeral state, ensuring they are clean before executing new jobs. The module we adopted facilitates the creation of ephemeral runners, but we encountered the limitation of requiring one EC2 instance per job, which isn't ideal.

Using ephemeral runners with GitHub Actions necessitates the use of the workflow_job event. Since these runners are stateless, each must authenticate with Docker Hub to avoid rate limits, and we must utilize specific pool configurations.

Challenge #2 — Scaling:

While we established ephemeral runners, we needed to ensure a consistent pool of approximately 300 runners. A Lambda function manages this pool, checking every few minutes to maintain the expected size and adjusting as needed.

The configuration facilitates scaling down surplus runners throughout the day, based on API calls to GitHub.

Challenge #3 — Throttling:

To meet our CI workload demands, we faced rate limits from AWS SSM Parameter Store, which only supports a limited number of requests per second. Our solution involved forking the open-source module to address these throttling issues.

Challenge #4 — Spot Instance Availability:

Using Spot instances at scale requires careful testing for availability. We frequently encountered interruption notices, which led to premature terminations of our runners. To improve stability, we diversified our instance types and families, reducing interruptions significantly.

Challenge #5 — Cost:

Although Spot instances are cost-effective, the frequent recycling of these instances increased our overall expenses for AWS services related to high EC2 usage. We implemented several measures to mitigate costs, including garbage collection for SSM parameters and utilizing VPC endpoints.

Acknowledgments

Special thanks to Pavel Susloparov and Shruti Venkatesh for their invaluable feedback and suggestions for this article. Dev works as a Senior SRE on the platform infrastructure team at VTS, with a passion for building developer tools and reliable systems while driving continuous improvements.