Skip to main content

Migrating from CircleCI to Github Actions

Introduction

We are encouraging development teams to move away from CircleCI to Github Actions for build, test and deployments, and have built Github Actions that pretty-much behave the same as CircleCI, but without using external services (eg. quay.io)

This document is a guide to migrationg from CircleCI to Github Actions. Obviously it can’t detail every change that needs to be made, since there may be differences in deployment pipelines from component to component (different environments, approval gates and test regimes) but it’s intended to act as a guide.

It is also a work in progress - if, when following this document you encounter any issues, please raise them in #ask-prisons-digital-sre

Finally this document is intended as a guide to migrating apps built from the Kotlin and Typescript templates, although elements of it may well be useful to migrate other applications.

Moving from CircleCI to Github Actions has a couple of pre-requisities that need to be completed before moving over, and then a couple of ‘on-the-day’ moves to deactivate CircleCI and start using Github Actions. These are covered below.

Prerequisites

These are the same as any project built using the existing templates, mainly:

Teams are managed by the hmpps-github-teams project.

Updating Cloud Platform namespaces

The key to being able to deploy with Github Actions is storing Kubernetes credentials within Github itself, rather than in Circle CI environment variables or contexts. The good news is that Cloud Platforms can facilitate this by creating service accounts that can populate the environments section of Github, and both apply permissions for approval/deployment (in the same way as contexts did in CircleCI) and store the secrets so that they can be referred to by Github Actions.

Furthermore, these secrets can automatically be rotated, improving security. This section details how to create the service account and populate the environments

Namespaces and mappings

It is assumed that each deployment environment has its own namespace, but this namespace can be shared by a number of components within a product (eg. ui and api).

This is the important bit: for this to make sense, the environments should all match: (links show examples within the template repo)

HMPPS Template Module

To simplify deployment to a Cloud Platforms using Github Actions, a Terraform module called cloud-platform-terraform-hmpps-template has been created to orchestrate the main configurations of the github repository (including a service account and the environment secrets required). This also has the benefit of automatically rotating secrets, since it is continually deployed by Terraform.

Within the hmpps-templates-dev Cloud Platform namespace definition there are two files that refer to this module. This can coexist with other configurations within a namespace, although care should be taken to make sure duplicate definitions aren’t created - further information on this is below.

The simplest way to add the moving parts to enable deployment for a repository is to create a copy of either template-kotlin.tf or template-typescript.tf in the namespace, and rename it to the name of the Github Repository to which it will apply.

Then set the following fields appropriate to the repository:

Mandatory fields
  • github_repo = “your github repository”
  • application = “your application”
  • github_team = “your github team”
  • application_insights_instance = "dev, preprod or prod
  • source_template_repo = “hmpps-template-typescript” or “hmpps-template-kotlin” or “none” This is required to deploy the correct Application Insights configuration
Default fields
  • is_production = var.is_production
    This inherits the is_production namespace variable
  • environment = var.environment namespace variable for the environment; this must match the environment name used in helm values file e.g. values-dev.yaml
Optional fields
  • reviewer_teams = [“List”, “Of”, “Github”, “Teams”] If this is set, only the teams listed will be allowed to deploy to this environment. Comment it out if it’s not needed.

Note: reviewer_teams is required if is_production is true.

  • selected_branch_patterns = ["branch, "patterns/"] If this is set, the selected branches will be allowed to be deployed.
  • protected_branches_only = true Comment this out if selected branches is in use
    Note: One or other of the above needs to be set.

Once this PR has been approved and the Terraform has run, the Github repositories to which this namespace applies will have a corresponding environment entry within the settings:

github environment

Potential issues - duplicates and out-of-date github versions

Within the template-typescript.tf there is a definition of a redis instance. If this is defined separately within the terraform files for this namespace (eg. if it is a shared resource across instances), it should be removed from the this definition, since it will otherwise end up failing to deploy due to duplicate definitions.

It may also be that the required version of the github module within the resources/versions.tf file of the existing namespace may be too old - this needs to be set to ">= 6.5.0" (as per the template namespace)- it will not affect any existing terraform modules, but it’s required for this one to work.

Creating the build/test/deploy workflow

A boilerplate Github workflow file can be found in the appropriate template repository:

migrate-repo.sh script

A script has been written that is intended to ‘get you working’ based on the template pipelines with the same deployment environments as the existing CircleCI config.

This can be run from your project with this command:

/bin/bash -c "$(curl -fsSL https://github.com/ministryofjustice/hmpps-github-actions/raw/refs/heads/main/migrate-repo.sh)"

In short, the script does a number of things, including:

  • migrating the build/test/deploy steps (including basic branch filtering and specific configurations) for each environment
  • configuring basic equivalents to the ‘executors’ (for example postgres and localstack) to enable integration testing
  • removing the build/test/deploy job from CircleCI so it doesn’t run both CircleCI and Github jobs at the same time

More information about this script is available in the repository: hmpps-github-actions/docs/workflow-migration.md

Whether or not you choose to run the migration script (it will make a backup of the exisiting CircleCI config file so it can be reverted if necessary), it is strongly recommended that you refer to the original config to make sure that Github Actions is doing what CircleCI used to do.

In general, it should be fairly obvious which changes should be made to the the pipeline file so it behaves in a similar manner.

Removing CircleCI

Here are the processes required to remove CircleCI from the process, and replace the elements with Github Actions.

  • Remove .circleci/config.yml
  • On the Github site:
    • within Settings -> Webhooks, delete https://circleci.com/hooks/github
    • within Settings -> Branches -> main, there will be some status checks that include ci/circleci:
      • for each status check, there will be a corresponding one for Github Actions - enter the status check names in the search box and add the non-CircleCI response
      • then remove the circle CI one

That completes the move of the app to Github Actions

Appendix - Application Insights

There is a new method by which the Application Insights secret is deployed. This uses an SSM value that’s referenced within an appinsights.tf resource within the hmpps-templates-dev namespace definition.

This creates a secret called application-insights within the namespace with the corresponding access key.

With existing (CircleCI deployed) apps, the Application Insights secret is added to the application’s secrets during the bootstrap process, and referred to in the Helm values.yaml configuration:

  env:
    APPLICATIONINSIGHTS_CONNECTION_STRING: "InstrumentationKey=$(APPINSIGHTS_INSTRUMENTATIONKEY)"
    .
    .

  namespace_secrets:
    hmpps-your-application:
      APPINSIGHTS_INSTRUMENTATIONKEY: 'APPINSIGHTS_INSTRUMENTATIONKEY'
.
.

To use the Terraform managed secret (which new projects use by default), it’s a simple case of adding the appinsights.tf file to your namespaces (ensuring that your var.environment is set to either dev, preprod or prod) and then modifying the values.yaml file to move the secret to its own application - application-insights

  namespace_secrets:
    hmpps-your-application:
      EXISTING_ENV: EXISTING_SECRET_VALUE
      .
      .

    application-insights:
      APPLICATIONINSIGHTS_CONNECTION_STRING: "APPLICATIONINSIGHTS_CONNECTION_STRING"
  .
  .

Note that if your application is fairly old then it could be you just reference APPINSIGHTS_INSTRUMENTATIONKEY and don’t reference APPLICATIONINSIGHTS_CONNECTION_STRING at all in your configuration.
For backend applications that use the application insights java agent it is fine to use the new configuration with the latest java agent.
For frontend applications you will have to switch to using the APPLICATIONINSIGHTS_CONNECTION_STRING instead. See the template typescript project for how to switch over. This needs to happen before
March 2025 anyway since the old style key will stop being supported past that date.

This page was last reviewed on 28-Oct-2024, next review will be on 28-Jam-2025.
Edit this page here.