Creating A New Service (Github Action)

Introduction

By using the application platform shared components it is possible to get a skeleton service into production very quickly. We recommend getting a skeleton application deployed all the way to production as early as possible. This removes any surprises you might encounter with deploying your application into production and allows you to start adding value to users as early as possible. Note that depending upon your project you may wish to limit access to your production service using an IP allowlist. Control access by using roles assigned to users or services - ask in MOJ Slack channel #hmpps-auth-audit-registers for more details.

It is also a work in progress - if, when following this document you encounter any issues, please raise them in #ask-prisons-digital-sre

Prerequisites

  • You must be a member of a team in the Ministry of Justice GitHub organisation. That team must belong to parent team HMPPS Developers.
  • The team must have an MOJ Slack channel. You should have two slack channels dedicated to send alerts from Github Actions for failed jobs etc. Non-prod and Prod slack channel by component or by team
    Eg: SRE we have #hmpps-sre-alerts and #hmpps-sre-alerts-nonprod.
  • The new service needs a good name. Note that by convention we use the service name in the DNS domain used by the service. As this will have a size limit try not to make your service name too long - a maximum of 25 characters should be ok.
  • Decide which DNS domain to use depending on your user base. The following domains are already available and require no extra work. Any others will need assistance from Cloud Platform.
    • .hmpps.service.justice.gov.uk
    • .prison.service.justice.gov.uk
    • .probation.service.justice.gov.uk
  • git clone the cloud-platform-environments git repository to your local machine.
  • Decide upon the type of project - for backend projects we will be using the Kotlin Template and for frontend projects we will be using the Typescript template.
  • Install the Kubernetes CLI kubectl

Creating Service

  1. Generating your project and build pipeline
  2. Creating resources in Cloud Platform
  3. Customising Github Project
  4. Configuring the build pipeline
  5. Developing locally

Generating your project and build pipeline

The easiest way to create a new application is using the project bootstrapper. This will create everything you need to get your skeleton project into production.

Bootstrapping the project

Current process is as below, it will be replaced with Github Repository Request Form submission from Developer Portal. Work in progress

Clone the bootstrap GitHub project from the link in the project bootstrapper section and take a look at the README.

  • Create a new branch
  • Add your project to the projects.json file. You should be able to copy from the README example for deploying from a template. The value of github_template_repo should be hmpps-template-kotlin for backend projects or hmpps-template-typescript for frontend projects.
  • Raise a Pull Request and ask for a review in the MOJ Slack channel #hmpps-dev or in #ask-prisons-digital-sre.
  • Once merged, ask again in #ask-prisons-digital-sre for somebody to apply the changes.

Once complete you will have a new project on Github and service catalog populated.

  • Find the project on GitHub. (Don’t worry about the instructions in the README - we’ll do that next).
  • Find the project on Developer Portal to make sure the service catalog entries added.

Creating resources in Cloud Platform

Namespaces

Cloud Platforms host a Kubernetes cluster which is logically divided into namespaces aka environments. We deploy applications into these namespaces using Helm. Each application will generally require three namespaces/environments - dev, preprod and prod.

Sometimes it makes sense to group applications together into a single namespace, for example you may have a related UI and API that belong together. This is fine but we would still require multiple environment namespaces for dev, preprod and prod.

Sometimes we need additional environment namespaces, for example some probation applications also have a staging namespace to work with the Delius staging environment. This is fine but will mean some extra work in the helm configurations when copying the template applications.

Do I need a new namespace?

If your application is in a new service area you will definitely need namespaces for the dev, preprod and prod environments. If your application fits nicely alongside existing application(s) you can skip creating a new namespace and reuse the existing ones. However, you will probably need to create a Kubernetes certificate resource for your new application.

Creating new namespaces

If you do need new namespaces then you will need to add some terraform to the cloud-platform-environments git repository.

Each new namespace needs new Kubernetes resources such as an ingress (to allow inbound traffic), access roles (to give humans and build tools access to the namespace), a certificate (to use TLS) amongst others.

dev namespace

Initially we will create a dev namespace. We’ll refer to this as <your-new-service-name>-dev where <your-new-service-name> is whatever you’ve called your new service.

  • In the cloud-platform-environments repo create a new branch
  • Look in directory namespaces/live-1.cloud-platform.service.justice.gov.uk
  • The github Kotlin and Typescript template projects are both configured to use the hmpps-templates-dev directory, and it can be used as the basis of your project’s namespace configuration for the -dev, -preprod and -prod environments with appropriate changes.
  • Create a copy of this template directory, and rename it, appending dev to your new service name, e.g. <your-new-service-name>-dev

You will now have a bunch of Kubernetes definition files in your new directory. There will be several places that need updating:

  • Change the defaults under directory resources/variables
  • Replace the template namespace references with your new namespace name (e.g. replace all instances of hmpps-template-kotlin with <your-new-service-name>-dev). Some projects will not have a github repository at this point, the following stages after creating the namespaces will create this for you. The cloud-platform.justice.gov.uk/source-code: for you project will be https://github.com/ministryofjustice/<your-new-service-name>.git
  • When finished perform a quick search for the word template in your new directory in case you missed any
  • Add your GitHub team to the rbac yaml - copy the existing subject of kind Group but replace hmpps-sre with your GitHub team. (Note: please copy, do not replace the hmpps-sre Group - it is required for administration tasks on the namespace).
  • Update the certificate’s DNS name. By convention we prefer <your-new-service-name>-dev.hmpps|prison|probation.service.justice.gov.uk - where you choose one of hmpps, prison or probation
  • Check that the certificate’s DNS name isn’t longer than 63 characters - if it is you’ll need to choose a shorter name
  • Make sure the certificate’s property spec.secretName doesn’t include the word dev. This secret name should be the same for all namespaces/environments.

Once your updates are complete push your branch back to the repo and raise a new Pull Request. You’ll then need to ask on MOJ Slack channel #ask_cloud_platform for a review of the PR.

Once approved, merge the PR.

A Concourse job will run in the background which should eventually create your new namespace. When the namespace is ready the command kubectl get namespaces | grep <your-new-service-name>-dev should return your new namespace.

Updating Cloud Platform namespaces

The key to being able to deploy with Github Actions is storing Kubernetes credentials within Github itself. The good news is that Cloud Platforms can facilitate this by creating service accounts that can populate the environments section of Github, and both apply permissions for approval/deployment and store the secrets so that they can be referred to by Github Actions.

Furthermore, these secrets can automatically be rotated, improving security. This section details how to create the service account and populate the environments

Namespaces and mappings

It is assumed that each deployment environment has its own namespace, but this namespace can be shared by a number of components within a product (eg. ui and api).

This is the important bit: for this to make sense, the environments should all match: (links show examples within the template repo)

github Service acccount

Within the hmpps-templates-dev namespace definition there is a file called resources/serviceaccount-github.tf

This should be copied into the appropriate namespace’s resources directory, and the following changes made:

  • github_repos

Each of the repositories that uses this namespace should be listed on row 5

  • github_environments

This refers to the var.environment variable within resources/variables.tf - this is one of the parameters that needs to match with all the others.

  • data “github_team”
  • resource “github_repository_environment” “env” -> reviewers

These configure the review constraints for the environments. Add the appropriate github teams that are authorised to review and approve deployments for this environment, based on the existing entries.

time definition

So that tokens can be rotated, the following entry should be added within resources/versions.tf:

    time = {
      source  = "hashicorp/time"
      version = "~> 0.9.0"
    }

Once this PR has been approved and the Terraform has run, the Github repositories to which this namespace applies will have a corresponding environment entry within the settings page, environment link.

Customising Github Project

Your project is now a copy of the template project. You’ll want to rename some things so you don’t see the word template everywhere. There’s a GitHub Action for this.

  • Go to the GitHub project and select the Actions tab
  • You should see a workflow called rename-project-create-pr - select the workflow from the drop down, fill in information for all the field for slack notification and product id and click the Run workflow button, a banner will appear at the top of the section saying workflow run was successfully requested
  • Once completed click on the Pull Requests tab - you should see a single PR raised by the Action. Review helm_deploy config files for secrets and TLS cert. Review and merge this PR.
  • Update security job workflow cron timings with next PR.

You should now be able to clone the GitHub project, successfully run the tests, start the application locally and see the /health and /info pages.

This is a good time to update README.md to replace the template readme with content more relevant to your service. See the service standards for inspiration.

Configuring the build pipeline

Typescript application secrets

If you are trying to create a Typescript application your deployment is probably failing. There are a few secrets that are only required by Typescript applications. Once these secrets are created your application should start deploying. It’s worth running the command kubectl -n <your-new-service-name>-dev get events to try and spot which secrets are missing.

session secret

The Typescript template project requires a secret which is used to sign session cookies. This is how it knows that cookies haven’t been tampered with when presented by a client. Using the Cloud Platform secrets guide update the secret <your-new-service-name> by adding a new key SESSION_SECRET followed by a random value (don’t forget to base64 encode the value).

Note: Please do not use / configure the Cloud Platform Secrets Manager, since this prevents scheduled secret rotation scripts from running successfully.

authorization_code client

Users sign in to the UI application using the Oauth2 authorization_grant flow. After a successful sign in the UI application calls hmpps-auth to receive a valid access token. We need a client and secret to prove to hmpps-auth that we are allowed to request access tokens. These secrets are stored in Kubernetes secrets API_CLIENT_ID and API_CLIENT_SECRET.

To request a new authorization code client from hmpps-auth raise a ticket on the DPS Tech Team JIRA board in this format. Then ask in MOJ Slack channel #hmpps-auth-audit-registers for the ticket to be prioritised. Once the clients have been created in hmpps-auth the DPS Tech team will copy the secrets into your namespace.

client_credentials client

When a UI application needs to call an API it must prove that it is authorised to call that API using the Oauth2 client_credentials flow. The UI application calls hmpps-auth to request an access token it can present to other APIs as authorisation. We need a client and secret to prove to hmpps-auth we can have an access token. These secrets are stored in Kubernetes secrets SYSTEM_CLIENT_ID and SYSTEM_CLIENT_SECRET.

To request a new client credentials client from hmpps-auth raise a ticket on the DPS Tech Team JIRA board in this format. Then ask in MOJ Slack channel #hmpps-auth-audit-registers for the ticket to be prioritised. Once the clients have been created in hmpps-auth the DPS Tech team will copy the secrets into your namespace.

Checking the application is healthy

We have now deployed the application to dev and need to check it is up and running as expected. Run command kubectl -n <your-new-service-name>-dev get ingress where you should find the host of your application. In a browser go to page https://<hostname>/health and you should see a health page with status UP.

Troubleshooting - Missing certificate

If you see an error such as Your connection is not private or Potential security risk ahead then it implies that the application is not using the certificate we generated with terraform earlier. Look in project file helm_deploy/<your-new-service-name>/values.yaml for property generic-service.ingress.tlsSecretName - this is the secret we copy the certificate from. Then run command kubectl -n <your-new-service>-dev get secrets to find a secret of type kubernetes.io/tls - this is the secret holding the certificate and should match the secret name in our project. If not then go back to the cloud-platform-environments project for dev and find the certificates yaml. Change the property spec.secretName to match the real secret name and issue a PR. You may also need to do this for preprod and prod.

Promoting to preprod

In the new project under the helm_deploy directory you should see a file called values-dev.yaml. This contains overrides to helm deployment configuration for the dev namespace. Create a values-preprod.yaml file in the same format. For Kotlin template projects this needs overrides to the generic-service.replicaCount, the generic-service.ingress.host and the generic-prometheus-alerts.alertSeverity properties. For Typescript template projects copy values-dev.yaml and update all properties in the generic-service section.

If you look at the project in Github Action workflow .github/workflows/pipeline.yml you should see that the last step in the jobs is deploy_dev is currently uncommented. In .github/workflows/pipeline.yml you will see a big section commented out - this contains extra steps to deploy to preprod and prod. Uncomment deploy_preprod deployment step.

Once these changes have been made raise a PR and merge when reviewed. Your Github Action build post deploy to dev completion, should show request approval to deploy to preprod. Approve the preprod approval in Github actions workflow runs. Once deployed check that the deployment is healthy in a similar way to checking the dev deployment but replacing dev with preprod.

If you are creating a Typescript application you may still have a failing deployment. Make sure you create the same secrets in preprod that you created for dev.

Adding an external health check to prod

For production we use an external health check monitor called pingdom. This checks the application is up and can be reached from outside of our networks.

  • In the cloud-platform-environments project add a pingdom health check for the application. Find another production namespace (e.g. hmpps-audit-prod) and copy file resources/pingdom.tf into your new directory. Update the name and change the url to match your certificate’s dnsName.
  • Add pingdom version details to the file resources/versions.tf (copy an existing production namespace such as hmpps-audit-prod).
  • Raise a PR, ask in MOJ Slack channel #ask-cloud-platform for a review and merge when approved.
  • As we haven’t deployed to prod yet we should receive an alert from pingdom saying that the application is down. This proves that the pingdom check is working so wait for the alert before promoting to prod.

Promoting to prod

In the new project under the helm_deploy directory you should see a file called values-dev.yaml. This contains overrides to helm deployment configuration for the dev namespace. Create a values-prod.yaml file in the same format. For Kotlin template projects this needs overrides to the generic-service.replicaCount, the generic-service.ingress.host and the generic-prometheus-alerts.alertSeverity properties. For Typescript template projects copy values-dev.yaml and update all properties in the generic-service section.

If you look at the project in Github Action workflow .github/workflows/pipeline.yml you should see that the last step in the jobs is deploy_dev is currently uncommented. In .github/workflows/pipeline.yml you will see a big section commented out - this contains extra steps to deploy to prod. Uncomment deploy_prod deployment step.

Once these changes have been made raise a PR and merge when reviewed. Your Github Action build post deploy to prod/dev completion based on steps in cluded in jobs section, should show request approval to deploy to prod. Approve the prod approval in Github actions workflow runs. Once deployed check that the deployment is healthy in a similar way to checking the dev deployment but replacing dev with preprod.

Developing locally

With our growing number of service dependencies (APIs, DBs, etc) it can sometimes be impractical to spin up dependencies locally, with Docker for instance. In which case developers prefer to point their locally running application at our ‘dev’ environment. This therefore requires credentials to authenticate with HMPPS Auth in dev.

We used to require that developers request their own personal credentials that would mirror their service’s configuration and roles. We are no longer considering this necessary for the dev environment.

We will relax the IP allow listing to support use of dev service credentials within MoJ GlobalProtect and MoJ Digital VPN IP ranges. This will reduce the support burden on our team and avoid issues where developer and service credential configuration gets out of step.

The default process to follow will now be:

  • Request HMPPS Auth credentials as normal for your service and store them as Kubernetes secrets
  • When a developer needs credentials to develop against the dev environment, they can inspect their service’s Kubernetes secrets for the credentials they need and set them as local environment variables.

Personal credentials can still be requested if required (for maintenance of queues for example).

Note (1): Because preprod and prod environments contain production data, we are still restricting service credentials to the service host (e.g. Cloud Platform) IP ranges in these environments. This means that if you need to call preprod or prod APIs from your local machine, and you are SC cleared or have a suitable waiver, you’ll still need to request personal client credentials.
Note (2): Remember that the credentials are for a particular OAuth2 flow (authorization_code or client_credentials). This means, for example, that you won’t be able to use your service’s authorization_code credentials to make a client_credentials token request for calling an API.

This page was last reviewed on 07-Sep-2024, next review will be on 07-Jan-2024.