Terraform statefile refresh for existing Github Teams not included in code
Github Users and Teams are maintained using terraform hmpps-github-teams. Currently there are many teams which already exists in Github and not added in code. This document covers the step to add the teams in terraform statefile without recreating them.
1. Retrieve AWS account details
2. Login to AWS account
3. Backup existing statefile on local machine
4. Get existing Team and Team members details
5. Update repository Terraform files (terraform/teams.tf and terraform/users.tf) to include existing team details
6. Run terraform plan to get the plan details to check only new teams are included in plan.
7. Import the existing team from github in AWS S3 terraform statefile.
8. Import team members from github in AWS S3 terraform statefile
9. Push local terraform code changes to Github Repository
** The example below comes from a recent hmpps-book-a-video-link import into the statefile**
1. Retrieve AWS account details
AWS account details are stored in the hmpps-portfolio-management-prod namespace.
kubectl get secrets -n hmpps-portfolio-management-prod s3-bucket-output -o json | jq '.data | map_values(@base64d)'
You will need the values for bucket_name, s3_access_key_id and s3_secret_access_key.
Optional:
Export the bucket name to a variable
export s3_bucket=$(kubectl get secrets -n hmpps-portfolio-management-prod s3-bucket-output -o json | jq '.data.bucket_name | @base64d' | tr -d \")
2. Login to AWS account
Provide the decoded values for AWS Access Key ID and AWS Secret Access Key.
$ aws configure AWS Access Key ID [****************]: ******** AWS Secret Access Key [****************]: ******* Default region name [eu-west-2]: Default output format [json]:
3. Backup existing statefile on local machine
Save a local copy of the existing statefile before importing existing teams.
aws s3 cp s3://${s3_bucket}/hmpps-github-teams-dev.tfstate ~/.
Setup terraform locally
Setup the terraform locally.
- Remove the existing .terraform directory
cd hmpps-github-teams/terraform rm -fr .terraform
- Copy the existing state to your local drive:
aws s3 cp s3://${s3_bucket}/hmpps-github-teams-dev.tfstate ~/.
response example:
download: s3://cloud-platform-88ea07d3a3d448b5ba79bfaf76b15457/hmpps-github-teams-dev.tfstate to ./hmpps-github-teams-dev.tfstate
- Initiate the terraform backend
terraform init
reponse example:
Initializing the backend... Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Initializing provider plugins... - Finding hashicorp/aws versions matching "~> 5.0"... - Finding integrations/github versions matching "~> 6.0"... - Installing hashicorp/aws v5.74.0... - Installed hashicorp/aws v5.74.0 (signed by HashiCorp) - Installing integrations/github v6.3.1... - Installed integrations/github v6.3.1 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2) . .
4. Get existing Team and Team members details
To include a team in hmpps-developers, you will need:
- The team name
- The team ID
- The team description
- The team parent (
hmpps-developers) - The team members
This command will return JSON including the team name and parent (which should be null)
example:
gh api -X GET 'orgs/ministryofjustice/teams/hmpps-book-a-video-link' | jq '{name, id, description, parent}'
response example:
{
"name": "hmpps-book-a-video-link",
"id": 10891591,
"description": "HMPPS book a video link team",
"parent": null
}
This command will return the list of members of this team:
example:
$ gh api -X GET 'orgs/ministryofjustice/teams/hmpps-book-a-video-link/members' |jq '.[].login'
response example:
"xxx-moj" "yyy-moj" "another-user" "bazz"
5. Update repository Terraform files (terraform/teams.tf and terraform/users.tf) to include existing team details
Create a new branch on within hmpps-github-teams, and update the terraform/teams.tf to include the team details retrieved from above commands:
{
name = "hmpps-book-a-video-link"
parent = "hmpps-developers"
description = "HMPPS book a video link team"
},
Then update terraform/users.tf to include team members in this team for all the teams members.
{
full_name = "*****",
email = "******@justice.gov.uk"
github_username = "*****"
github_teams = ["activities-and-appointments", "connect-dps-collaborators-devs", "connect-dps-collaborators-live", "hmpps-book-a-video-link"]
},
6. Run terraform plan to get the plan details to check only new teams are included in plan.
terraform plan
example output:
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
~ update in-place
Terraform will perform the following actions:
# github_team.parent_team["hmpps-book-a-video-link"] will be created
+ resource "github_team" "parent_team" {
+ create_default_maintainer = true
+ description = "HMPPS book a video link team • This team is managed by Terraform, see https://github.com/ministryofjustice/hmpps-github-teams - DO NOT UPDATE MANUALLY!"
+ etag = (known after apply)
+ id = (known after apply)
+ members_count = (known after apply)
+ name = "hmpps-book-a-video-link"
+ node_id = (known after apply)
+ parent_team_id = "4043920"
+ parent_team_read_id = (known after apply)
+ parent_team_read_slug = (known after apply)
+ privacy = "closed"
+ slug = (known after apply)
}
# github_team_members.hmpps_developers_members will be updated in-place
~ resource "github_team_members" "hmpps-book-a-video-link" {
id = "4043920"
# (1 unchanged attribute hidden)
+ members {
+ role = "member"
+ username = "*******"
}
# (183 unchanged blocks hidden)
}
# github_team_members.members["hmpps-book-a-video-link"] will be updated in-place
~ resource "github_team_members" "members" {
id = "10891591"
# (1 unchanged attribute hidden)
+ members {
+ role = "member"
+ username = "*****"
}
# (13 unchanged blocks hidden)
}
Plan: 1 to add, 1 to change, 0 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.
7. Import the existing team from github in AWS S3 terraform statefile.
The following command will import into the statefile on AWS S3 bucket.
example:
terraform import 'github_team.parent_team["hmpps-book-a-video-link"]' hmpps-book-a-video-link
example response:
data.github_team.parent: Reading... data.github_team.parent: Read complete after 4s [id=4043920] github_team.parent_team["hmpps-book-a-video-link"]: Importing from ID "hmpps-book-a-video-link"... github_team.parent_team["hmpps-book-a-video-link"]: Import prepared! Prepared github_team for import github_team.parent_team["hmpps-book-a-video-link"]: Refreshing state... [id=10891591] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform. $
8. Import team members from github in AWS S3 terraform statefile
The following will import the team members into the statefile within the AWS S3 bucket.
Replace 10891591 with relevant teams id retrieved from Link
example
terraform import 'github_team_members.members["hmpps-book-a-video-link"]' 10891591
example response:
data.github_team.parent: Reading... data.github_team.parent: Read complete after 3s [id=4043920] github_team_members.members["hmpps-book-a-video-link"]: Importing from ID "10891591"... github_team_members.members["hmpps-book-a-video-link"]: Import prepared! Prepared github_team_members for import github_team_members.members["hmpps-book-a-video-link"]: Refreshing state... [id=10891591] Import successful! The resources that were imported are shown above. These resources are now in your Terraform state and will henceforth be managed by Terraform.
9. Push local terraform code changes to Github Repository
Finally, push the branch and check the plan in the output of the actions. It should not show any changes as the statefile now has a full set of team details. If there are no issues, raise a PR to merge the branch to main
Edit this page here.