Terraform statefile refresh for existing Github Teams not included in code

Github Users and Teams are maintained using terraform hmpps-github-teams. Currently there are many teams which already exists in Github and not added in code. This document covers the step to add the teams in terraform statefile without recreating them.

1. Retrieve AWS account details
2. Login to AWS account
3. Backup existing statefile on local machine
4. Get existing Team and Team members details
5. Update repository Terraform files (terraform/teams.tf and terraform/users.tf) to include existing team details
6. Run terraform plan to get the plan details to check only new teams are included in plan.
7. Import the existing team from github in AWS S3 terraform statefile.
8. Import team members from github in AWS S3 terraform statefile
9. Push local terraform code changes to Github Repository

Example below is from recent hmpps-book-a-video-link import in statefile

1. Retrieve AWS account details

AWS account details are stored in hmpps-portfolio-management-dev namespace.

kubectl get secrets -n hmpps-portfolio-management-dev  s3-bucket-output -o yaml

Decode the values for bucket_name, s3_access_key_id and s3_secret_access_key.

echo $VALUE_FROM_ABOVE_COMMAND | base64 --decode 

2. Login to AWS account

Provide the decoded values for AWS Access Key ID and AWS Secret Access Key.

$ aws configure
AWS Access Key ID [****************]: ********
AWS Secret Access Key [****************]: *******
Default region name [eu-west-2]:
Default output format [json]:

3. Backup existing statefile on local machine

Backup the existing statefile before importing existing teams.

aws s3 cp s3://cloud-platform-59ac75c5c19ea6fee9760900194529a5/hmpps-github-teams-dev.tfstate ~/. 

#### Setup terraform locally 

Setup the terraform locally. 

* Remove the existing .terraform directory 
 $ cd hmpps-github-teams/terraform
$ rm -fr .terraform
$ aws s3 cp s3://cloud-platform-59ac75c5c19ea6fee9760900194529a5/hmpps-github-teams-dev.tfstate ~/.
download: s3://cloud-platform-59ac75c5c19ea6fee9760900194529a5/hmpps-github-teams-dev.tfstate to ./hmpps-github-teams-dev.tfstate
$ terraform init
Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Finding integrations/github versions matching "~> 6.0"...
- Installing hashicorp/aws v5.74.0...
- Installed hashicorp/aws v5.74.0 (signed by HashiCorp)
- Installing integrations/github v6.3.1...
- Installed integrations/github v6.3.1 (signed by a HashiCorp partner, key ID 38027F80D7FD5FB2)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$

4. Get existing Team and Team members details
$ gh api -X GET 'orgs/ministryofjustice/teams/hmpps-book-a-video-link'
{
  "name": "hmpps-book-a-video-link",
  "id": 10891591,
  "node_id": "T_kwDOACGfts4ApjFH",
  "slug": "hmpps-book-a-video-link",
  "description": "",
  "privacy": "closed",
  "notification_setting": "notifications_enabled",
  "url": "https://api.github.com/organizations/2203574/team/10891591",
  "html_url": "https://github.com/orgs/ministryofjustice/teams/hmpps-book-a-video-link",
  "members_url": "https://api.github.com/organizations/2203574/team/10891591/members{/member}",
  "repositories_url": "https://api.github.com/organizations/2203574/team/10891591/repos",
  "permission": "pull",
  "created_at": "2024-09-04T08:42:05Z",
  "updated_at": "2024-09-04T08:43:57Z",
  "members_count": 4,
  "repos_count": 0,
  "organization": {
    "login": "ministryofjustice",
    "id": 2203574,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjIyMDM1NzQ=",
    "url": "https://api.github.com/orgs/ministryofjustice",
    "repos_url": "https://api.github.com/orgs/ministryofjustice/repos",
    "events_url": "https://api.github.com/orgs/ministryofjustice/events",
    "hooks_url": "https://api.github.com/orgs/ministryofjustice/hooks",
    "issues_url": "https://api.github.com/orgs/ministryofjustice/issues",
    "members_url": "https://api.github.com/orgs/ministryofjustice/members{/member}",
    "public_members_url": "https://api.github.com/orgs/ministryofjustice/public_members{/member}",
    "avatar_url": "https://avatars.githubusercontent.com/u/2203574?v=4",
    "description": "",
    "name": "Ministry of Justice",
    "company": null,
    "blog": "https://www.justice.gov.uk/",
    "location": "London, UK",
    "email": null,
    "twitter_username": null,
    "is_verified": true,
    "has_organization_projects": true,
    "has_repository_projects": true,
    "public_repos": 2184,
    "public_gists": 0,
    "followers": 567,
    "following": 0,
    "html_url": "https://github.com/ministryofjustice",
    "created_at": "2012-08-23T11:22:38Z",
    "updated_at": "2024-09-16T10:17:47Z",
    "archived_at": null,
    "type": "Organization"
  },
  "parent": null
}
$ gh api -X GET 'orgs/ministryofjustice/teams/hmpps-book-a-video-link/members'
[
  {
    "login": "*****",
    "id": 30229564,
    "node_id": "MDQ6VXNlcjMwMjI5NTY0",
    "avatar_url": "https://avatars.githubusercontent.com/u/30229564?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/*****",
    "html_url": "https://github.com/*****",
    "followers_url": "https://api.github.com/users/*****/followers",
    "following_url": "https://api.github.com/users/*****/following{/other_user}",
    "gists_url": "https://api.github.com/users/*****/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/*****/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/*****/subscriptions",
    "organizations_url": "https://api.github.com/users/*****/orgs",
    "repos_url": "https://api.github.com/users/*****/repos",
    "events_url": "https://api.github.com/users/*****/events{/privacy}",
    "received_events_url": "https://api.github.com/users/*****/received_events",
    "type": "User",
    "user_view_type": "public",
    "site_admin": false
  }
]
$

5. Update repository Terraform files (terraform/teams.tf and terraform/users.tf) to include existing team details
Create branch on hmpps-github-teams, Update terraform/teams.tf to include the details retrieved from above commands
    {
      name        = "hmpps-book-a-video-link"
      parent      = "hmpps-developers"
      description = "HMPPS book a video link team"
    },

Update terraform/users.tf to include team members in this team for all the teams members. 

    {
      full_name       = "*****",
      email           = "******@justice.gov.uk"
      github_username = "*****"
      github_teams    = ["activities-and-appointments", "connect-dps-collaborators-devs", "connect-dps-collaborators-live", "hmpps-book-a-video-link"]
    },

6. Run terraform plan to get the plan details to check only new teams are included in plan.
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
Terraform will perform the following actions:
  # github_team.parent_team["hmpps-book-a-video-link"] will be created
  + resource "github_team" "parent_team" {
      + create_default_maintainer = true
      + description               = "HMPPS book a video link team • This team is managed by Terraform, see https://github.com/ministryofjustice/hmpps-github-teams - DO NOT UPDATE MANUALLY!"
      + etag                      = (known after apply)
      + id                        = (known after apply)
      + members_count             = (known after apply)
      + name                      = "hmpps-book-a-video-link"
      + node_id                   = (known after apply)
      + parent_team_id            = "4043920"
      + parent_team_read_id       = (known after apply)
      + parent_team_read_slug     = (known after apply)
      + privacy                   = "closed"
      + slug                      = (known after apply)
    }
  # github_team_members.hmpps_developers_members will be updated in-place
  ~ resource "github_team_members" "hmpps-book-a-video-link" {
        id      = "4043920"
        # (1 unchanged attribute hidden)
      + members {
          + role     = "member"
          + username = "*******"
        }
        # (183 unchanged blocks hidden)
    }
  # github_team_members.members["hmpps-book-a-video-link"] will be updated in-place
  ~ resource "github_team_members" "members" {
        id      = "10891591"
        # (1 unchanged attribute hidden)
      + members {
          + role     = "member"
          + username = "*****"
        }
        # (13 unchanged blocks hidden)
    }

Plan: 1 to add, 1 to change, 0 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.


7. Import the existing team from github in AWS S3 terraform statefile.
$  terraform import 'github_team.parent_team["hmpps-book-a-video-link"]' hmpps-book-a-video-link
data.github_team.parent: Reading...
data.github_team.parent: Read complete after 4s [id=4043920]
github_team.parent_team["hmpps-book-a-video-link"]: Importing from ID "hmpps-book-a-video-link"...
github_team.parent_team["hmpps-book-a-video-link"]: Import prepared!
  Prepared github_team for import
github_team.parent_team["hmpps-book-a-video-link"]: Refreshing state... [id=10891591]
Import successful!
The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.
$

This import command will import in statefile on AWS S3 bucket.
8. Import team members from github in AWS S3 terraform statefile
Replace 10891591 with relevant teams id retrieved from Link
 $  terraform import 'github_team_members.members["hmpps-book-a-video-link"]' 10891591
data.github_team.parent: Reading...
data.github_team.parent: Read complete after 3s [id=4043920]
github_team_members.members["hmpps-book-a-video-link"]: Importing from ID "10891591"...
github_team_members.members["hmpps-book-a-video-link"]: Import prepared!
  Prepared github_team_members for import
github_team_members.members["hmpps-book-a-video-link"]: Refreshing state... [id=10891591]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

$

This import will import in statefile on AWS S3 bucket.
9. Push local terraform code changes to Github Repository
Push the branch, check the plan, it should not show any changes as statefile already has the team details. If all good merge the branch to main.