Menu
Article DevOps

TIL How to write a Python CLI tool that writes Terraform YAML

Mathew Duggan

• 3 min read

I'm trying to use more YAML in my Terraform as a source of truth instead of endlessly repeating the creation of resources and to make CLIs to automate the creation of the YAML. One area that I've had a lot of luck with this is GCP IAM. This is due to a limitation in GCP that doesn't allow for the combination of pre-existing IAM roles into custom roles, which is annoying. I end up needing to assigns people the same permissions to many different projects and wanted to come up with an easier way to do this.

I did run into one small problem. When attempting to write out the YAML file, PyYAML was inserting these strange YAML tags into the end file that looked like this: !!python/tuple.

It turns out this is intended behavior, as PyYAML is serializing arbitrary objects as generic YAML, it is inserting deserialization hint tags. This would break the Terraform yamldecode as it couldn't understand the tags being inserted. The breaking code looks as follows.

with open(path,'r') as yamlfile:
    current_yaml = yaml.safe_load(yamlfile)
    current_yaml['iam_roles'].append(permissions)

if current_yaml:
    with open(path,'w') as yamlfile:
        yaml.encoding = None
        yaml.dump(current_yaml, yamlfile, indent=4, sort_keys=False)

I ended up stumbling across a custom Emitter setting to fix this issue for Terraform. This is probably not a safe option to enable, but it does seem to work for me and does what I would expect.

The flag is: yaml.emitter.Emitter.prepare_tag = lambda self, tag: ''

So the whole thing, including the click elements looks as follows.

import click
import yaml

@click.command()

@click.option('--desc', prompt='For what is this role for? Example: analytics-developer, devops, etc', help='Grouping to assign in yaml for searching')

@click.option('--role', prompt='What GCP role do you want to assign?', help="All GCP premade roles can be found here: https://cloud.google.com/iam/docs/understanding-roles#basic")

@click.option('--assignment', prompt="Who is this role assigned to?", help="This needs the syntax group:, serviceAccount: or user: before the string. Example: group:[email protected] or serviceAccount:[email protected]")

@click.option('--path', prompt="Enter the relative path to the yaml you want to modify.", help="This is the relative path from this script to the yaml file you wish to append to", default='project-roles.yaml')

@click.option('--projects', multiple=True, type=click.Choice(['test', 'example1', 'example2', 'example3']))

def iam_augmenter(path, desc, role, assignment, projects):
    permissions = {}
    permissions["desc"] = desc
    permissions["role"] = role
    permissions["assignment"] = assignment
    permissions["projects"] = projects

    with open(path,'r') as yamlfile:
        current_yaml = yaml.safe_load(yamlfile)
        current_yaml['iam_roles'].append(permissions)

    if current_yaml:
        with open(path,'w') as yamlfile:
            yaml.emitter.Emitter.prepare_tag = lambda self, tag: ''
            yaml.encoding = None
            yaml.dump(current_yaml, yamlfile, indent=4, sort_keys=False)

if __name__ == '__main__':
    iam_augmenter()

This worked as intended, allowing me to easily append to an existing YAML file with the following format:

iam_roles:
  - desc: analytics-reader-bigquery-data-viewer
    role: roles/bigquery.dataViewer
    assignment: group:[email protected]
    projects:
    - example1
    - example2
    - example3

This allowed me to easily add the whole thing to automation that can be called from a variety of locations, meaning we can keep using the YAML file as the source of truth but quickly append to it from different sources. Figured I would share as this took me an hour to figure out and maybe it'll save you some time.

The Terraform that parses the file looks like this:

locals {
  all_iam_roles = yamldecode(file("project-roles.yaml"))["iam_roles"]


  stock_roles = flatten([for iam_role in local.all_iam_roles :
    {
      "description" = "${iam_role.desc}"
      "role"        = "${iam_role.role}"
      "member"      = "${iam_role.assignment}"
      "project"     = "${iam_role.projects}"
    }
  ])
  
  # Shortname for projects to full names
  test          = "test-dev"
  example1      = "example1-dev"
  example2      = "example2-dev"
  example3      = "example3-dev"
}

resource "google_project_iam_member" "test-dev" {
  for_each = {
    for x in local.stock_roles : x.description => x
    if contains(x.project, local.test) == true
  }
  project = local.test
  role    = each.value.role
  member  = each.value.member
}

resource "google_project_iam_member" "example1-dev" {
  for_each = {
    for x in local.stock_roles : x.description => x
    if contains(x.project, local.example1) == true
  }
  project = local.example1
  role    = each.value.role
  member  = each.value.member
}

Hopefully this provides someone out there in GCP land some help with handling large numbers of IAM permissions. I've found it to be much easier to wrangle as a Python CLI that I can hook up to different sources.

Did I miss something or do you have questions I didn't address? Hit me up on Mastodon: https://c.im/@matdevdug

Share