A tool to help with some Terraform operations.
For the time being it generates state migration scripts that work also for Terraform workspaces.
The idea of migrations comes from tfmigrate. Then this blog post made me realize that terraform state mv had a bug and how to workaround it.
DISCLAIMER Manipulating Terraform state is inherently dangerous. It is your responsibility to be careful and ensure you UNDERSTAND what you are doing.
This is BETA code, although we already use it in production.
The project follows semantic versioning. In particular, we are currently at major version 0: anything MAY change at any time. The public API SHOULD NOT be considered stable.
The overall approach is for Terravalet to generate migration scripts, not to perform any changes directly. This for two reasons:
- Safety. The operator can review the generated migration scripts for correctness.
- Gitops-style. The migration scripts are meant to be stored in git in the same branch (and thus same PR) that performs the Terraform changes and can optionally be hooked to an automatic deployment system.
Terravalet takes as input the output of terraform plan per each involved root module and generates one UP and one DOWN migration script.
At least until Terraform 0.14, terraform state mv has a bug: if a remote backend for the state is configured (which will always be the case for prod), it will remove entries from the remote state but it will not add entries to it. It will fail silently and leave an empty backup file, so you loose your state.
For this reason Terravalet operates on local state and leaves to the operator to perform terraform state pull and terraform state push.
Be careful when using Terraform workspaces, since they are invisible and persistent global state :-(. Remember to always explicitly run terraform workspace select before anything else.
There are two modes of operation:
- Rename resources within the same state, with optional fuzzy match.
- Move resources from one state to another.
they will be explained in the following sections.
You can also look at the tests and in particular at the files below testdata/ for a rough idea.
Only one Terraform root module (and thus only one state) is involved. This actually covers two different use cases:
- Renaming resources within the same root module.
- Moving resources to/from a non-root Terraform module (this will actually rename the resources, since they will get or loose the
module.prefix).
$ cd $ROOT_MODULE_DIR
$ terraform workspace select $WS
$ terraform plan -no-color 2>&1 | tee plan.txt
$ terraform state pull > local.tfstate
$ cp local.tfstate local.tfstate.BACK
The backup is needed to recover in case of errors. It must be done now.
Take as input the Terraform plan plan.txt (explicit) and the local state local.tfstate (implicit) and generate UP and DOWN migration scripts:
$ terravalet rename \
-plan plan.txt -up 001_TITLE.up.sh -down 001_TITLE.down.sh
Depending on how the elements have been renamed in the Terraform configuration, it is possible that the exact match will fail:
$ terravalet rename \
-plan plan.txt -up 001_TITLE.up.sh -down 001_TITLE.down.sh
match_exact:
unmatched create:
aws_route53_record.private["foo"]
unmatched destroy:
aws_route53_record.foo_private
In this case, you can attempt fuzzy matching.
WARNING Fuzzy match can make mistakes. It is up to you to validate that the migration makes sense.
If the exact match failed, it is possible to enable q-gram distance fuzzy matching with the -fuzzy-match flag:
$ terravalet rename-fuzzy-match \
-plan plan.txt -up 001_TITLE.up.sh -down 001_TITLE.down.sh
WARNING fuzzy match enabled. Double-check the following matches:
9 aws_route53_record.foo_private -> aws_route53_record.private["foo"]
- Review the contents of
001_TITLE.up.sh. - Run it:
sh ./001_TITLE.up.sh
terraform state push local.tfstate. In case of error, DO NOT FORCE the push unless you understand very well what you are doing.
Push the local.tfstate.BACK.
Two Terraform root modules (and thus two states) are involved. The names of the resources stay the same, but we move them from the $SRC_ROOT root module to the $DST_ROOT root module.
Source root:
$ cd $SRC_ROOT
$ terraform workspace select $WS
$ terraform plan -no-color 2>&1 | tee src-plan.txt
$ terraform state pull > local.tfstate
$ cp local.tfstate local.tfstate.BACK
Destination root:
$ cd $DST_ROOT
$ terraform workspace select $WS
$ terraform plan -no-color 2>&1 | tee dst-plan.txt
$ terraform state pull > local.tfstate
$ cp local.tfstate local.tfstate.BACK
The backups are needed to recover in case of errors. They must be done now.
Take as input the two Terraform plans src-plan.txt, dst-plan.txt, the two local state files in the corresponding directories and generate UP and DOWN migration scripts.
Assuming the following directory layout, where repo is the top-level directory and src and dst are the two Terraform root modules:
repo/
├── src/
├── dst/
the generated migration scripts will be easier to understand and portable from one operator to another if you run terravalet from the repo directory and use relative paths:
$ cd repo
$ terravalet move \
-src-plan src/src-plan.txt -dst-plan dst/dst-plan.txt \
-src-state src/local.tfstate -dst-state dst/local.tfstate \
-up 001_TITLE.up.sh -down 001_TITLE.down.sh
- Review the contents of
001_TITLE.up.sh. - Run it:
sh ./001_TITLE.up.sh
In case of error, DO NOT FORCE the push unless you understand very well what you are doing.
$ cd src
$ terraform state push local.tfstate
and
$ cd dst
$ terraform state push local.tfstate
Push the two backups src/local.tfstate.BACK and dst/local.tfstate.BACK.
- Download the archive for your platform from the releases page.
- Unarchive and copy the
terravaletexecutable somewhere in your$PATH.
- Install Go.
- Install task.
- Run
task$ task - Copy the executable
bin/terravaletto a directory in your$PATH.
- Install github-release.
- Install gopass or equivalent.
- Configure a GitHub token:
3.1 Go to Personal Access tokens
3.2 Click on "Generate new token"
3.3 Select only the
reposcope - Store the token securely with a tool like
gopass. The nameGITHUB_TOKENis expected bygithub-release$ gopass insert gh/terravalet/GITHUB_TOKEN
- Update CHANGELOG
- Update this README and/or additional documentation.
- Commit and push.
- Begin the release process with
$ env RELEASE_TAG=v0.1.0 gopass env gh/terravalet task release - Finish the release process by following the instructions printed by
taskabove. - To recover from a half-baked release, see the hints in the Taskfile.
This code is released under the MIT license, see file LICENSE.