Migrating off Terraform Cloud

I moved off of Terraform Cloud. Or rather, a quick Google search suggests I’ve moved off of “HashiCorp Cloud Platform”, which was formerly Terraform Cloud. The top three entries from my search are all ads for alternatives, with two literally showing up as “Terraform Cloud Alternative”. Cue the Spider-Man meme.

I don’t remember why it seemed like a good idea to use TF Cloud for this website two years ago—I think some former job corrupted my sensibilities—but I went with it thinking I’d get better visibility and more management than if I used an S3 bucket. I guess both of those things were true: the UI did show me my state in a decently-readable way, and the plans and applies all took place on HashiCorp servers with some nice options for passing in variables, and for handling auth with GitHub actions and AWS. Overall it was actually a decent product.

However, and this is why I joke that my sensibilities must have been corrupted, it’s not the right product for a solo engineer working on a personal project. “Terraform Cloud” might’ve been a good product for my use case two years ago (I honestly can’t remember), but “HashiCorp Cloud Platform” is undoubtedly targeting enterprises, and the value props are aimed accordingly.

In an enterprise it makes sense to have highly-managed runs. You don’t want to deal with locking when there are dozens or hundreds of engineers committing code that’s going to run plans and applies on the same TF codebase. You do want to be able to see your state file at a glance to debug and triage issues. You also want drift detection, in case folks have been poking around in the AWS console. And since you’ll certainly be paying big bucks for the enterprise plan and support, there won’t be limits on how many resources you can have, or how many projects you can stand up. They’re happy to take the caps off for folks who pay, and the end result is that it makes an enterprise’s infrastructure as code simpler.

For a single person or a small team though, I’d argue that it’s actually more complex to be managed. I’m not willing to pay anything since I don’t need sophisticated locking or easily-readable state. I can store my state in S3 and I have permissions to grab it from S3 to look at it if I need to. And with S3 as my backend, I know I can store my state practically for free. (At the time of writing it’s about 2 cents per GB per month to use S3, and my state files are on the order of KB, so we’re talking millionths of a cent—so low that AWS won’t bother charging you). Plus, I can run my Terraform for free from my laptop, or for free from GitHub Actions if I don’t overdo it in a single month. Using HashiCorp Cloud Platform isn’t adding anything to my CI/CD flow that I actually need, but it is adding the cognitive overhead of juggling yet another vendor, along with the stress of knowing that I’m limited in how many resources and projects I can create.

As an aside, I’ve seen companies fall in the same trap: folks get lured in by shiny solutions to problems they don’t actually have (usually because they haven’t thought that hard about what their problems are), and next thing you know they’re paying an arm and a leg and feeling good about it because they solved an imagined issue. I think JIRA is a great example of this: everybody pays for JIRA, so everyone else thinks they should pay for JIRA too, as if using it guarantees that your team will be estimating and planning and scheduling well. But the real problem for most teams is they just don’t know how to plan. I’ve only worked at one place that did a good job planning, and that was because they actually followed Scrum in its entirety, rather than cherry-picking pieces of it and calling it a day. They did not use JIRA. Every other place I’ve been has used JIRA and their planning processes have either been whims and gut checks, or 6-month waterfalls disguised by sprints.

In short, keep it simple. Solve the problems you actually have. Over-engineering for small or non-existent problems breeds complexity and ironically creates problems.

Migration steps

Stepping off my soapbox for a second, let me get to what I actually wanted to say: the official documentation for migrating off of Terraform Cloud doesn’t work if you’re using the cloud block instead of the remote backend.

In those docs, they say it’s as easy as running terraform init, rewriting your backend block to use s3 instead of remote, then running terraform init -migrate-state. That might be true if you’re using the remote backend, but if you followed their official advice when you set up TF Cloud and used the cloud block instead, it most certainly does not work. You’ll receive this charming error:

Error: Migrating state from Terraform Cloud to another backend is not yet implemented.

Much like adding subscription cancellation buttons to apps, I wouldn’t expect this to get fixed anytime soon (at least not without regulation). I watched that whole debacle at a previous job. Barring the risk of legal action, it just doesn’t make sense for a company to prioritize functionality that makes it easier for users to churn. I’m not saying there’s some malicious plot on HashiCorp’s part to keep people locked in, but let’s be honest: fixing this is pretty low in the backlog.

But luckily for us, there is a workaround. There’s some unofficial documentation living in a GitHub issue titled “Document the migration path from Terraform Cloud to a standard backend”, and the issue is still open as I’m writing this, so I’m assuming the official documentation part is not complete. But if we take a look at that comment, we can see there is a way off of Terraform Cloud.

The basic steps are:

1. cd into the directory where your TF lives
2. Grab any credentials you need and run terraform init
3. Run terraform state pull >temp.tfstate to pull down your state from TF Cloud
4. Remove your .terraform directory to “forget” you’re initialized
5. Remove the cloud block and replace it with your new s3 backend block
6. Run terraform init again
7. Run terraform state push temp.tfstate to push your state up to S3

After that you’d need to migrate any variables and secrets stored in TF Cloud out to whatever system you’re using to plan and apply your Terraform. In my case I didn’t have any variables to move, since I usually define constants in a locals block and then pass in a few things like environment through the CLI. If I did have secrets to migrate though I’d probably be moving them into GitHub Actions Secrets in my repo, since I use GitHub Actions to run my Terraform.

Concluding thoughts

Hopefully this helps anyone stuck in the same boat as me! S3 might not be the ideal backend for all teams in all situations, but remember that simple is good, and simple starts with addressing the problems you actually have, not the ones you think you have. In my case, I thought that two years in I’d have problems Terraform Cloud would’ve preemptively solved for me. The problem I actually had was that the docs aren’t written for the pathological customer. To all my fellow cheapskates: best of luck with your migration!