Testing on Terraform

Just as an NB: This article is based on Terraform 0.11. The principles should work for Terraform 0.12. There are not a lot of code examples here, but there are links to some.

Anyway, on with the article…

Infrastructure as code. It’s ace, isn’t it? You just declare what you need, run your script and it’s all there waiting.

Sometimes.

Because sometimes it all blows up in your face like a grenade burrito. Do you want to eat a grenade burrito? Because I don’t, I prefer less explosive foods like chips and hobnobs.

So this is why we test. Like normal code, infra code should be tested. But infra code isn’t actually 100% like normal code, especially something like Terraform, which, at least in its pre-0.12 state, is mainly declared config with minimal logic.

What kind of tests?

Somebody reminded me the other day that different people have different definitions of what comprises a unit test, so in my case, I’m going with the definition that any test that requires infra to be spun up is an integration test. Other people’s definitions may vary, but to me, a unit test should pretty much be a no-op test.

So anyway, in the world of infra code Integration Tests are king, unit tests much less so. Why is this?

Because if you declare an instance in Terraform, writing a test to see if it creates an instance is basically testing Hashicorp’s code, which you don’t need to do. Hashicorp have already done that.

I mean, sure you CAN do that, but you have to think about business value and if you’re essentially writing tests for something that will be proven in a much more complete way, ie: an application smoke test, then why would you write that test twice?

What should I test?

I get it, it can be really daunting all this testing stuff. If you’ve lived a life of basic scripts and a few bits of IaC before now, the idea of testing your code and what even to test can be confusing.

There are schools of thought on this. Some say every individual piece should be tested, which means your velocity goes down to zero and any changes immediately become a ballache of epic proportions.

Test what you NEED to test. And to me, that falls into 2 categories.

Functional Testing: Does the thing do what you wanted it to do?

Non Functional Testing: Security and compliance tests, performance tests, etc.

When it comes to functional testing you have to ask, “what am I building?”, if it’s an app on an instance with a database, then for functional tests you need to be able to test that the app is running and the communication is in place for the database. You may even find some of the developer smoke tests can do this for you; you may just need to change some ports and endpoints.

You don’t necessarily need to test every strut is in place in your infra because if it weren’t then it would fall over.

Using a Star Wars example, If your AT-ATs knee joints are broken, then it can’t stand up or walk, so rather than test every knee and hip joint, you just need to ask, “can it stand?”, “can it walk?”

Once you have some functional tests in place you need to make sure you have some good Non-Functional Tests in place. One group you CANNOT ignore, and I mean this, it’s 2019, everything’s getting hacked, you CANNOT ignore security and compliance.

In it’s most basic form in AWS, this could be ensuring that there are no open security groups, or the NACLs are appropriately set to block unwanted traffic. It can, of course, include a lot more, depending on what you spin up.

One that you SHOULD do, but not as part of a production deployment is performance testing.

If you don’t performance test, how do you know your latest whiz-bang change hasn’t introduced a bucketload of latency?

Answer: You don’t.

But whatever you do, don’t put this into your prod deployment pipeline. Performance tests are designed to stress the system. You should already know if that code has introduced latency by running it in a lower environment.

You should certainly have a nightly build that runs performance tests against the environment.

What if I REALLY want to unit test? what are my options?

Not many. And only one viable option. As far as I’m aware the only test framework that doesn’t spin up infra as part of its test cycle is Terraform-Compliance which even brings BDD style declarations into your code. If you decide to use Terraform-Compliance, I’ll allow it!

I’ve been reliably informed that it’s in the process of being updated for 0.12 and some of its idiosyncrasies are being fixed (I had issues with a complex local{} declaration and have been told that will be fixed in the next release)

Some might argue that RSpec-terraform exists, but I’d argue back that there hasn’t been a commit in that repo since 2015, so yeah. Probably best not to use that.

If you MUST unit test, then use Terraform compliance. You’ll still need to integration test though.

I get it, I get it… I should integration test, what should I use?

Well, there’s quite a few options!

If you’re familiar with Go you can use Terratest, although there is a fair amount of boilerplating to be done. This wraps up Terraform rather well and can use Terraform’s outputs to run tests.

But of course, the thing to remember is that once it’s all up and running, it’s not Terraform you’re testing. It’s your deployment! Which means you can use more generic tools to test things.

For example, I tend to use Inspec to do compliance and security testing and have outputs defined in Terraform for things I want to check. Getting the terraform outputs into an easily readable file is as simple as running this:

terraform output --json > inspec/sectest/files/tf_output.json

This assumes the Inspec tests live in inspec/sectest. When you’ve done this, any controls in that test suite will be able to use that JSON file. Then it’s as simple as putting a control in place and getting things like the security_group ID from it.

That’s a basic example, and rather than rewrite someone’s whole post on this, I suggest you read this blog article which gives you a much more in-depth crash course on inspec and some of its uses.

Also, Inspec’s docs are fantastic.

Testing when using modules.

One thing that inevitably happens when working with Terraform is you’ll end up with modules everywhere, spread out across multiple SCM repos, and that can be a nightmare. You can end up feeling like you’re trapped inside an M.C.Escher painting just trying to figure out what’s going on.

So what should you do?

Well, the answer is that you should test every module by itself. That way if something goes horribly wrong, you can be confident that the module you’re using isn’t the problem. Then you hopefully don't have to go down the rabbit hole trying to trace where the problem actually is.

This is where you have to be more aggressive with your testing. As opposed to when you’re building an actual app where you’re just interested in whether your app works and not necessarily the supporting structure, as that’s implied, in this case, it’s the structure you’re interested in.

Modules can be complex and be passed numerous variables to conditionally set things or even conditionally build things. So there can be a number of different cases to test. This is where traditional integration tests can be too slow and something like Terraform-Compliance may be necessary to run unit tests as well on edge cases.

Yes, I just went back on what I said not long ago, but stick with me, I have my reasons.

After all, if a Terraform build can take up to 10 minutes and you have 30 potential scenarios, it’d be nice if you can break that down to say, 5 scenarios you DEFINITELY need to integration test and 25 that you could test in a quick fashion with unit tests.

And of course, the way you split those is in whether the conditionals are likely to interact. You may be lucky and find that you can have two integration tests, one with everything off and one with everything on, but that’s unlikely. A lot of the time in a module you’ll have something with multiple conditions. For example:

count = "${var.create_group && var.force_update && var.lifecycle_hook ? 1 : 0}"

In this case, all three conditionals must be true, and if those conditionals are used elsewhere in the module, then they’re going to have to be tested separately. Whereas something as simple as ${var.name} you could test just once assuming there are no conditionals set on it.

So in cases like this, you need to:

Check every potential option that can cause a change

If you can group these together, then you may be able to reduce the number of test runs but be aware of any potential interactions or multiple conditionals.With some of your conditionals, for example, ones that change text strings only, you may be able to leverage unit testing to make test runs faster.

Make sure you run everything in a pipeline.

I’ve made one big assumption here, and that is that you’ll be running your Terraform and all related tests in a CI/CD pipeline. Be it Jenkins, Gitlab, Travis, CircleCI. Whatever. Because you SHOULD be. Humans forget, pipelines don’t.

This is also a standard for developers, they make a commit, the code is tested automatically and they get feedback as to whether it’s worked.

Running these things is not as complicated as you think. You don’t need to download a million plugins. Most of these CI/CD platforms are happy running shell commands and most will run in a docker image where you can have the tools you need installed.

Your pipeline should fail if your tests fail and give out an exit 1. Most platforms do this as a defacto standard.

And, most importantly, these tests should be run on a lower environment. Not prod, not anywhere users connect in, and it should be in an environment sandboxed away from others so that you minimize any potential interactions.

In Conclusion

Infrastructure as code. It’s ace, isn’t it? You just declare what you need, run your script and it’s all there waiting.

And if you test it thoroughly enough, then that should be most of the time.

The things to keep in mind are:

• Make sure you test what you need. But only test what you need to.

• Bear in mind that for integration tests you can use more generic tools and feed them the outputs from Terraform.

• Always test in a lower, non-live environment.

• Run your tests in a CI pipeline for repeatability.

Hopefully, this has helped. I appreciate it’s high level and if people want me to expand on sections I can