Your Resource for All Things Apps, Ops, and Infrastructure

HashiCorp + Puppet: Better Together

Now that HashiConf 2018 has wrapped up, there’s a ton of buzz in the industry about HashiCorp’s suite of tools: Terraform, Vault, Consul, Nomad, Packer, and Vagrant. We also heard many exciting announcements at Puppetize Live from Puppet about Puppet Enterprise, Discovery, Continuous Delivery, Insights, etc. I have also heard some discussions about the need for Puppet and configuration management when using HashiCorp tools, so I thought I’d discuss how well these two sets of tools actually complement—and not replace—one other.

Terraform was created over three years ago to help provision and manage infrastructure, and has seen many improvements during that time. Terraform is an open source tool at the core, but also has an Enterprise version which provides a GUI and many enhancements for corporate users, like Workspaces, HA, and replicated state files. While Terraform is great at being able to provision infrastructure on many clouds and platforms, it doesn’t do as well with the internal configuration of a VM like Puppet Enterprise does. They not only have a very similar configuration language, but also generate dependency graphs, as well as manage the desired state in a similar method. Puppet can configure and manage just about anything on a Linux/Windows VM after it has been provisioned by Terraform Enterprise and it can help ensure that configuration does not drift. Here’s a HashiCorp articlethat describes their view on this, as well. There is a bit of overlap in this area, with the new AWS and Azure ARM modules by Puppet, but they are limited to just those two platforms.

HashiCorp’s Consul has the ability to manage configuration files and push out changes from its key/value store in nearly real-time using consul-template. This is a great alternative for apps needing near real-time change management, especially for dynamic/cluster and container-based systems—even as the size of the infrastructure goes the convergence time stays small. Consul can also be used as back-end for Puppet to store key/value pairs via this forge module. Consul can be fully managed by Puppet with this approved module. Consul can still be used in conjunction with Puppet’s built-in Hiera, but it provides a way for key/values to easily be managed by external systems via API, replicated via a Consul cluster in real-time, and they can also be controlled via ACL’s and Sentinel Policies built into the core of Consul. Puppet has the added functionality to manage services, packages, databases, full application stack deployment, VM infrastructure, and much more. Consul’s real-time, multi-service architecture provides a very fault-tolerant and fast architecture for not only key-value storage, but also service discovery, service health checking, and replication.

Puppet natively does not have any built-in secrets management, however, it can use hiera-eyaml for basic encryption of secrets in a yaml file. While the encryption itself is good, the implementation lacks any kind of RBAC or managed access to update/maintain (outside of what the Git service has), automatic rolling of passwords, etc. This is a great solution for a small team or those just getting started with Puppet, but it doesn’t scale to multiple teams with real enterprise secrets management. This is where HashiCorp’s Vault comes into play. Vault provides many Enterprise features such as tightly controlled access to tokens and secrets via RBAC, auditing, unified API, encryption-as-a-service and automated secret rotation. If you couple this with Puppet’s new Deferred function for looking up secrets, each host can look up the secrets only it has access to, at the time that it needs them—rather than having your Puppet Master have access to all the secrets of your environment. HashiCorp has written a fantastic article on this topic, too.

HashiCorp’s Packer has been a great tool for generating immutable images. One of the most common ways to create that image is to have Terraform spin up the infrastructure and the VM, then have Puppet update the content inside the VM (based on the role of that new image), and finally have Packer generate the image that can be distributed to many different clouds and platforms. Another common workflow I’ve seen is to have Packer build a ‘standard’ image with the base OS (which allows you to easily have the latest patches/updates at all times), then have Terraform provision that image and install the Puppet agent with the proper role. There are pros and cons to both methods, depending on what software you are loading, requirements for setup time, or how many images you want to manage.

For a long time, Vagrant has been used by Puppet developers to spin up a local system to be able to quickly test new Puppet code. This allows you to rapidly validate your changes in a locally-controlled environment before rolling them out to an organization or submitting to a longer CI/CD pipeline for full linting, testing, and integration.

Both HashiCorp and Puppet provide first-rate tools to help you manage the state and security of your entire environment—from the writing and testing of code to the provisioning of production infrastructure—all while securing everything along the way. These Enterprise-ready tools complement each other very well in many ways and help to ensure a stable, secure, and adaptable infrastructure to support your business and security needs.

This is the first blog in a series where I’ll start to dig deeper into each of the HashiCorp tools and what those integrations with Puppet Enterprise look like using working examples.

(A huge thanks to Matt Stone of Puppet and Brian Green of HashiCorp for their contributions to this post.)

Subscribe to the AHEAD i/o Newsletter