Thursday, December 23, 2021

Terraform for dummies part 5: Terraform deployment On-premises (KVM)

This image has an empty alt attribute; its file name is image.pngIntro

For a long time, Terraform was associated with deploying resources in the cloud. But what many people don’t know is that terraform already had private and community based providers that worked perfectly on non cloud environments. Today, we will discover how to deploy a compute vm in a KVM host. Not only that, but we will also do it on top of VirtualBox in a nested virtualization environment. As always, I will provide the vagrant build to allow you to launch the lab for a front-row experience. It is indeed the cheapest way to use terraform on-prem on your laptop.

Terraform on-prem

Libvirtd provider is a community based project built by Duncan Mac-Vicar. There is no difference between using terraform on cloud platforms and doing it with Libvirtd provider. In this lab I had to enable nested virtualization in my VBox to make it easier to run the demo.The resulting hypervision is qemu-kvm, a non bare-metal KVM environment also known as type 2 hypervisor (virtual hardware emulation).     

How to get started

No need to subscribe to a cloud Free-tier using credit cards to play with terraform. You can start this lab right now on your laptop with my vagrant build. The environment comes with all necessary modules & packages to deploy vms using terraform.

Lab Content:
- KCLI (wrapper tool for managing vms)
- Terraform 1.0
- Libvirt terraform provider
- Terraform configuration samples to get started ( ,

GitHub repo

  • Clone the repo
  • C:\Users\brokedba> git clone
    C:\Users\brokedba> cd KVM-on-virtualbox

  • Start the vm (make sure you have 2Cores and 4GB RAM to spare before the launch)
  • C:\Users\*\KVM-on-virtualbox> vagrant up
    C:\Users\*\KVM-on-virtualbox> vagrant ssh ---- access to KVM host

    Now you have a new virtual machine shipped with kvm and terraform which will help us complete the lab.
    Note: Terraform files will be located under   /root/projects/terraform/

What you should know 

  • Libvirt provider in Terraform registry 

    Up until terraform version 0.12, Hashicorp didn’t officially recognize this libvirt provider, you could still run config files if the plugin was in a local plugin folder (i.e. /root/.terraform.d/plugins/)
    But after version 0.13,  terraform enforced Explicit Provider Source Locations. As result, you’ll need few tweaks to make it run in terraform. Everything is documented in GitHub issue1 & 2 but I’ll summarize it below.   

    The steps to run libvirt provider in terraform v1.0 (Already done in my build)

    - Download the Binary (current vers: 0.6.12). For my part I used an older version for fedora (0.6.2)

    [root@localhost]# wget URL
    [root@localhost]# tar xvf terraform-provider-libvirt-**.tar.gz

    - Add the plugin in a local registry

    [root@localhost]# mkdir –p ~/.local/share/terraform/plugins/
    [root@localhost]# mv terraform-provider-libvirt ~/.local/share/terraform/plugins/

    - Add the below code block to the file to map libvirt references with the actual provider

    [root@localhost]# vi
    terraform { required_version = ">= 0.13" required_providers { libvirt = { source = "dmacvicar/libvirt" version = "0.6.2" } } }
    ... REST of the Config

    - Initialize and validate by running terraform init which will detect and add libvirt plugin in the local registry  

    [root@localhost]# terraform init

     Initializing the backend...
     Initializing provider plugins...
    - Finding dmacvicar/libvirt versions matching "0.6.2"...
    - Installing dmacvicar/libvirt v0.6.2...
    - Installed dmacvicar/libvirt v0.6.2 (unauthenticated)


Terraform deployment  

  • Deploy basic ubuntu vm 

    Let’s first provision a simple ubuntu vm on our KVM environment . Again in a nested virtualization mode we are using hardware emulated hypervision “Qemu”, and this will require a small hack by setting a special variable. Will
    explain why further down.  Just bear with me for now.

[root@localhost]# export TERRAFORM_LIBVIRT_TEST_DOMAIN_TYPE="qemu"

  • Next  we need to find our configuration file, let’s check declared resource behind that
  • [root@/*/ubuntu/]# ls /root/projects/terraform/ubuntu/
    .. --- you can click to download or read content

    [root@/*/ubuntu/]# vi

    provider "libvirt" {
    uri = "qemu:///system"}
    terraform {
      required_providers {
        libvirt = {
          source  = "dmacvicar/libvirt"
          version = "0.6.2"
    } ## 1. --------> Section that declares the provider in Terraform registry

    # 2. ----> We fetch the smallest ubuntu image from the cloud image repo
    resource "libvirt_volume" "ubuntu-disk" {
    name   = "ubuntu-qcow2"
    pool   = "default" ## ---> This should be same as your disk pool name
    source =
    format = "qcow2"

    # 3. -----> Create the compute vm
    resource "libvirt_domain" "ubuntu-vm" {
    name   = "ubuntu-vm"
    memory = "512"
    vcpu   = 1

    network_interface {
       network_name = "
    default" ## ---> This should be the same as your network name

    console { # ----> define a console for the domain.
       type        = "pty"
       target_port = "0"
       target_type = "serial" }

    disk {   volume_id = } # ----> map/attach the disk
    graphics { ## ---> graphics settings
       type        = "spice"
       listen_type = "address"
       autoport    = "true"}

  • Run terraform init to initialize the setup and fetch the called providers in the tf file, like we did earlier.
  • [root@localhost]# terraform init

  • Run terraform plan
  • [root@localhost]# terraform plan
    Terraform will perform the following actions:

      # libvirt_domain.ubuntu-vm will be created
      + resource "libvirt_domain" "ubuntu-vm" {
          + arch        = (known after apply)
          + disk        = [
              + {
                  + block_device = null
                  + file         = null
                  + scsi         = null
                  + url          = null
                  + volume_id    = (known after apply)
                  + wwn          = null
          + emulator    = (known after apply)
          + fw_cfg_name = "opt/com.coreos/config"
          + id          = (known after apply)
          + machine     = (known after apply)
          + memory      = 512
          + name        = "ubuntu-vm"
          + qemu_agent  = false
          + running     = true
          + vcpu        = 1

          + console {
              + source_host    = ""
              + source_service = "0"
              + target_port    = "0"
              + target_type    = "serial"
              + type           = "pty"

          + graphics {
              + autoport       = true
              + listen_address = ""
              + listen_type    = "address"
              + type           = "spice"

          + network_interface {
              + addresses    = (known after apply)
              + hostname     = (known after apply)
              + mac          = (known after apply)
              + network_id   = (known after apply)
              + network_name = "default"

      # libvirt_volume.ubuntu-disk will be created
      + resource "libvirt_volume" "ubuntu-disk" {
          + format = "qcow2"
          + id     = (known after apply)
          + name   = "ubuntu-qcow2"
          + pool   = "default"
          + size   = (known after apply)

          + source =

    Plan: 2 to add, 0 to change, 0 to destroy.

  • Run terraform apply to deploy the vm which was declared in the plan command output 
  • [root@localhost]# terraform apply -auto-approve
    Plan: 2 to add, 0 to change, 0 to destroy.
    libvirt_volume.ubuntu-disk: Creating...
    libvirt_volume.ubuntu-disk: Creation complete after 17s [id=/u01/guest_images/ubuntu-qcow2]
    libvirt_domain.ubuntu-vm: Creating...
    libvirt_domain.ubuntu-vm: Creation complete after 0s [id=29735a37-ef91-4c26-b194-05887b1fb264]

    Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

  • Wait a little a bit and run kcli command or virsh list
  • [root@localhost ubuntu]# kcli list vm
    |    Name   | Status |      Ips       | Source | Plan | Profile |
    | ubuntu-vm |   up   | |        |      |         |

  • Cool, we have a vm with an IP address but you still need to login to it. Cloud images just don’t come with root passwords so let’s destroy it now and jump into our second example.
  • [root@localhost]# terraform destroy -auto-approve
    Destroy complete! Resources: 2 destroyed.


  • Deploy a vm with CloudInit 

Same way as with Cloud vms we can also call startup scripts to do anything we want during the bootstrap.
I chose Centos in this example where CloudInit bootstrap actions were:

- Set a new password to root user

- Add an SSH key to root user

- Change the hostname 

  • Create a CloudInit config file: please careful with the indentation. It can also be downloaded here cloud_init.cfg.

# cd ~/projects/terraform
[root@~/projects/terraform]# cat cloud_init.cfg
disable_root: 0
  - name: root
    ssh-authorized-keys: ### –> add a public SSH key
      - ${file("~/.ssh/")}
ssh_pwauth: True
chpasswd: ### –> change the password
  list: |
  expire: False

  - hostnamectl set-hostname terracentos

  • I will only display the part where CloudInit is involved but you can read the full content here

# cd ~/projects/terraform
[root@~/projects/terraform]# cat
provider "libvirt" {

resource "libvirt_volume" "centos7-qcow2" {

## 1. ----> Instantiate cloudinit as a media drive to add our startup tasks
resource "libvirt_cloudinit_disk" "commoninit" {
name           = "commoninit.iso"
pool           = "default" ## ---> This should be same as your disk pool name
user_data      = data.template_file.user_data.rendered
## 2. ----> Data source converting the cloudinit file into a userdata format
data "template_file" "user_data" { template = file("${path.module}/cloud_init.cfg")}

resource "libvirt_domain" "centovm" {

  name   = "centovm"
  memory = "1024"
  vcpu   = 1

cloudinit = ## 3. ----> map CloudInit

...---> Rest of the usual domain declaration

  • We can now run a terraform INIT then PLAN (don’t forget to set TERRAFORM_LIBVIRT_TEST_DOMAIN_TYPE variable)

    [root@~/projects/terraform]# terraform init

    [root@~/projects/terraform]# terraform plan
    ... Other resources declaration
    # libvirt_cloudinit_disk.commoninit will be created
    + resource "libvirt_cloudinit_disk" "commoninit" {
    + id        = (known after apply)
        + name      = "commoninit.iso"
        + pool      = "default"
        + user_data = <<-EOT
              disable_root: 0
                - name: root
                    - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQ** root@localhost.localdomain

              ssh_pwauth: True
                list: |
                expire: False

                - hostnamectl set-hostname terracentos

    ... Remaining declaration

  • Run the Apply

    [root@~/projects/terraform]# terraform apply -auto-approve
    Plan: 3 to add, 0 to change, 0 to destroy.
    libvirt_cloudinit_disk.commoninit: Creation complete after 1m22s [id=/u01/guest_images/commoninit.iso;61c50cfc-**]

    Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

  • Wait a little a bit after completion and run kcli command to confirm an IP is allocated.

    [root@~/projects/terraform]# kcli list vm
    |    Name   | Status |      Ips       | Source | Plan | Profile |
    | centovm |   up   | |        |      |         |

  • Login into the vm using ssh and password authentication

    -- 1. SSH
    ssh -i ~/.ssh/id_rsa root@

    Warning: Permanently added '' (RSA) to the list of known hosts.

    [root@terracentos ~]# cat /etc/centos-release
    CentOS Linux release 7.8.2003 (Core)

    -- 2. Password

    [root@~/projects/terraform]# virsh console centovm
    Connected to domain centovm Escape character is ^]
    CentOS Linux 7 (Core)
    Kernel 3.10.0-1127.el7.x86_64 on an x86_64

    terracentos login: root
    [root@terracentos ~]#

And here you go, your local terraform vm was changed during startup using a simple config file just like the ones on AWS ;) .

Undocumented QEMU tip on Terraform

    • I can now explain why we needed to set the environment variable to “qemu” in order to have your deployment working. In fact, the vm will never start-up without this trick. Let’s find why

      • Nested virtualization doesn't seem to support kvm domain type.
      • This issue is similar to what openstak and miniduke encounter when they use kvm within vbox "could not find capabilities for domaintype=kvm"
      • I needed to make libvirt provider chose qemu instead of kvm during the provisioning
      • With the help of @titogarrido we found out that the logic inside its code “domain_def.go
        implied kvm was the only supported virtualization but checked a mysterious variable first
      • Finally found the workaround by setting the variable to qemu which is an old hack from days when authors were testing travis 
    I asked them to replace that variable by an attribute inside terraform code but the bug is still there, see more in my issue

    --- Workaround for non BareMetal hosts (nested)

    --- Below Go check happens where qemu is selected (domain_def.go)

    if v := os.Getenv("TERRAFORM_LIBVIRT_TEST_DOMAIN_TYPE"); v != "" {
    		domainDef.Type = v
    	} else {domainDef.Type = "kvm"}


  • We have seen in this lab just how easy it was to deploy resources using terraform on-promises
         (no more credit card needed :) 
  • I am very happy to make this little contribution via my vagrant build shipped with KVM on VirtualBox
  • I hope you’ll give this lab a try as it’s super easy and fun !! :)
  • I would like to thank @titogarrido, who has accepted to dig deeper with me so we can find the bug and
  • I love pair programming tools like tmate that allowed us to live collaborate while he was in Brazil and me in Canada.  
  • If you want to know about my vagrant build check my previous blog post 
  • Learn more about libvirt provider usage in the official terraform registry for libvirt provider dmacvicar/libvirt
  • Thank you for reading !

    1 comment: