Note: The description in this article is based on Terraform version 1.14.4, VMware Cloud Director Provider version 3.14, and VMware Cloud Director 10.6.
Terraform options for managing multiple similar objects
Introduction
Documentation and articles online generally show basic examples where we create one or two objects of the same type using separate resource blocks. I haven't found much official information on how to handle practical situations where we have hundreds of objects that we gradually add, remove, or modify. Useful advice can be found in various discussions where people ask about exactly this problem.
Those discussions also address how many VMs to maintain within a single configuration (project/workspace). If there are thousands, the configuration will be large, the state file very large, and every operation will take time. Different things call for different solutions. One extreme is a separate configuration for each VM. But a reasonable approach may be to split the configuration into groups in some way.
One resource block for similar objects
We probably don't want a separate resource block for every VM in our configuration, where much of the information repeats. It is better to have separate variables where we only set unique values for each VM. By default, a resource block configures one real object, but we have the option to use the meta-argument for_each or count.
Data types
Terraform supports many different data types. Primitive types hold a single data value, such as string, number, or bool. Collections hold multiple values of the same type; we can use list, map, and set. The object type object is not a collection, but defines a specific structure (mapping attribute names to value types).
List
An ordered collection of elements of the same type. Elements in a list are identified by consecutive integers starting from zero.
variable "list" {
type = list(string)
}
list = ["value1", "value2"]
var.list[0]
Map
A mapping of arbitrary key names to values of the same type. Any number of items. Values are identified by key name.
variable "map" {
type = map(string)
}
map = {
key1 = "value1"
key2 = "value2"
}
val.map.key1
Object
A mapping of explicit key names to values of defined (different) types. We therefore have a precisely defined object structure (schema). Values are identified by the defined key name.
variable "object" {
type = object({
key1 = number
key2 = string
})
}
object = {
key1 = 1
key2 = "value2"
}
val.object.key1
How to configure multiple VMs
Data types for VMs
We can define a variable that will contain specific data for VM creation. A suitable type is object. An example object with certain attributes:
variable "vm" {
type = object({
vm_name = string,
vm_ip = string,
vm_ram = number
})
}
Because we want to define/create multiple VMs, we can use list or map with our described object inside.
variable "vm_list" {
type = list(object({
vm_name = string,
vm_ip = string,
vm_ram = number
})
)
}
Using for_each in a resource
If we want to use a single resource block to create and manage multiple similar objects, we can use the meta-argument for_each (or count). The input for for_each is a collection (map or set of strings). It creates an instance for each item. The data structure is iterated repeatedly and resources are configured for each item in the structure.
Many examples use a variable of type list (as shown above). Since a list cannot be used as input for for_each, it needs to be converted. The for expression is used for this purpose.
for_each = { for vm in var.vm_list : vm.vm_name => vm }
For iterates through the list of VMs and for each VM object creates a mapping where the key is the name and the value is the entire object. The result is an object (due to the curly braces { }), which is automatically converted to a map for use in for_each.
Within the block where we use for_each, we can reference instance values using each.key and each.value. In our case, the key is the VM name and individual object attributes can be accessed using each.value.vm_ip.
Example configuration of multiple VMs
In the variables.tf file we add the definition of new variables.
# VM parameters unique
variable "vm_list" {
type = list(object({
vm_name = string,
vm_ip = string
})
)
}
In the terraform.tfvars file we set their values.
# VM list, unique parameters
vm_list = [
{
vm_name = "Demo-VM"
vm_ip = "172.30.21.100"
}, {
vm_name = "Next-VM"
vm_ip = "172.30.21.101"
}
]
In the vcd_vapp_vm resource we use for_each to create each VM from the list.
# create VMs from template wtih manual IP assignment
resource "vcd_vapp_vm" "VMs" {
for_each = { for vm in var.vm_list : vm.vm_name => vm }
vapp_name = var.vapp_name
name = each.value.vm_name
computer_name = each.value.vm_name
vapp_template_id = data.vcd_catalog_vapp_template.my-vapp-template.id
network {
type = "org"
name = var.org_net
ip_allocation_mode = "MANUAL"
ip = each.value.vm_ip
is_primary = true
}
}
VM variable with map instead of list
Many examples use the list type and then additionally use for when passing it to for_each. It appears to be functionally equivalent to use map directly, which then removes the need for for. I'm not sure whether there could be an issue if we wanted to rename a VM. Below is an example where the VM name is used as the key, making the vm_name attribute redundant (though it is still included in the example).
We will also use a predefined value for the memory size attribute using optional. If no value is specified in the variable, the default value of 4096 is used.
variable "vm_list" {
type = map(object({
vm_name = string,
vm_ip = string,
vm_ram = optional(number, 4096)
})
)
}
vm_list = {
"Demo-VM" = {
vm_name = "Demo-VM"
vm_ip = "172.30.21.100"
}
"Next-VM" = {
vm_name = "Next-VM"
vm_ip = "172.30.21.101"
}
}
resource "vcd_vapp_vm" "VMs" {
for_each = var.vm_list
vapp_name = var.vapp_name
name = each.key
computer_name = each.key
memory = each.value.vm_ram
vapp_template_id = data.vcd_catalog_vapp_template.my-vapp-template.id
network {
type = "org"
name = var.org_net
ip_allocation_mode = "MANUAL"
ip = each.value.vm_ip
is_primary = true
}
}
Complete example of creating multiple VMs
This is just an example showing the skeleton of the solution. It is a modification of the configuration described in previous articles. There we also created a vApp; here we only create two VMs. The contents of the individual files are listed below.
terraform.tf
# Version and providers requirement terraform { required_version = ">= 1.5.0" required_providers { vcd = { source = "vmware/vcd" version = "~> 3.14" } } } # Configure the VMware Cloud Director Provider provider "vcd" { user = var.vcd_user password = var.vcd_pass auth_type = "integrated" org = var.vcd_org vdc = var.vcd_vdc url = var.vcd_url max_retry_timeout = var.vcd_max_retry_timeout allow_unverified_ssl = var.vcd_allow_unverified_ssl logging = var.vcd_log logging_file = var.vcd_log_file }
variables.tf
# Logging variable "vcd_log" { description = "Enable Logging" type = bool } variable "vcd_log_file" { description = "Logging File" type = string } # VMware Cloud Director variables variable "vcd_user" { description = "VCD username" type = string } variable "vcd_pass" { description = "VCD user password" type = string sensitive = true } variable "vcd_org" { description = "VMware Cloud Director Organization" type = string } variable "vcd_vdc" { description = "Virtual Data Center" type = string } variable "vcd_url" { description = "VCD API URL" type = string } variable "vcd_max_retry_timeout" { type = number } variable "vcd_allow_unverified_ssl" { description = "Allow unverified SSL" type = bool } variable "vcd_catalog_name" { description = "VCD Catalog" type = string } variable "vcd_template_name" { description = "Catalog Template" type = string } # VM parameters common variable "vapp_name" { description = "vApp name for VM" type = string } variable "org_net" { description = "vApp Network name" type = string } variable "vm_ip_mask" { description = "vApp Network Subnet Mask" type = string } variable "vm_gw" { description = "vApp Network Gateway" type = string } variable "vm_dns" { description = "VM DNS server" type = string } # VM parameters unique variable "vm_list" { type = map(object({ ip = string ram = optional(number, 4096) cpus = optional(number, 2) cpu_cores = optional(number, 2) customization = bool }) ) }
terraform.tfvars
# Logging vcd_log = false vcd_log_file = "vcd-debug.log" # VMware Cloud Director Connection Variables vcd_org = "Firma" vcd_vdc = "Firma vDC" vcd_url = "https://server.domain.com/api" vcd_user = "test_user" vcd_pass = "xxx" vcd_max_retry_timeout = "240" vcd_allow_unverified_ssl = false # Catalog vcd_catalog_name = "Firma_catalog2" vcd_template_name = "template-Alma"
vm.auto.tfvars
We will put the configuration (assigning values to variables) of individual VMs into a separate file.
# VM parameters common vapp_name = "Demo-vApp" org_net = "Demo_net" vm_ip_mask = "22" vm_gw = "172.30.20.1" vm_dns = "172.30.20.1" # VM list, unique parameters vm_list = { "Demo-VM" = { ip = "172.30.21.100" ram = 2048 customization = false } "Next-VM" = { ip = "172.30.21.101" customization = true } }
main.tf
# read Catalog data "vcd_catalog" "my-catalog" { name = var.vcd_catalog_name } # read Catalog Template data "vcd_catalog_vapp_template" "my-vapp-template" { catalog_id = data.vcd_catalog.my-catalog.id name = var.vcd_template_name } # create VMs from template resource "vcd_vapp_vm" "VMs" { for_each = var.vm_list vapp_name = var.vapp_name name = each.key computer_name = each.key memory = each.value.ram cpus = each.value.cpus cpu_cores = each.value.cpu_cores vapp_template_id = data.vcd_catalog_vapp_template.my-vapp-template.id network { type = "org" name = var.org_net ip_allocation_mode = "MANUAL" ip = each.value.ip is_primary = true } customization { enabled = each.value.customization initscript = templatefile("initscript.tftpl", { vm_name = each.key, vm_ip_mask = "${each.value.ip}/${var.vm_ip_mask}", vm_gw = var.vm_gw, vm_dns = var.vm_dns }) } }
Provisioning VMs
Using the Terraform CLI we perform initialization, display the planned changes, and apply the changes to create the new VMs.
terraform init terraform plan terraform apply
If we need to create additional VMs, we simply add their configuration to the vm_list variable in the vm.auto.tfvars file. Then we run terraform apply again.
Guest OS Customization
In the previous part we described running a script using Guest OS Customization. We had to do it in two steps. The first step created the VM and configured the OS (ran the script). Then the settings had to be adjusted and customization disabled.
Here we added a customization attribute to the VM object in the vm.auto.tfvars configuration, which determines whether customization is enabled or not. So when creating a VM, we set it to true. For the second step, it is enough to change this value to false and run terraform apply again.
Conclusion
The described example is basic; if we want to manage a complex environment, more advanced methods will likely be needed. Modules and locals are commonly used. The configuration of individual VMs can be stored in separate files (e.g. YAML), loaded from a specific folder. We can have a default VM configuration that is merged with the configuration for a specific VM (overriding the shared common values). Another option is to keep the configuration in a CSV file.
There are no comments yet.