Proxmox - Setup Debian Cloud-Init Templates¶
This post documents the steps needed to create a minified debian cloud image template, which you can easily clone, and have a running VM in seconds with no configuration needed.
This includes common packages, ssh-keys, IP information.
The end result of this project is...
- You clone a template, and start it.
- The cloned VM automatically starts up, updates itself, and starts running, with common packages installed, and your ssh details pre-configured.
- A slimmed down distribution, without excess (My end result was a 400M image.)
- No need to edit the VM after cloning, other then adding CPU/RAM, or other hardware as needed.
Verbiage / Documentation¶
Cloud-Init is a technology which allows you to clone a template, with customized user-settings. This is how you can spin up VMs in AWS in seconds.
These, are special images provided, which includes support for cloud-init. For this example, we will specifically be leveraging the "genericcloud" image, which is suitable for our VMs.
This- image does not include many bare metal hardware drivers, which aren't needed, since the focus of this post is creating a VM template.
Proxmox Templates / Clones Documentation is here, if you wish to learn more.
Getting Started¶
Step 1. Acquire Cloud Image¶
From Debian Official Cloud Images, we want to select the Bookworm/Latest folder at the bottom of the page.
Inside of this folder, is quite a few different files.
We SPECIFICALLY want, the "genericcloud"-"amd64".qcow2 image. Its name is "debian-12-genericcloud-amd64.qcow2"
Warning
While- the above link DOES link to the image- I don't recommend blindly copying and pasting links from the internet.
We will now want to store this image on one of our proxmox nodes, in a temporary location.
root@kube01:~/template# wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-amd64.qcow2
...
2024-07-02 09:41:06 (15.7 MB/s) - ‘debian-12-genericcloud-amd64.qcow2’ saved [343378944/343378944]
root@kube01:~/template# ls -al
total 335344
drwxr-xr-x 2 root root 4096 Jul 2 09:40 .
drwx------ 10 root root 4096 Jul 2 09:40 ..
-rw-r--r-- 1 root root 343378944 Jul 1 13:27 debian-12-genericcloud-amd64.qcow2
Keep this terminal open, we will come back to it in a bit.
Step 2. Customize Image¶
At this point, we have a cloud init image. However, I personally like to customize my images, to install a few basic tools, as well as the qemu-guest-agent.
Install packages into the base image¶
You will need to install the libguestfs-tools package, via your package manager. If you don't wish to install the tools on your proxmox machine, feel free to customize the image on your desktop, a LXC, VM, etc.
root@kube01:~/template# sudo apt-get install libguestfs-tools
Next- we can leverage virt-customize to customize our image.
I personally, install these utilities on most of my machines, so, I will include them into this template.
- curl
- wget
- nano
- rsync
- htop
At an absolute minimum though, we need to add the qemu-guest-agent
Place the packages names you would like, into the virt-customize command.
root@kube01:~/template# virt-customize -a debian-12-genericcloud-amd64.qcow2 --install qemu-guest-agent,curl,wget,nano,rsync,htop
[ 0.0] Examining the guest ...
[ 8.8] Setting a random seed
virt-customize: warning: random seed could not be set for this type of
guest
[ 8.9] Setting the machine ID in /etc/machine-id
[ 8.9] Installing packages: qemu-guest-agent curl wget nano rsync htop
[ 18.4] Finishing off
Info
Please read the Documentation for virt-customize. There is far more functionality then just adding packages.
I personally, rely on ansible to do the final provisioning, as such, I am only installing the bare minimum software.
(Optional) Expand the image size¶
Info
This step is optional- because you can just grow the partition in Proxmox, and either start or reboot the cloned VM.
It will automatically grow the root partition.
Next up, I am going to set the size of this image, to 16G, as its original size is 2GB.
You can view the current size of the image, using qemu-img info
root@kube01:~/template# qemu-img info debian-12-genericcloud-amd64.qcow2
image: debian-12-genericcloud-amd64.qcow2
file format: qcow2
virtual size: 2 GiB (2147483648 bytes)
disk size: 607 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
Child node '/file':
filename: debian-12-genericcloud-amd64.qcow2
protocol type: file
file length: 607 MiB (636682240 bytes)
disk size: 607 MiB
To resize the disk, use qemu-img resize
root@kube01:~/template# qemu-img resize debian-12-genericcloud-amd64.qcow2 16G
Image resized.
(Optional) Compress and shrink the image¶
We can also use qemu-img to compress, and shrink the final image.
root@kube01:~/template# qemu-img convert -O qcow2 -c -o preallocation=off debian-12-genericcloud-amd64.qcow2 debian-12-genericcloud-amd64-shrink.qcow2
root@kube01:~/template# qemu-img info debian-12-genericcloud-amd64-shrink.qcow2
image: debian-12-genericcloud-amd64-shrink.qcow2
file format: qcow2
virtual size: 16 GiB (17179869184 bytes)
disk size: 421 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
Child node '/file':
filename: debian-12-genericcloud-amd64-shrink.qcow2
protocol type: file
file length: 422 MiB (442643456 bytes)
disk size: 421 MiB
This- saved around 200M, which is pretty impressive when you consider the file was only 600M originally.
Commands used¶
Full commands used:
apt-get install libguestfs-tools
virt-customize -a debian-12-genericcloud-amd64.qcow2 --install qemu-guest-agent,curl,wget,nano,rsync,htop
qemu-img resize debian-12-genericcloud-amd64.qcow2 16G
qemu-img convert -O qcow2 -c -o preallocation=off debian-12-genericcloud-amd64.qcow2 debian-12-genericcloud-amd64-shrink.qcow2
At this point, we should have a suitable base image.
Step 3. Create a VM¶
Now that we have a suitable image, we need to build a base VM template. I will be doing this step through the GUI.
For the general tab, give a name which implies this is a template.
I also have a template source pool, and tag I assigned. But, this is optional.
Finally- I set a shutdown timeout of 5 minutes. This- is also optional.
For the OS Tab- click "don't use any media". No other changes are needed here.
On the system tab, I enabled the checkbox for Qemu-Agent.
On the disks tab, you will want to REMOVE the default disks. We will add
For CPU / Memory tab- just set whatever defaults you want.
I chose 2 CPU cores, x86-64-v2-AES on the type as its easily compatible with all of my nodes.
For memory, I allocated 1024G of ram, and disabled ballooning. Remember- we will clone this template for creating VMs, and adjust the CPU/RAM there.
For network, I chose my default vlan interface, where I typically create new nodes. I did select firewall.
On the confirm tab- don't select start after created.
Step 4. Attach qcow2 image to VM¶
Next, we will attach the .qcow2 image we downloaded earlier.
Skip back into the terminal / ssh session from the first step.
Grab the ID from the VM we just created. Alternatively, you can find it via the CLI of the host you created it on.
root@kube01:~/template# qm list
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
139 DebianCloudInitTemplate stopped 1024 0.00 0
Next, attach it, using qm importdisk $VM_ID $IMAGEPATH $YOUR_STORAGE
I want to store my template on my ceph-cluster, which is named ceph-block
root@kube01:~/template# qm importdisk 139 debian-12-genericcloud-amd64.qcow2 ceph-block
importing disk 'debian-12-genericcloud-amd64.qcow2' to VM 139 ...
transferred 0.0 B of 16.0 GiB (0.00%)
transferred 20.7 MiB of 16.0 GiB (1.01%)
...
transferred 2.0 GiB of 16.0 GiB (99.60%)
transferred 2.0 GiB of 16.0 GiB (100.00%)
Successfully imported disk as 'unused0:ceph-block:vm-139-disk-0'
root@kube01:~/template#
Step 6. Template Tweaks¶
Lets go back to the VM in the GUI now.
Hardware Tab¶
Remove the CD/DVD drive (Unless- you need one.)
Attach the disk we created from the qcow2.
- Click on the unused disk, then click edit. Set Bus/Device to "VirtIO Block" as position 0
- I also like to set write back cache. However, this is optional.
- Click Add.
Next- Click Add -> CloudInit Device
Select your storage. I leave IDE selected as the default bus.
Click Add.
If, you plan on using UEFI, you can also add an EFI disk at this step.
Cloud Init Tab¶
Now- we can customize the cloud init tab, with a default user, password (if desired), dns,ip, and ssh information.
Since, I use ansible to provision, and prepare all of my machines- My cloudinit is specific to allow it to log in.
Once it runs, it will automatically remove the temporary accounts, configure SSH, NTP, DNS, static IP options, etc.
But- the key here is, for its initial login, it can log into VMs cloned from this template, using the username "temp" with ansible's public key.
For the IP Config, I set dhcp as the default.
Options Tab¶
Under options, there are a few things we will need to customize.
First up- Boot order.
I always uncheck network boot, since I don't use it. As well, make sure to unselect the cloudinit drive.
We only want the primary disk, containing our cloned qcow2 selected here.
Final touches?¶
I personally, like customizing a default set of sane firewall rules for my VMs. However- this is completely optional, and up to you.
As a good example, I have a firewall rule defined at the datacenter level, for only allowing ssh access, for known, good hosts.
Here- is my personal starting point for firewall rules.
If you do plan on including firewall rules, make sure to enable "Firewall" under Firewall -> Options. It is not enabled by default.
Step 6. Convert to template¶
This step is questionably optional, because you don't need to convert this to a template, in order to clone it.
Info
If you wish to use a linked-clone, you DO need to convert your VM to a template.
Warning
Without converting to a template, you will need to change the MAC address for every clone.
(aka, you should make this a template, if you plan on cloning it)
To convert this to a template- Right click on the VM, and select, Convert to template.
After you have converted it to a template, you will no longer be able to start/stop or snapshot this machine. You would need to clone it first, and then start the clone.
One final configuration item- Edit the network interface on your template. Set its MAC Address to 00:00:00:00:00:00
Warning
If you miss this step- all of your cloned VMs will have the same MAC address.
This, will likely cause a lot of problems. Make sure to do this step!
The final result¶
Clone¶
Now- just clone the template, and voila- you will have a patched up debian machine ready for your workloads.
Right click the template (or VM, if you didn't convert to a template), and select Clone.
Select a target node and a new name for your new VM.
Next- the Mode dialog has a few options.
Proxmox - Documentation for cloning
The TLDR;
IF you use a linked clone, your new VM only needs to store the "changes" from the base-template's image. These- can only be done to templates.
A full clone, is exactly that. Its a full clone. You can full clone VMs or templates.
Info
If you don't have cluster-wide, reliable, shared storage for all of your nodes, I recommend using a full clone.
The linked clone, MUST remain on the same storage as the original template.
The cloned VMs REQUIRES the image from the template.
If- you try to delete the image used by a linked clone, you will get this:
For this- I will be using a linked clone, as I leverage my Ceph Cluster for cluster-wide storage.
After you have performed your changes, click Clone.
Execute¶
Once you start your VM, it will automatically start the process of downloading updates, and configuring itself.
And- lastly, you will be left, with a login prompt.
If- you followed through with the step to update the base-image, to add qemu-guest-agent, you will also be able to see IP details via the proxmox web interface.
Lastly- since we specified the username, and ssh-keys via the CloudInit, we can also log into this VM.
root@Ansible ~/Ansible# ssh [email protected]
Linux my-new-vm 6.1.0-22-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1 (2024-06-21) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
And- all of our additional packages are installed.
temp@my-new-vm:~$ which htop
/usr/bin/htop
temp@my-new-vm:~$ which nano
/usr/bin/nano
temp@my-new-vm:~$ which curl
/usr/bin/curl
Other Notes¶
Ceph- Editing disks after converting to template¶
If you edit the disks after converting them to a template. The new disks will not be properly flagged as "base".
To fix this, convert your template back into a VM.
nano /etc/pve/qemu-server/139.conf
#Set template=1, to template=0
Then, once your GUI reflects the changes, convert the VM back into a template again.
Patching linked clones through template¶
If- you are thinking of creating a template, spinning up linked clones, and patching the base template- this method isn't ideal for doing that.
Ceph - Space Usage of Linked Clone vs Full Clone¶
Looking at ceph- my base image is only using 1.3G of total disk space.
If- we look at a VM cloned using Full Clone- we can see, it is using 1.4G of disk space. The parent field is also empty (as it was a full copy)
Looking at a disk from a Linked clone, we can see, it is also using 1.3/1.4G of disk space, but, you can see the "layering" feature is enabled, and it has a parent specified.
Now- you might be saying- That doesn't sound right at all.
And, I would agree. I would assume it is working as intended. But- I am out of time to determine how to find the actual child's usage over the parent.
# Get base image children.
root@kube01:~# rbd children ceph-block/base-139-disk-0
ceph-block/vm-141-disk-0
# Get base image details
root@kube01:~# rbd info ceph-block/base-139-disk-0
rbd image 'base-139-disk-0':
size 16 GiB in 4096 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: 773ac59ea6a2db
block_name_prefix: rbd_data.773ac59ea6a2db
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 2 16:56:02 2024
access_timestamp: Tue Jul 2 17:00:41 2024
modify_timestamp: Tue Jul 2 16:56:02 2024
# Get image details for the specific snapshot the child is using
root@kube01:~# rbd info ceph-block/base-139-disk-0@__base__
rbd image 'base-139-disk-0':
size 16 GiB in 4096 objects
order 22 (4 MiB objects)
snapshot_count: 1
id: 773ac59ea6a2db
block_name_prefix: rbd_data.773ac59ea6a2db
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 2 16:56:02 2024
access_timestamp: Tue Jul 2 17:00:41 2024
modify_timestamp: Tue Jul 2 16:56:02 2024
protected: True
# Get child image details.
root@kube01:~# rbd info ceph-block/vm-141-disk-0
rbd image 'vm-141-disk-0':
size 16 GiB in 4096 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 7744398791b79f
block_name_prefix: rbd_data.7744398791b79f
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Tue Jul 2 17:12:12 2024
access_timestamp: Tue Jul 2 17:12:12 2024
modify_timestamp: Tue Jul 2 17:12:12 2024
parent: ceph-block/base-139-disk-0@__base__
overlap: 16 GiB
How to add disk-space after cloning?¶
Easy- Add the disk space via proxmox.
Then- you can take one of two paths.
Reboot the system¶
The debian cloud templates will automatically resize during boot.
Just- add disk space, and reboot.
[ 6.697063] EXT4-fs (vda1): resizing filesystem from 4161531 to 20938747 blocks
[ 6.813343] EXT4-fs (vda1): resized filesystem to 20938747
Using growpart- no reboot needed.¶
SSH into the VM, and run growpart
temp@test:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sr0 11:0 1 4M 0 rom
vda 254:0 0 32G 0 disk
|-vda1 254:1 0 15.9G 0 part /
|-vda14 254:14 0 3M 0 part
`-vda15 254:15 0 124M 0 part /boot/efi
root@test:/home/temp# growpart /dev/vda 1
CHANGED: partition=1 start=262144 old: size=33292255 end=33554398 new: size=66846687 end=67108830
root@test:/home/temp# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sr0 11:0 1 4M 0 rom
vda 254:0 0 32G 0 disk
|-vda1 254:1 0 31.9G 0 part /
|-vda14 254:14 0 3M 0 part
`-vda15 254:15 0 124M 0 part /boot/efi
Help, all of my machines are getting the same IP address from DHCP.¶
Edit the hardware of your template. Set the Network Device's MAC address to 00:00:00:00:00:00
New machines will receive a random mac address.
(If- you missed the step above, and you have already cloned a dozen machines, you can "clear" the mac address field of the VMs NIC(s), and a new address will be automatically configured.)