Proxmox Backup Server - No space left on device¶
I was recently greeted with backup failures, from what appeared to be a full disk.
102: 2026-01-22 06:00:08 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/AF/UPID:pbs:000002CC:000004AF:00000396:69721148:backup:Unraid\\x2dMain\\x3avm-102:root@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
As, it turns out, this is an inode issue and not a full mount point.
The Symptom¶
If you are receiving this error: failed: ENOSPC: No space left on device
but, there is no shortage of free space...
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 795M 820K 794M 1% /run
/dev/mapper/pbs-root 62G 20G 40G 34% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /tmp
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs 3.9G 196K 3.9G 1% /run/proxmox-backup
/dev/vdc1 8.0T 857G 7.2T 11% /mnt/datastore/Remote
/dev/vdb1 8.0T 980G 7.1T 12% /mnt/datastore/Local
tmpfs 795M 8.0K 795M 1% /run/user/0
tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
Then, you might have an issue with your inode table being full. You can run df -i to confirm.
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 1006498 527 1005971 1% /dev
tmpfs 1016470 811 1015659 1% /run
/dev/mapper/pbs-root 4096000 4070273 25727 100% /
tmpfs 1016470 1 1016469 1% /dev/shm
tmpfs 1048576 9 1048567 1% /tmp
tmpfs 1016470 5 1016465 1% /run/lock
tmpfs 1024 1 1023 1% /run/credentials/systemd-journald.service
tmpfs 0 0 0 - /run/proxmox-backup
/dev/vdc1 858993344 322821 858670523 1% /mnt/datastore/Remote
/dev/vdb1 858993344 751941 858241403 1% /mnt/datastore/Local
tmpfs 203294 30 203264 1% /run/user/0
tmpfs 1024 1 1023 1% /run/credentials/getty@tty1.service
The Cause¶
The root issue, is ext4 has a fixed inode table, and I formatted my root disk as ext4.
If, you used XFS, or other file system, you shouldn't run into this issue.
For that reason, I would recommend against formatting the root device for your PBS server... as ext4. You can avoid this issue completely.
While you can (and should) enable proxmox log retention...
This combined with manual purging of older logs did not correct my issue.
My Solution¶
I initially attempted to cleanup any task logs older then 90 days.
find /var/log/proxmox-backup -mtime +90 -delete
However, as you can see, I am still at 100% inode usage, with only 25k free inodes.
So, instead, I am going to create a file-backed device formatted with XFS to store my logs, which will ensure the task logs cannot exhaust the system inode table.
Here is the script I came up with
#!/usr/bin/env bash
# Creates a XFS loopback image and persistently mount for PBS task logs.
# Target mount: /var/log/proxmox-backup
## Where the image file should be stored
IMG="/mnt/datastore/Unraid-Main/pbs-tasklogs.img"
## What size should the image file be.
IMG_SIZE="20G"
## Temp loopback mount point.
LOOP_MNT="/mnt/pbs-tasklogs"
## The directory to which it will be mounted to.
TARGET_DIR="/var/log/proxmox-backup"
if [[ $EUID -ne 0 ]]; then
echo "ERROR: run as root" >&2
exit 1
fi
if [[ ! -d "$(dirname "$IMG")" ]]; then
echo "ERROR: parent directory does not exist: $(dirname "$IMG")" >&2
exit 1
fi
if [[ -e "$IMG" ]]; then
echo "ERROR: image already exists: $IMG" >&2
exit 1
fi
if mountpoint -q "$TARGET_DIR"; then
echo "ERROR: $TARGET_DIR is already a mountpoint; refusing to proceed." >&2
exit 1
fi
mkdir -p "$LOOP_MNT"
echo "Creating sparse image: $IMG ($IMG_SIZE)"
fallocate -l "$IMG_SIZE" "$IMG"
echo "Formatting as XFS"
mkfs.xfs -f "$IMG" >/dev/null
echo "Mounting loop image at $LOOP_MNT"
mount -o loop,noatime "$IMG" "$LOOP_MNT"
echo "Stopping proxmox backup server"
systemctl stop proxmox-backup-proxy proxmox-backup-api 2>/dev/null || true
echo "Migrate existing logs into loop mount (Warning- this may take a long time...)"
rsync -a --delete "${TARGET_DIR}/" "${LOOP_MNT}/" || true
echo "Freeing inodes. (This may take a long time.)"
find "$TARGET_DIR" -xdev -mindepth 1 -delete
mkdir -p "$TARGET_DIR"
mount --bind "$LOOP_MNT" "$TARGET_DIR"
echo "Updating /etc/fstab"
FSTAB_LINE1="${IMG} ${LOOP_MNT} xfs loop,noatime 0 0"
FSTAB_LINE2="${LOOP_MNT} ${TARGET_DIR} none bind 0 0"
grep -Fqx "$FSTAB_LINE1" /etc/fstab || echo "$FSTAB_LINE1" >> /etc/fstab
grep -Fqx "$FSTAB_LINE2" /etc/fstab || echo "$FSTAB_LINE2" >> /etc/fstab
echo "Reloading SystemD"
systemctl daemon-reload
echo "Starting proxmox backup server"
systemctl start proxmox-backup-api proxmox-backup-proxy 2>/dev/null || true
echo "----------------------------------------------------------------------------------------------------"
echo "--- df -h (target) ---"
df -h "$TARGET_DIR" || true
echo "--- df -i (target) ---"
df -i "$TARGET_DIR" || true
echo "--- mount | grep (pbs-tasklogs) ---"
mount | grep -E "(${LOOP_MNT}|${TARGET_DIR})" || true
echo "Done."
I am simply running into an inode issue. By provisioning a file-back block device, I am relocating the files/inodes into this file and off of the root device.
You can provision additional storage and mount if, if desired. However, I personally did not see too much benefit to doing so.
Disclaimers¶
I cannot guarantee the above script will work for you. I also cannot guarantee it will not randomly decide to nuke your entire lab.
Make sure you have backups / snapshots, in the event something catastrophically unexpected occurs.
I can tell you, I personally tested and executed the script on my installation, and if you are seeing this blog post, then it was successful.
Results¶
It took quite a while for the logs to relocate. Here is output captured after a few minutes of running.
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pbs-root 62G 20G 40G 34% /
/dev/loop0 20G 4.9G 16G 25% /mnt/pbs-tasklogs
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pbs-root 4096000 4070274 25726 100% /
/dev/loop0 10485760 1739611 8746149 17% /mnt/pbs-tasklogs
After oh... 10 minutes.... It finally completed
root@pbs:~# nano makeLogMount
root@pbs:~# chmod +x makeLogMount
root@pbs:~# ./makeLogMount
Creating sparse image: /mnt/datastore/Unraid-Main/pbs-tasklogs.img (20G)
Formatting as XFS
Mounting loop image at /mnt/pbs-tasklogs
Stopping proxmox backup server
Migrate existing logs into loop mount
Updating /etc/fstab
Reloading SystemD
Starting proxmox backup server
----------------------------------------------------------------------------------------------------
--- df -h (target) ---
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 20G 11G 9.8G 52% /var/log/proxmox-backup
--- df -i (target) ---
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 10485760 3908312 6577448 38% /var/log/proxmox-backup
--- mount | grep (pbs-tasklogs) ---
/mnt/datastore/Unraid-Main/pbs-tasklogs.img on /mnt/pbs-tasklogs type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/mnt/datastore/Unraid-Main/pbs-tasklogs.img on /var/log/proxmox-backup type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
Done.
root@pbs:~#
inode tables look much, much better now.. with all of the inodes from logs being inside of the file-backed block device.
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pbs-root 62G 12G 48G 20% /
/dev/loop0 20G 11G 9.9G 51% /mnt/pbs-tasklogs
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pbs-root 4096000 161965 3934035 4% /
/dev/loop0 10485760 3908312 6577448 38% /mnt/pbs-tasklogs
root@pbs:~#
Summary¶
This is the solution I used, and while I cannot promise it is the best solution, It resolved my issues.
There are many ways to resolve this issue. A better solution is to not format your root device as ext4!
I will note, I chose this solution specifically to isolate logs from being able to fill up the root partition as well.
This keeps the logs completely separate.