Proxmox Backup Server - No space left on device
I was recently greeted with backup failures, from what appeared to be a full disk.
102: 2026-01-22 06:00:08 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/AF/UPID:pbs:000002CC:000004AF:00000396:69721148:backup:Unraid\\x2dMain\\x3avm-102:root@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
As, it turns out, this is an inode issue and not a full mount point.
The Symptom
If you are receiving this error: failed: ENOSPC: No space left on device
but, there is no shortage of free space…
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 795M 820K 794M 1% /run
/dev/mapper/pbs-root 62G 20G 40G 34% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 0 3.9G 0% /tmp
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1.0M 0 1.0M 0% /run/credentials/systemd-journald.service
tmpfs 3.9G 196K 3.9G 1% /run/proxmox-backup
/dev/vdc1 8.0T 857G 7.2T 11% /mnt/datastore/Remote
/dev/vdb1 8.0T 980G 7.1T 12% /mnt/datastore/Local
tmpfs 795M 8.0K 795M 1% /run/user/0
tmpfs 1.0M 0 1.0M 0% /run/credentials/getty@tty1.service
Then, you might have an issue with your inode table being full. You can run df -i to confirm.
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 1006498 527 1005971 1% /dev
tmpfs 1016470 811 1015659 1% /run
/dev/mapper/pbs-root 4096000 4070273 25727 100% /
tmpfs 1016470 1 1016469 1% /dev/shm
tmpfs 1048576 9 1048567 1% /tmp
tmpfs 1016470 5 1016465 1% /run/lock
tmpfs 1024 1 1023 1% /run/credentials/systemd-journald.service
tmpfs 0 0 0 - /run/proxmox-backup
/dev/vdc1 858993344 322821 858670523 1% /mnt/datastore/Remote
/dev/vdb1 858993344 751941 858241403 1% /mnt/datastore/Local
tmpfs 203294 30 203264 1% /run/user/0
tmpfs 1024 1 1023 1% /run/credentials/getty@tty1.service
The Cause
The root issue, is ext4 has a fixed inode table, and I formatted my root disk as ext4.
If, you used XFS, or other file system, you shouldn't run into this issue.
For that reason, I would recommend against formatting the root device for your PBS server… as ext4. You can avoid this issue completely.
While you can (and should) enable proxmox log retention…
proxmox-backup-manager node update --task-log-max-days 90
This combined with manual purging of older logs did not correct my issue.
My Solution
I initially attempted to cleanup any task logs older then 90 days.
find /var/log/proxmox-backup -mtime +90 -delete
However, as you can see, I am still at 100% inode usage, with only 25k free inodes.
So, instead, I am going to create a file-backed device formatted with XFS to store my logs, which will ensure the task logs cannot exhaust the system inode table.
Here is the script I came up with
#!/usr/bin/env bash
# Creates a XFS loopback image and persistently mount for PBS task logs.
# Target mount: /var/log/proxmox-backup
## Where the image file should be stored
IMG="/mnt/datastore/Unraid-Main/pbs-tasklogs.img"
## What size should the image file be.
IMG_SIZE="20G"
## Temp loopback mount point.
LOOP_MNT="/mnt/pbs-tasklogs"
## The directory to which it will be mounted to.
TARGET_DIR="/var/log/proxmox-backup"
if [[ $EUID -ne 0 ]]; then
echo "ERROR: run as root" >&2
exit 1
fi
if [[ ! -d "$(dirname "$IMG")" ]]; then
echo "ERROR: parent directory does not exist: $(dirname "$IMG")" >&2
exit 1
fi
if [[ -e "$IMG" ]]; then
echo "ERROR: image already exists: $IMG" >&2
exit 1
fi
if mountpoint -q "$TARGET_DIR"; then
echo "ERROR: $TARGET_DIR is already a mountpoint; refusing to proceed." >&2
exit 1
fi
mkdir -p "$LOOP_MNT"
echo "Creating sparse image: $IMG ($IMG_SIZE)"
fallocate -l "$IMG_SIZE" "$IMG"
echo "Formatting as XFS"
mkfs.xfs -f "$IMG" >/dev/null
echo "Mounting loop image at $LOOP_MNT"
mount -o loop,noatime "$IMG" "$LOOP_MNT"
echo "Stopping proxmox backup server"
systemctl stop proxmox-backup-proxy proxmox-backup-api 2>/dev/null || true
echo "Migrate existing logs into loop mount (Warning- this may take a long time...)"
rsync -a --delete "${TARGET_DIR}/" "${LOOP_MNT}/" || true
echo "Freeing inodes. (This may take a long time.)"
find "$TARGET_DIR" -xdev -mindepth 1 -delete
mkdir -p "$TARGET_DIR"
mount --bind "$LOOP_MNT" "$TARGET_DIR"
echo "Updating /etc/fstab"
FSTAB_LINE1="${IMG} ${LOOP_MNT} xfs loop,noatime 0 0"
FSTAB_LINE2="${LOOP_MNT} ${TARGET_DIR} none bind 0 0"
grep -Fqx "$FSTAB_LINE1" /etc/fstab || echo "$FSTAB_LINE1" >> /etc/fstab
grep -Fqx "$FSTAB_LINE2" /etc/fstab || echo "$FSTAB_LINE2" >> /etc/fstab
echo "Reloading SystemD"
systemctl daemon-reload
echo "Starting proxmox backup server"
systemctl start proxmox-backup-api proxmox-backup-proxy 2>/dev/null || true
echo "----------------------------------------------------------------------------------------------------"
echo "--- df -h (target) ---"
df -h "$TARGET_DIR" || true
echo "--- df -i (target) ---"
df -i "$TARGET_DIR" || true
echo "--- mount | grep (pbs-tasklogs) ---"
mount | grep -E "(${LOOP_MNT}|${TARGET_DIR})" || true
echo "Done."
I did not provision another mount, because there is plenty of free room on the root disk.
I am simply running into an inode issue. By provisioning a file-back block device, I am relocating the files/inodes into this file and off of the root device.
You can provision additional storage and mount if, if desired. However, I personally did not see too much benefit to doing so.
Disclaimers
I cannot guarantee the above script will work for you. I also cannot guarantee it will not randomly decide to nuke your entire lab.
Make sure you have backups / snapshots, in the event something catastrophically unexpected occurs.
I can tell you, I personally tested and executed the script on my installation, and if you are seeing this blog post, then it was successful.
Results
It took quite a while for the logs to relocate. Here is output captured after a few minutes of running.
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pbs-root 62G 20G 40G 34% /
/dev/loop0 20G 4.9G 16G 25% /mnt/pbs-tasklogs
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pbs-root 4096000 4070274 25726 100% /
/dev/loop0 10485760 1739611 8746149 17% /mnt/pbs-tasklogs
After oh… 10 minutes…. It finally completed
root@pbs:~# nano makeLogMount
root@pbs:~# chmod +x makeLogMount
root@pbs:~# ./makeLogMount
Creating sparse image: /mnt/datastore/Unraid-Main/pbs-tasklogs.img (20G)
Formatting as XFS
Mounting loop image at /mnt/pbs-tasklogs
Stopping proxmox backup server
Migrate existing logs into loop mount
Updating /etc/fstab
Reloading SystemD
Starting proxmox backup server
----------------------------------------------------------------------------------------------------
--- df -h (target) ---
Filesystem Size Used Avail Use% Mounted on
/dev/loop0 20G 11G 9.8G 52% /var/log/proxmox-backup
--- df -i (target) ---
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/loop0 10485760 3908312 6577448 38% /var/log/proxmox-backup
--- mount | grep (pbs-tasklogs) ---
/mnt/datastore/Unraid-Main/pbs-tasklogs.img on /mnt/pbs-tasklogs type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/mnt/datastore/Unraid-Main/pbs-tasklogs.img on /var/log/proxmox-backup type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
Done.
root@pbs:~#
inode tables look much, much better now.. with all of the inodes from logs being inside of the file-backed block device.
root@pbs:~# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pbs-root 62G 12G 48G 20% /
/dev/loop0 20G 11G 9.9G 51% /mnt/pbs-tasklogs
root@pbs:~# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/pbs-root 4096000 161965 3934035 4% /
/dev/loop0 10485760 3908312 6577448 38% /mnt/pbs-tasklogs
root@pbs:~#
Summary
This is the solution I used, and while I cannot promise it is the best solution, It resolved my issues.
There are many ways to resolve this issue. A better solution is to not format your root device as ext4!
I will note, I chose this solution specifically to isolate logs from being able to fill up the root partition as well.
This keeps the logs completely separate.