When the filesystem is out of chunks to write data into, No space left on device will be reported. This will prevent journal files from being recorded, containers from starting and so on.
The common reaction to this error is to run df -h and you’ll see that there is still some free space. That command isn’t measuring the btrfs primitives (chunks, metadata, etc), which is what really matters.
Running sudo btrfs fi show will give you the btrfs view of how much free space you have. When starting/stopping many Docker containers or doing a large amount of random writes, chunks will become duplicated in an inefficient manner over time.
Re-balancing the filesystem (official btrfs docs) will relocate data from empty or near-empty chunks to free up space. This operation can be done without downtime.
First, let’s see how much free space we have:
$ sudo btrfs fi show
Label: 'ROOT' uuid: 82a40c46-557e-4848-ad4d-10c6e36ed5ad
Total devices 1 FS bytes used 13.44GiB
devid 1 size 32.68GiB used 32.68GiB path /dev/xvda9
Btrfs v3.14_pre20140414
The answer: not a lot. We can re-balance to fix that.
The re-balance command can be configured to only relocate data in chunks up to a certain percentage used. This will prevent you from moving around a lot of data without a lot of benefit. If your disk is completely full, you may need to delete a few containers to create space for the re-balance operation to work with.
Let’s try to relocate chunks with less than 5% of usage:
$ sudo btrfs fi balance start -dusage=5 /
Done, had to relocate 5 out of 45 chunks
$ sudo btrfs fi show
Label: 'ROOT' uuid: 82a40c46-557e-4848-ad4d-10c6e36ed5ad
Total devices 1 FS bytes used 13.39GiB
devid 1 size 32.68GiB used 28.93GiB path /dev/xvda9
Btrfs v3.14_pre20140414
The operation took about a minute on a cloud server and gained us 4GiB of space on the filesystem. It’s up to you to find out what percentage works best for your workload, the speed of your disks, etc.
If your balance operation is taking a long time, you can open a new shell and find the status:
$ sudo btrfs balance status /
Balance on '/' is running
0 out of about 1 chunks balanced (1 considered), 100% left
ref: coreos.com