No space left, with 80 GB space free

No space left, with 80 GB space free

[Tomasz Chmielewski]

I've been trying to run PostgreSQL with a ~20-30 GB database on a btrfs 
filesystem (rest of the filesystem filled with KVM images, compressed 
archives etc.). Snapshots are cool, and let you rollback to a different 
database version easily ;)

Unfortunately, running PostgreSQL on btrfs wasn't very reliable as I was 
getting rather frequent "No space left" for some disk intensive 
operations (CREATE INDEX, REINDEX, VACUUM ANALYZE).

As the error happens, I still have around 80 GB free space on the 
filesystem:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb4             336G  255G   80G  77% /mnt/btrfs


What's even more interesting, after the error happened, I'm able to 
write a ~85 GB file with dd before it exits with "out of space" (with 
and without the oflag=direct):

# dd if=/dev/urandom of=bigfile bs=1M
dd: writing `bigfile': No space left on device
81056+0 records in
81055+0 records out
84992327680 bytes (85 GB) copied, 9429.28 s, 9.0 MB/s


The kernel used was 2.6.39-rc5.

If it matters, I had the filesystem mounted with compress-force flag:

/dev/sdb4 on /mnt/btrfs type btrfs (rw,noatime,compress-force)



-- 
Tomasz Chmielewski
http://wpkg.org

Re: No space left, with 80 GB space free

[Tomasz Chmielewski]

On 16.05.2011 20:08, Tomasz Chmielewski wrote:

> The kernel used was 2.6.39-rc5.
> 
> If it matters, I had the filesystem mounted with compress-force flag:

Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?


# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb4             336G  257G   78G  77% /mnt/btrfs


coredb=# DELETE FROM core_trevent where "date" < E'2011-05-07 00:00';
ERROR:  could not write block 1014999 of relation base/16384/16521: No space left on device
CONTEXT:  writing block 1014999 of relation base/16384/16521



-- 
Tomasz Chmielewski
http://wpkg.org

Re: No space left, with 80 GB space free

[Miguel Garrido]

On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski <mangoo@wpkg.org> =
wrote:
>
> Nobody has a clue what makes btrfs run out of space when used with Po=
stgreSQL, even when there is plenty of free space left?
>
>
> # df -h
> Filesystem =A0 =A0 =A0 =A0 =A0 =A0Size =A0Used Avail Use% Mounted on
> /dev/sdb4 =A0 =A0 =A0 =A0 =A0 =A0 336G =A0257G =A0 78G =A077% /mnt/bt=
rfs
>
>
> coredb=3D# DELETE FROM core_trevent where "date" < E'2011-05-07 00:00=
';
> ERROR: =A0could not write block 1014999 of relation base/16384/16521:=
 No space left on device
> CONTEXT: =A0writing block 1014999 of relation base/16384/16521
>

Hi Tomasz,

What does btrfs fi df /mountpoint show?

Please see this link:
https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_=
out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21
As well as this one:
https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_=
to_check_the_amount_of_free_space.3F


--=20
MG
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: No space left, with 80 GB space free

[Tomasz Chmielewski]

On 21.05.2011 01:05, Miguel Garrido wrote:
> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org>  wrote:
>>
>> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
>>
>>
>> # df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb4             336G  257G   78G  77% /mnt/btrfs
>>
>>
>> coredb=# DELETE FROM core_trevent where "date"<  E'2011-05-07 00:00';
>> ERROR:  could not write block 1014999 of relation base/16384/16521: No space left on device
>> CONTEXT:  writing block 1014999 of relation base/16384/16521
>>
>
> Hi Tomasz,
>
> What does btrfs fi df /mountpoint show?

Used space is almost identical as with normal df.


> Please see this link:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_of_space.2C_but_it_looks_like_I_should_have_lots_left.21

Doesn't apply in my case: kernel is 2.6.39, device is relatively large.

It is *probably* related to compression used (compress-force - disabled 
the option, and didn't see out of space since).

-- 
Tomasz Chmielewski
http://wpkg.org

Re: No space left, with 80 GB space free

[Tomasz Chmielewski]

On 21.05.2011 01:05, Miguel Garrido wrote:
> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org>  wrote:
>>
>> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
>>
>>
>> # df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb4             336G  257G   78G  77% /mnt/btrfs
>>
>>
>> coredb=# DELETE FROM core_trevent where "date"<  E'2011-05-07 00:00';
>> ERROR:  could not write block 1014999 of relation base/16384/16521: No space left on device
>> CONTEXT:  writing block 1014999 of relation base/16384/16521

Hmm, this "No space left on device" only shows up if I write to the 
btrfs filesystem relatively fast.

For example, this way, I can only write around 1 GB (~60 MB/s write):

# cp -v acer.img.xz /mnt/btrfs/
`ac.img.xz' -> `/mnt/btrfs/ac.img.xz'
cp: writing `/mnt/btrfs/ac.img.xz': No space left on device


This way, when the writes are CPU-bound, I'm able to fully fill the 
btrfs filesystem (~8 MB/s):

# dd if=/dev/urandom of=/mnt/btrfs/bigfile


Delayed allocation?

-- 
Tomasz Chmielewski
http://wpkg.org

Re: No space left, with 80 GB space free

[Tomasz Chmielewski]

On 21.05.2011 01:05, Miguel Garrido wrote:
> On Fri, May 20, 2011 at 12:35 PM, Tomasz Chmielewski<mangoo@wpkg.org>  wrote:
>>
>> Nobody has a clue what makes btrfs run out of space when used with PostgreSQL, even when there is plenty of free space left?
>>
>>
>> # df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sdb4             336G  257G   78G  77% /mnt/btrfs
>>
>>
>> coredb=# DELETE FROM core_trevent where "date"<  E'2011-05-07 00:00';
>> ERROR:  could not write block 1014999 of relation base/16384/16521: No space left on device
>> CONTEXT:  writing block 1014999 of relation base/16384/16521

Just a follow up - I'm able to reproduce the issue with 3.0.1 kernel, with plain rsync.
I'm trying to write a ~ 4 GB file to btrfs.

"Fast" writes will make btrfs output "No space left on device".
On the other hand, if I write slowly, btrfs will be happy and write the whole file.


cp ("full speed copy") fails after a few seconds, so let's try to write with ~5 MB/s - still fails:

# rsync -a -v --progress --bwlimit=5000 . /mnt/btrfs/temp
sending incremental file list
./
randomfile
   645136384  13%    4.87MB/s    0:13:32  
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "/mnt/btrfs/temp/randomfile": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(302) [receiver=3.0.7]
rsync: connection unexpectedly closed (31 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]


Let's write with ~1 MB/s then - succeeds:

# rsync -a -v --progress --bwlimit=1000 . /mnt/btrfs/temp
sending incremental file list
./
randomfile
  4693411469 100%  988.30kB/s    1:17:17 (xfer#1, to-check=6/8)
(...)
sent 4758749321 bytes  received 112 bytes  1012176.84 bytes/sec
total size is 4758167972  speedup is 1.00


I don't think it's expected?


I see these in dmesg while copying the files:

[ 1703.978600] btrfs: free space inode generation (0) did not match free space cache generation (22496)
[ 1703.978605] btrfs: failed to load free space cache for block group 92371156992
[ 1704.026476] btrfs: free space inode generation (0) did not match free space cache generation (22502)
[ 1704.026481] btrfs: failed to load free space cache for block group 114919735296
[ 1704.035239] btrfs: free space inode generation (0) did not match free space cache generation (22503)
[ 1704.035243] btrfs: failed to load free space cache for block group 118140960768
[ 1704.035252] btrfs: free space inode generation (0) did not match free space cache generation (22481)
[ 1704.035255] btrfs: failed to load free space cache for block group 119214702592
[ 1704.294757] btrfs: free space inode generation (0) did not match free space cache generation (22503)
[ 1704.294761] btrfs: failed to load free space cache for block group 243231883264


-- 
Tomasz Chmielewski
http://wpkg.org