No space left on device (28)

No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi,

while copying a 15GB file to my btrfs volume i'm getting "No space left
on device".

Some information:

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.97GB
Metadata: total=8.00MB, used=0.00

# df -h
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt

# btrfs fi show
Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
        Total devices 1 FS bytes used 15.07TB
        devid    1 size 18.19TB used 18.19TB path /dev/dm-3

# btrfs fi balance start -dusage=5 /mnt/
Done, had to relocate 0 out of 18604 chunks

What wrong here? What is about the last 3TB?

Greets.
Stefan

Re: No space left on device (28)

[Chris Mason]

Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
> Hi,
> 
> while copying a 15GB file to my btrfs volume i'm getting "No space left
> on device".
> 
> Some information:
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.97GB
> Metadata: total=8.00MB, used=0.00
> 
> # df -h
> Filesystem             Size  Used Avail Use% Mounted on
> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> 
> # btrfs fi show
> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>         Total devices 1 FS bytes used 15.07TB
>         devid    1 size 18.19TB used 18.19TB path /dev/dm-3
> 
> # btrfs fi balance start -dusage=5 /mnt/
> Done, had to relocate 0 out of 18604 chunks
> 
> What wrong here? What is about the last 3TB?

Which kernel are you running?

-chris

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi Chris,

Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:

> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>> Hi,
>> 
>> while copying a 15GB file to my btrfs volume i'm getting "No space left
>> on device".
>> 
>> Some information:
>> 
>> # btrfs filesystem df /mnt/
>> Data: total=18.14TB, used=15.05TB
>> System, DUP: total=8.00MB, used=1.94MB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=23.98GB, used=17.97GB
>> Metadata: total=8.00MB, used=0.00
>> 
>> # df -h
>> Filesystem             Size  Used Avail Use% Mounted on
>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>> 
>> # btrfs fi show
>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>        Total devices 1 FS bytes used 15.07TB
>>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>> 
>> # btrfs fi balance start -dusage=5 /mnt/
>> Done, had to relocate 0 out of 18604 chunks
>> 
>> What wrong here? What is about the last 3TB?
> 
> Which kernel are you running?
> 
> -chris

vanilla 3.8.3.

Stefan

Re: No space left on device (28)

[Chris Mason]

Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
> Hi Chris,
> 
> Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
> 
> > Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
> >> Hi,
> >> 
> >> while copying a 15GB file to my btrfs volume i'm getting "No space left
> >> on device".
> >> 
> >> Some information:
> >> 
> >> # btrfs filesystem df /mnt/
> >> Data: total=18.14TB, used=15.05TB
> >> System, DUP: total=8.00MB, used=1.94MB
> >> System: total=4.00MB, used=0.00
> >> Metadata, DUP: total=23.98GB, used=17.97GB
> >> Metadata: total=8.00MB, used=0.00
> >> 
> >> # df -h
> >> Filesystem             Size  Used Avail Use% Mounted on
> >> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> >> 
> >> # btrfs fi show
> >> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
> >>        Total devices 1 FS bytes used 15.07TB
> >>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
> >> 
> >> # btrfs fi balance start -dusage=5 /mnt/
> >> Done, had to relocate 0 out of 18604 chunks
> >> 
> >> What wrong here? What is about the last 3TB?
> > 
> > Which kernel are you running?
> > 
> > -chris
> 
> vanilla 3.8.3.

Ok, with the 3.9 merge window Josef changed how we do the reservations.
Are you able to try a slightly more experimental kernel?

-chris

Re: No space left on device (28)

[Stefan Priebe]

Yes, i've now upgraded to 3.9-rc3 same result.

rsync: rename 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" 
-> 
".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": 
No space left on device (28)

# btrfs filesystem df /mnt/
Data: total=18.14TB, used=15.05TB
System, DUP: total=8.00MB, used=1.94MB
System: total=4.00MB, used=0.00
Metadata, DUP: total=23.98GB, used=17.98GB
Metadata: total=8.00MB, used=0.00

Stefan

Am 21.03.2013 20:02, schrieb Chris Mason:
> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>> Hi Chris,
>>
>> Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
>>
>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>> Hi,
>>>>
>>>> while copying a 15GB file to my btrfs volume i'm getting "No space left
>>>> on device".
>>>>
>>>> Some information:
>>>>
>>>> # btrfs filesystem df /mnt/
>>>> Data: total=18.14TB, used=15.05TB
>>>> System, DUP: total=8.00MB, used=1.94MB
>>>> System: total=4.00MB, used=0.00
>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>> Metadata: total=8.00MB, used=0.00
>>>>
>>>> # df -h
>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>
>>>> # btrfs fi show
>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>         Total devices 1 FS bytes used 15.07TB
>>>>         devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>>>>
>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>> Done, had to relocate 0 out of 18604 chunks
>>>>
>>>> What wrong here? What is about the last 3TB?
>>>
>>> Which kernel are you running?
>>>
>>> -chris
>>
>> vanilla 3.8.3.
>
> Ok, with the 3.9 merge window Josef changed how we do the reservations.
> Are you able to try a slightly more experimental kernel?
>
> -chris
>

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Should i try to downgrade?

Greets,
Stefan

Am 21.03.2013 um 20:42 schrieb Stefan Priebe <s.priebe@profihost.ag>:

> Yes, i've now upgraded to 3.9-rc3 same result.
> 
> rsync: rename "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" -> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": No space left on device (28)
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.98GB
> Metadata: total=8.00MB, used=0.00
> 
> Stefan
> 
> Am 21.03.2013 20:02, schrieb Chris Mason:
>> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>>> Hi Chris,
>>> 
>>> Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
>>> 
>>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>>> Hi,
>>>>> 
>>>>> while copying a 15GB file to my btrfs volume i'm getting "No space left
>>>>> on device".
>>>>> 
>>>>> Some information:
>>>>> 
>>>>> # btrfs filesystem df /mnt/
>>>>> Data: total=18.14TB, used=15.05TB
>>>>> System, DUP: total=8.00MB, used=1.94MB
>>>>> System: total=4.00MB, used=0.00
>>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>>> Metadata: total=8.00MB, used=0.00
>>>>> 
>>>>> # df -h
>>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>> 
>>>>> # btrfs fi show
>>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>>        Total devices 1 FS bytes used 15.07TB
>>>>>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>>>>> 
>>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>>> Done, had to relocate 0 out of 18604 chunks
>>>>> 
>>>>> What wrong here? What is about the last 3TB?
>>>> 
>>>> Which kernel are you running?
>>>> 
>>>> -chris
>>> 
>>> vanilla 3.8.3.
>> 
>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>> Are you able to try a slightly more experimental kernel?
>> 
>> -chris
>> 

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi Chris,

>>>>> Which kernel are you running?
>>>>>
>>>>> -chris
>>>>
>>>> vanilla 3.8.3.
>>>
>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>>> Are you able to try a slightly more experimental kernel?

any ideas what i can check? 3.9-rc3 gives me same results.

Greets,
Stefan

Re: No space left on device (28)

[Josef Bacik]

On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Chris,
> 
> >>>>> Which kernel are you running?
> >>>>>
> >>>>> -chris
> >>>>
> >>>> vanilla 3.8.3.
> >>>
> >>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
> >>> Are you able to try a slightly more experimental kernel?
> 
> any ideas what i can check? 3.9-rc3 gives me same results.
> 

Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
patch for you to run so I can narrow down what's going on.  Thanks,

Josef

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi Josef,
Am 22.03.2013 14:53, schrieb Josef Bacik:
> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
>> Hi Chris,
>>
>>>>>>> Which kernel are you running?
>>>>>>>
>>>>>>> -chris
>>>>>>
>>>>>> vanilla 3.8.3.
>>>>>
>>>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>>>>> Are you able to try a slightly more experimental kernel?
>>
>> any ideas what i can check? 3.9-rc3 gives me same results.
>>
> 
> Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
> patch for you to run so I can narrow down what's going on.  Thanks,

Great!

Thanks - just wanted to know that it's not my fault. I'm happy to test
the patch and provide feedback.

Stefan

Re: No space left on device (28)

[Josef Bacik]

On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
> Am 22.03.2013 14:53, schrieb Josef Bacik:
> > On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
> >> Hi Chris,
> >>
> >>>>>>> Which kernel are you running?
> >>>>>>>
> >>>>>>> -chris
> >>>>>>
> >>>>>> vanilla 3.8.3.
> >>>>>
> >>>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
> >>>>> Are you able to try a slightly more experimental kernel?
> >>
> >> any ideas what i can check? 3.9-rc3 gives me same results.
> >>
> > 
> > Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
> > patch for you to run so I can narrow down what's going on.  Thanks,
> 
> Great!
> 
> Thanks - just wanted to know that it's not my fault. I'm happy to test
> the patch and provide feedback.
> 

Ok I think we are way over-reserving for rename, can you give this patch a whirl
and see what happens?  If it still fails can you capture dmesg and reply with
that so I can see what's going on.  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 9ac2eca..aabaea6 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4161,6 +4161,11 @@ out:
 			ret = 0;
 	}
 	if (flushing) {
+		if (ret == -ENOSPC) {
+			printk(KERN_ERR "returning enospc, dumping space info\n");
+			dump_space_info(space_info, 0, 0);
+		}
+
 		spin_lock(&space_info->lock);
 		space_info->flush = 0;
 		wake_up_all(&space_info->wait);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ca1b767..3980ae7 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -3679,11 +3679,9 @@ static struct btrfs_trans_handle *__unlink_start_trans(struct inode *dir,
 	 * 1 for the dir item
 	 * 1 for the dir index
 	 * 1 for the inode ref
-	 * 1 for the inode ref in the tree log
-	 * 2 for the dir entries in the log
 	 * 1 for the inode
 	 */
-	trans = btrfs_start_transaction(root, 8);
+	trans = btrfs_start_transaction(root, 5);
 	if (!IS_ERR(trans) || PTR_ERR(trans) != -ENOSPC)
 		return trans;
 
@@ -8127,7 +8125,7 @@ static int btrfs_rename(struct inode *old_dir, struct dentry *old_dentry,
 	 * inodes.  So 5 * 2 is 10, plus 1 for the new link, so 11 total items
 	 * should cover the worst case number of items we'll modify.
 	 */
-	trans = btrfs_start_transaction(root, 20);
+	trans = btrfs_start_transaction(root, 11);
 	if (IS_ERR(trans)) {
                 ret = PTR_ERR(trans);
                 goto out_notrans;
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe]

Hi Josef,
Am 22.03.2013 16:54, schrieb Josef Bacik:
> On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote:
>> Hi Josef,
>> Am 22.03.2013 14:53, schrieb Josef Bacik:
>>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
>>>> Hi Chris,
>>>>
>>>>>>>>> Which kernel are you running?
>>>>>>>>>
>>>>>>>>> -chris
>>>>>>>>
>>>>>>>> vanilla 3.8.3.
>>>>>>>
>>>>>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>>>>>>> Are you able to try a slightly more experimental kernel?
>>>>
>>>> any ideas what i can check? 3.9-rc3 gives me same results.
>>>>
>>>
>>> Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
>>> patch for you to run so I can narrow down what's going on.  Thanks,
>>
>> Great!
>>
>> Thanks - just wanted to know that it's not my fault. I'm happy to test
>> the patch and provide feedback.
>>
> Ok I think we are way over-reserving for rename, can you give this patch a whirl
> and see what happens?  If it still fails can you capture dmesg and reply with
> that so I can see what's going on.  Thanks,

Thanks for the patch. I was able to copy some files and i'm still able 
put it copy some then fails on some then copies some then fails on some.

Rsync output:
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-wm8350.c
        12156 100%   11.17kB/s    0:00:01 (xfer#21395, to-check=1008/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/rtc-x1205.c
        16460 100%   15.05kB/s    0:00:01 (xfer#21396, to-check=1007/52625)
.software/kernel/linux-3.9-rc3/drivers/rtc/systohc.c
         1223 100%    1.12kB/s    0:00:01 (xfer#21397, to-check=1006/52625)
.software/kernel/linux-3.9-rc3/drivers/s390/
.software/kernel/linux-3.9-rc3/drivers/s390/Makefile
          144 100%    0.13kB/s    0:00:01 (xfer#21398, to-check=1052/52672)
.software/kernel/linux-3.9-rc3/drivers/s390/block/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/block" failed: No 
space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/char/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/char" failed: No space 
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/cio/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/cio" failed: No space 
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.88pm8607.c.YBi0Rz" 
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/crypto/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/crypto" failed: No 
space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/kvm/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/kvm" failed: No space 
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/s390/net/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/net" failed: No space 
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Kconfig.0lsfuE" 
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.Makefile.C9wI7I" 
failed: No space left on device (28)
.software/kernel/linux-3.9-rc3/drivers/s390/scsi/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/s390/scsi" failed: No space 
left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/sbus/
.software/kernel/linux-3.9-rc3/drivers/sbus/Makefile
           70 100%    0.06kB/s    0:00:01 (xfer#21399, to-check=1003/52824)
.software/kernel/linux-3.9-rc3/drivers/sbus/char/
rsync: recv_generator: mkdir 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/sbus/char" failed: No space 
left on device (28)
rsync: mkstemp 
"/mnt/.software/kernel/linux-3.9-rc3/drivers/regulator/.aat2870-regulator.c.BGDwPN" 
failed: No space left on device (28)
*** Skipping any contents from this failed directory ***
.software/kernel/linux-3.9-rc3/drivers/scsi/
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.ko.cmd
          208 100%    0.16kB/s    0:00:01 (xfer#21400, to-check=1407/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.mod.o.cmd
        27987 100%   21.54kB/s    0:00:01 (xfer#21401, to-check=1406/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-9xxx.o.cmd
        43340 100%   32.63kB/s    0:00:01 (xfer#21402, to-check=1405/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.ko.cmd
          204 100%   15.32kB/s    0:00:00 (xfer#21403, to-check=1404/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.mod.o.cmd
        27975 100%    1.91MB/s    0:00:00 (xfer#21404, to-check=1403/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-sas.o.cmd
        43327 100%  983.99kB/s    0:00:00 (xfer#21405, to-check=1402/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.ko.cmd
          208 100%    4.72kB/s    0:00:00 (xfer#21406, to-check=1401/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.mod.o.cmd
        27987 100%  621.16kB/s    0:00:00 (xfer#21407, to-check=1400/53242)
.software/kernel/linux-3.9-rc3/drivers/scsi/.3w-xxxx.o.cmd
        43340 100%  742.53kB/s    0:00:00 (xfer#21408, to-check=1399/53242)

dmesg Output was pretty long so i posted to pastebin:
http://pastebin.com/raw.php?i=5kUS5MGA

Greets,
Stefan

Re: No space left on device (28)

[Josef Bacik]

On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe wrote:
> Hi Josef,
> Am 22.03.2013 16:54, schrieb Josef Bacik:
> > On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote:
> >> Hi Josef,
> >> Am 22.03.2013 14:53, schrieb Josef Bacik:
> >>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
> >>>> Hi Chris,
> >>>>
> >>>>>>>>> Which kernel are you running?
> >>>>>>>>>
> >>>>>>>>> -chris
> >>>>>>>>
> >>>>>>>> vanilla 3.8.3.
> >>>>>>>
> >>>>>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
> >>>>>>> Are you able to try a slightly more experimental kernel?
> >>>>
> >>>> any ideas what i can check? 3.9-rc3 gives me same results.
> >>>>
> >>>
> >>> Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
> >>> patch for you to run so I can narrow down what's going on.  Thanks,
> >>
> >> Great!
> >>
> >> Thanks - just wanted to know that it's not my fault. I'm happy to test
> >> the patch and provide feedback.
> >>
> > Ok I think we are way over-reserving for rename, can you give this patch a whirl
> > and see what happens?  If it still fails can you capture dmesg and reply with
> > that so I can see what's going on.  Thanks,
> 
> Thanks for the patch. I was able to copy some files and i'm still able 
> put it copy some then fails on some then copies some then fails on some.
> 

Ok so I've been working on another ENOSPC related problem that I think is
affecting you as well.  So I'm going to nail that down and see if that helps you
too, if not we'll poke around your problem some more and try and work out what's
happening.  I'll get back to this fresh Monday morning.  Thanks,

Josef

Re: No space left on device (28)

[Stefan Priebe]

Hi Jsoef,

thanks!

Am 22.03.2013 21:49, schrieb Josef Bacik:
> On Fri, Mar 22, 2013 at 01:10:05PM -0600, Stefan Priebe wrote:
>> Hi Josef,
>> Am 22.03.2013 16:54, schrieb Josef Bacik:
>>> On Fri, Mar 22, 2013 at 07:56:41AM -0600, Stefan Priebe - Profihost AG wrote:
>>>> Hi Josef,
>>>> Am 22.03.2013 14:53, schrieb Josef Bacik:
>>>>> On Fri, Mar 22, 2013 at 06:11:56AM -0600, Stefan Priebe - Profihost AG wrote:
>>>>>> Hi Chris,
>>>>>>
>>>>>>>>>>> Which kernel are you running?
>>>>>>>>>>>
>>>>>>>>>>> -chris
>>>>>>>>>>
>>>>>>>>>> vanilla 3.8.3.
>>>>>>>>>
>>>>>>>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>>>>>>>>> Are you able to try a slightly more experimental kernel?
>>>>>>
>>>>>> any ideas what i can check? 3.9-rc3 gives me same results.
>>>>>>
>>>>>
>>>>> Sorry Stefan I'm almost done with what I'm working on and then I'll work up a
>>>>> patch for you to run so I can narrow down what's going on.  Thanks,
>>>>
>>>> Great!
>>>>
>>>> Thanks - just wanted to know that it's not my fault. I'm happy to test
>>>> the patch and provide feedback.
>>>>
>>> Ok I think we are way over-reserving for rename, can you give this patch a whirl
>>> and see what happens?  If it still fails can you capture dmesg and reply with
>>> that so I can see what's going on.  Thanks,
>>
>> Thanks for the patch. I was able to copy some files and i'm still able
>> put it copy some then fails on some then copies some then fails on some.
>>
>
> Ok so I've been working on another ENOSPC related problem that I think is
> affecting you as well.  So I'm going to nail that down and see if that helps you
> too, if not we'll poke around your problem some more and try and work out what's
> happening.  I'll get back to this fresh Monday morning.  Thanks,
>
> Josef
>

Re: No space left on device (28)

[Josef Bacik]

On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe wrote:
> Hi Jsoef,
> 
> thanks!

Ok I don't think the thing I just fixed will make any difference for you so
here's another debug patch, just apply it on top of what I've already sent you
and re-run and give me the dmesg again.  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index aabaea6..bf6433f 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4162,7 +4162,11 @@ out:
 	}
 	if (flushing) {
 		if (ret == -ENOSPC) {
-			printk(KERN_ERR "returning enospc, dumping space info\n");
+			spin_lock(&block_rsv->lock);
+			printk(KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, "
+			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
+			       block_rsv->size, block_rsv->reserved, flush, flush_state);
+			spin_unlock(&block_rsv->lock);
 			dump_space_info(space_info, 0, 0);
 		}
 
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe]

Hi,

output here:
[  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.548280] space_info 4 has 6439292928 free, is full
[  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=32768, may_use=6438354944, readonly=65536
[  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.552264] space_info 4 has 6439284736 free, is full
[  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  590.556258] space_info 4 has 6439284736 free, is full
[  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=40960, may_use=6438354944, readonly=65536
[  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  591.147373] space_info 4 has 6439235584 free, is full
[  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
reserved=90112, may_use=6438354944, readonly=65536
[  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[  595.562390] space_info 4 has 6439120896 free, is full
[  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
reserved=73728, may_use=6438297600, readonly=65536

Am 25.03.2013 21:14, schrieb Josef Bacik:
> On Fri, Mar 22, 2013 at 02:55:07PM -0600, Stefan Priebe wrote:
>> Hi Jsoef,
>>
>> thanks!
>
> Ok I don't think the thing I just fixed will make any difference for you so
> here's another debug patch, just apply it on top of what I've already sent you
> and re-run and give me the dmesg again.  Thanks,
>
> Josef
>
>
> diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
> index aabaea6..bf6433f 100644
> --- a/fs/btrfs/extent-tree.c
> +++ b/fs/btrfs/extent-tree.c
> @@ -4162,7 +4162,11 @@ out:
>   	}
>   	if (flushing) {
>   		if (ret == -ENOSPC) {
> -			printk(KERN_ERR "returning enospc, dumping space info\n");
> +			spin_lock(&block_rsv->lock);
> +			printk(KERN_ERR "returning enospc, space_info %u, size %Lu reserved %Lu, "
> +			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
> +			       block_rsv->size, block_rsv->reserved, flush, flush_state);
> +			spin_unlock(&block_rsv->lock);
>   			dump_space_info(space_info, 0, 0);
>   		}
>
>

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> Hi,
> 
> output here:
> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.548280] space_info 4 has 6439292928 free, is full
> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=32768, may_use=6438354944, readonly=65536
> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.552264] space_info 4 has 6439284736 free, is full
> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=40960, may_use=6438354944, readonly=65536
> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  590.556258] space_info 4 has 6439284736 free, is full
> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=40960, may_use=6438354944, readonly=65536
> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  591.147373] space_info 4 has 6439235584 free, is full
> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
> reserved=90112, may_use=6438354944, readonly=65536
> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [  595.562390] space_info 4 has 6439120896 free, is full
> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
> reserved=73728, may_use=6438297600, readonly=65536
> 

Weird, we have all the flushing stuff set and yet there is still a whole lot of
outstanding reservations.  Do you have compression enabled?  Thanks,

Josef

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi Josef,

Am 26.03.2013 13:53, schrieb Josef Bacik:
> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
>> Hi,
>>
>> output here:
>> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
>> 2, flush_state 7  dumping space info
>> [  590.548280] space_info 4 has 6439292928 free, is full
>> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
>> reserved=32768, may_use=6438354944, readonly=65536
>> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
>> 2, flush_state 7  dumping space info
>> [  590.552264] space_info 4 has 6439284736 free, is full
>> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
>> reserved=40960, may_use=6438354944, readonly=65536
>> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
>> 2, flush_state 7  dumping space info
>> [  590.556258] space_info 4 has 6439284736 free, is full
>> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
>> reserved=40960, may_use=6438354944, readonly=65536
>> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
>> 2, flush_state 7  dumping space info
>> [  591.147373] space_info 4 has 6439235584 free, is full
>> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
>> reserved=90112, may_use=6438354944, readonly=65536
>> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
>> 2, flush_state 7  dumping space info
>> [  595.562390] space_info 4 has 6439120896 free, is full
>> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
>> reserved=73728, may_use=6438297600, readonly=65536
>>
> 
> Weird, we have all the flushing stuff set and yet there is still a whole lot of
> outstanding reservations.  Do you have compression enabled?  Thanks,

no - it's just mounted with mount -o noatime

:~# cat /proc/mounts | grep btrfs
/dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0

Greets,
Stefan

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
> 
> Am 26.03.2013 13:53, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >> Hi,
> >>
> >> output here:
> >> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
> >> 2, flush_state 7  dumping space info
> >> [  590.548280] space_info 4 has 6439292928 free, is full
> >> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
> >> reserved=32768, may_use=6438354944, readonly=65536
> >> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
> >> 2, flush_state 7  dumping space info
> >> [  590.552264] space_info 4 has 6439284736 free, is full
> >> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
> >> 2, flush_state 7  dumping space info
> >> [  590.556258] space_info 4 has 6439284736 free, is full
> >> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
> >> reserved=40960, may_use=6438354944, readonly=65536
> >> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
> >> 2, flush_state 7  dumping space info
> >> [  591.147373] space_info 4 has 6439235584 free, is full
> >> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
> >> reserved=90112, may_use=6438354944, readonly=65536
> >> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
> >> 2, flush_state 7  dumping space info
> >> [  595.562390] space_info 4 has 6439120896 free, is full
> >> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
> >> reserved=73728, may_use=6438297600, readonly=65536
> >>
> > 
> > Weird, we have all the flushing stuff set and yet there is still a whole lot of
> > outstanding reservations.  Do you have compression enabled?  Thanks,
> 
> no - it's just mounted with mount -o noatime
> 
> :~# cat /proc/mounts | grep btrfs
> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>

Ok I think I see what's going on.  Can you try this patch and see if it fixes
it?  Thanks,

Josef

 
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index bf6433f..84e8909 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -3803,6 +3803,19 @@ static int can_overcommit(struct btrfs_root *root,
 	return 0;
 }
 
+static int btrfs_try_writeback(struct super_block *sb, unsigned long nr_pages,
+			       enum wb_reason reason)
+{
+	if (!writeback_in_progress(sb->s_bdi) &&
+	    down_read_trylock(&sb->s_umount)) {
+		writeback_inodes_sb_nr(sb, nr_pages, reason);
+		up_read(&sb->s_umount);
+		return 1;
+	}
+
+	return 0;
+}
+
 void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 				  unsigned long nr_pages)
 {
@@ -3810,8 +3823,7 @@ void btrfs_writeback_inodes_sb_nr(struct btrfs_root *root,
 	int started;
 
 	/* If we can not start writeback, just sync all the delalloc file. */
-	started = try_to_writeback_inodes_sb_nr(sb, nr_pages,
-						      WB_REASON_FS_FREE_SPACE);
+	started = btrfs_try_writeback(sb, nr_pages, WB_REASON_FS_FREE_SPACE);
 	if (!started) {
 		/*
 		 * We needn't worry the filesystem going from r/w to r/o though
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi Josef,


Am 26.03.2013 14:30, schrieb Josef Bacik:
> On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote:
>> Hi Josef,
>>
>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
>>>> Hi,
>>>>
>>>> output here:
>>>> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
>>>> 2, flush_state 7  dumping space info
>>>> [  590.548280] space_info 4 has 6439292928 free, is full
>>>> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
>>>> reserved=32768, may_use=6438354944, readonly=65536
>>>> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
>>>> 2, flush_state 7  dumping space info
>>>> [  590.552264] space_info 4 has 6439284736 free, is full
>>>> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
>>>> reserved=40960, may_use=6438354944, readonly=65536
>>>> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
>>>> 2, flush_state 7  dumping space info
>>>> [  590.556258] space_info 4 has 6439284736 free, is full
>>>> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
>>>> reserved=40960, may_use=6438354944, readonly=65536
>>>> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
>>>> 2, flush_state 7  dumping space info
>>>> [  591.147373] space_info 4 has 6439235584 free, is full
>>>> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
>>>> reserved=90112, may_use=6438354944, readonly=65536
>>>> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
>>>> 2, flush_state 7  dumping space info
>>>> [  595.562390] space_info 4 has 6439120896 free, is full
>>>> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
>>>> reserved=73728, may_use=6438297600, readonly=65536
>>>>
>>>
>>> Weird, we have all the flushing stuff set and yet there is still a whole lot of
>>> outstanding reservations.  Do you have compression enabled?  Thanks,
>>
>> no - it's just mounted with mount -o noatime
>>
>> :~# cat /proc/mounts | grep btrfs
>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>
> 
> Ok I think I see what's going on.  Can you try this patch and see if it fixes
> it?  Thanks,

It still does not fix the problem.

The rsync output looks like this so it does not work for file a but then
continues on c d e, ...

sync -av --progress /backup/ /mnt/
sending incremental file list
.etc_openvpn/ipp.txt
         229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
.etc_openvpn/openvpn-status.log
         360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
".etc_openvpn/ipp.txt": No space left on device (28)
.log/
.log/UcliEvt.log
      104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
.log/auth.log
    15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
.log/auth.log.1
    19431424  61%    7.35MB/s    0:00:01

the dmesg output looks like this:
[  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[  551.323694] space_info 4 has 6439526400 free, is full
[  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
reserved=49152, may_use=6438453248, readonly=65536

Stefan

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 07:49:16AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
> 
> 
> Am 26.03.2013 14:30, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote:
> >> Hi Josef,
> >>
> >> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >>>> Hi,
> >>>>
> >>>> output here:
> >>>> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.548280] space_info 4 has 6439292928 free, is full
> >>>> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=32768, may_use=6438354944, readonly=65536
> >>>> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.552264] space_info 4 has 6439284736 free, is full
> >>>> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.556258] space_info 4 has 6439284736 free, is full
> >>>> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  591.147373] space_info 4 has 6439235584 free, is full
> >>>> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=90112, may_use=6438354944, readonly=65536
> >>>> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  595.562390] space_info 4 has 6439120896 free, is full
> >>>> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
> >>>> reserved=73728, may_use=6438297600, readonly=65536
> >>>>
> >>>
> >>> Weird, we have all the flushing stuff set and yet there is still a whole lot of
> >>> outstanding reservations.  Do you have compression enabled?  Thanks,
> >>
> >> no - it's just mounted with mount -o noatime
> >>
> >> :~# cat /proc/mounts | grep btrfs
> >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>
> > 
> > Ok I think I see what's going on.  Can you try this patch and see if it fixes
> > it?  Thanks,
> 
> It still does not fix the problem.
> 
> The rsync output looks like this so it does not work for file a but then
> continues on c d e, ...
> 
> sync -av --progress /backup/ /mnt/
> sending incremental file list
> .etc_openvpn/ipp.txt
>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> .etc_openvpn/openvpn-status.log
>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> ".etc_openvpn/ipp.txt": No space left on device (28)
> .log/
> .log/UcliEvt.log
>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> .log/auth.log
>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> .log/auth.log.1
>     19431424  61%    7.35MB/s    0:00:01
> 
> the dmesg output looks like this:
> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7  dumping space info
> [  551.323694] space_info 4 has 6439526400 free, is full
> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> reserved=49152, may_use=6438453248, readonly=65536
>

Ok so then this is probably it, let me know if it helps.  Thanks,

Josef
 
diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index dc08d77..005c45d 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 	INIT_LIST_HEAD(&splice);
 	INIT_LIST_HEAD(&works);
 
+	mutex_lock(&root->fs_info->ordered_operations_mutex);
 	spin_lock(&root->fs_info->ordered_extent_lock);
 	list_splice_init(&root->fs_info->ordered_extents, &splice);
 	while (!list_empty(&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 
 		cond_resched();
 	}
+	mutex_unlock(&root->fs_info->ordered_operations_mutex);
 }
 
 /*
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi,
Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>> no - it's just mounted with mount -o noatime
>>>>
>>>> :~# cat /proc/mounts | grep btrfs
>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>>>
>>>
>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
>>> it?  Thanks,
>>
>> It still does not fix the problem.
>>
>> The rsync output looks like this so it does not work for file a but then
>> continues on c d e, ...
>>
>> sync -av --progress /backup/ /mnt/
>> sending incremental file list
>> .etc_openvpn/ipp.txt
>>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
>> .etc_openvpn/openvpn-status.log
>>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>> ".etc_openvpn/ipp.txt": No space left on device (28)
>> .log/
>> .log/UcliEvt.log
>>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
>> .log/auth.log
>>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
>> .log/auth.log.1
>>     19431424  61%    7.35MB/s    0:00:01
>>
>> the dmesg output looks like this:
>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [  551.323694] space_info 4 has 6439526400 free, is full
>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
>> reserved=49152, may_use=6438453248, readonly=65536
>>
> 
> Ok so then this is probably it, let me know if it helps.  Thanks,

OK it now has copied a lot of files (170) without an error all were very
small.

Then this again:
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.vhost_net.mod.GrhWet" ->
".software/kernel/linux-3.9-rc3/.tmp_versions/vhost_net.mod": No space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rhine.mod.X2Ofhz" ->
".software/kernel/linux-3.9-rc3/.tmp_versions/via-rhine.mod": No space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.via-rng.mod.RAyxkF"
-> ".software/kernel/linux-3.9-rc3/.tmp_versions/via-rng.mod": No space
left on device (28)
rsync: rename
"/mnt/.software/kernel/linux-3.9-rc3/.tmp_versions/.virtio.mod.w7INoL"
-> ".software/kernel/linux-3.9-rc3/.tmp_versions/virtio.mod": No space
left on device (28)^C

[ 5012.468988] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.472981] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.472982] space_info 4 has 6439399424 free, is full
[ 5012.472982] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.476974] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.476975] space_info 4 has 6439399424 free, is full
[ 5012.476976] space_info total=25748307968, used=19308773376, pinned=0,
reserved=69632, may_use=6438354944, readonly=65536
[ 5012.480968] returning enospc, space_info 3, size 0 reserved 0, flush
2, flush_state 7  dumping space info
[ 5012.480969] space_info 4 has 6439399424 free, is full

Stefan

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi,
> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>> no - it's just mounted with mount -o noatime
> >>>>
> >>>> :~# cat /proc/mounts | grep btrfs
> >>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>>>
> >>>
> >>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
> >>> it?  Thanks,
> >>
> >> It still does not fix the problem.
> >>
> >> The rsync output looks like this so it does not work for file a but then
> >> continues on c d e, ...
> >>
> >> sync -av --progress /backup/ /mnt/
> >> sending incremental file list
> >> .etc_openvpn/ipp.txt
> >>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> >> .etc_openvpn/openvpn-status.log
> >>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> >> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >> ".etc_openvpn/ipp.txt": No space left on device (28)
> >> .log/
> >> .log/UcliEvt.log
> >>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> >> .log/auth.log
> >>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> >> .log/auth.log.1
> >>     19431424  61%    7.35MB/s    0:00:01
> >>
> >> the dmesg output looks like this:
> >> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7  dumping space info
> >> [  551.323694] space_info 4 has 6439526400 free, is full
> >> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> >> reserved=49152, may_use=6438453248, readonly=65536
> >>
> > 
> > Ok so then this is probably it, let me know if it helps.  Thanks,
> 
> OK it now has copied a lot of files (170) without an error all were very
> small.
>

Welp progress is good.  Throw this into the mix and go again, it's just adding
some more debugging so I can make sure I'm going down the right rabbit hole.
Thanks,

Josef 

diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 84e8909..1cf810a 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4026,6 +4026,15 @@ static int flush_space(struct btrfs_root *root,
 
 	return ret;
 }
+
+static void dump_block_rsv(struct btrfs_block_rsv *block_rsv)
+{
+	spin_lock(&block_rsv->lock);
+	printk(KERN_ERR "dumping block rsv %u, size %Lu reserved %Lu\n",
+	       block_rsv->type, block_rsv->size, block_rsv->reserved);
+	spin_unlock(&block_rsv->lock);
+}
+
 /**
  * reserve_metadata_bytes - try to reserve bytes from the block_rsv's space
  * @root - the root we're allocating for
@@ -4179,6 +4188,9 @@ out:
 			       "flush %d, flush_state %d  dumping space info\n", block_rsv->type,
 			       block_rsv->size, block_rsv->reserved, flush, flush_state);
 			spin_unlock(&block_rsv->lock);
+			dump_block_rsv(&root->fs_info->delalloc_block_rsv);
+			dump_block_rsv(&root->fs_info->delayed_block_rsv);
+			dump_block_rsv(&root->fs_info->global_block_rsv);
 			dump_space_info(space_info, 0, 0);
 		}
 
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe]

HI,


Am 26.03.2013 16:25, schrieb Josef Bacik:
> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
>> Hi,
>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>>>> no - it's just mounted with mount -o noatime
>>>>>>
>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>>>>>
>>>>>
>>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
>>>>> it?  Thanks,
>>>>
>>>> It still does not fix the problem.
>>>>
>>>> The rsync output looks like this so it does not work for file a but then
>>>> continues on c d e, ...
>>>>
>>>> sync -av --progress /backup/ /mnt/
>>>> sending incremental file list
>>>> .etc_openvpn/ipp.txt
>>>>           229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
>>>> .etc_openvpn/openvpn-status.log
>>>>           360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
>>>> .log/
>>>> .log/UcliEvt.log
>>>>        104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
>>>> .log/auth.log
>>>>      15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
>>>> .log/auth.log.1
>>>>      19431424  61%    7.35MB/s    0:00:01
>>>>
>>>> the dmesg output looks like this:
>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [  551.323694] space_info 4 has 6439526400 free, is full
>>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
>>>> reserved=49152, may_use=6438453248, readonly=65536
>>>>
>>>
>>> Ok so then this is probably it, let me know if it helps.  Thanks,
>>
>> OK it now has copied a lot of files (170) without an error all were very
>> small.
>>
>
> Welp progress is good.  Throw this into the mix and go again, it's just adding
> some more debugging so I can make sure I'm going down the right rabbit hole.
> Thanks,

Output is now:
[ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.527392] dumping block rsv 2, size 0 reserved 0
[ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
[ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.646958] space_info 4 has 6439428096 free, is full
[ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, 
reserved=45056, may_use=6438453248, readonly=65536
[ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 
2, flush_state 7  dumping space info
[ 9587.727000] dumping block rsv 2, size 0 reserved 0
[ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
[ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
[ 9587.839935] space_info 4 has 6439428096 free, is full
[ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, 
reserved=45056, may_use=6438354944, readonly=65536

Stefan

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 10:19:19AM -0600, Stefan Priebe wrote:
> HI,
> 
> 
> Am 26.03.2013 16:25, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
> >> Hi,
> >> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>>>> no - it's just mounted with mount -o noatime
> >>>>>>
> >>>>>> :~# cat /proc/mounts | grep btrfs
> >>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>>>>>
> >>>>>
> >>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
> >>>>> it?  Thanks,
> >>>>
> >>>> It still does not fix the problem.
> >>>>
> >>>> The rsync output looks like this so it does not work for file a but then
> >>>> continues on c d e, ...
> >>>>
> >>>> sync -av --progress /backup/ /mnt/
> >>>> sending incremental file list
> >>>> .etc_openvpn/ipp.txt
> >>>>           229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> >>>> .etc_openvpn/openvpn-status.log
> >>>>           360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> >>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >>>> ".etc_openvpn/ipp.txt": No space left on device (28)
> >>>> .log/
> >>>> .log/UcliEvt.log
> >>>>        104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> >>>> .log/auth.log
> >>>>      15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> >>>> .log/auth.log.1
> >>>>      19431424  61%    7.35MB/s    0:00:01
> >>>>
> >>>> the dmesg output looks like this:
> >>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> >>>> 2, flush_state 7  dumping space info
> >>>> [  551.323694] space_info 4 has 6439526400 free, is full
> >>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> >>>> reserved=49152, may_use=6438453248, readonly=65536
> >>>>
> >>>
> >>> Ok so then this is probably it, let me know if it helps.  Thanks,
> >>
> >> OK it now has copied a lot of files (170) without an error all were very
> >> small.
> >>
> >
> > Welp progress is good.  Throw this into the mix and go again, it's just adding
> > some more debugging so I can make sure I'm going down the right rabbit hole.
> > Thanks,
> 
> Output is now:
> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
> [ 9587.646958] space_info 4 has 6439428096 free, is full
> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0, 
> reserved=45056, may_use=6438453248, readonly=65536
> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush 
> 2, flush_state 7  dumping space info
> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
> [ 9587.839935] space_info 4 has 6439428096 free, is full
> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0, 
> reserved=45056, may_use=6438354944, readonly=65536
> 

Well then that looks like I was going down the wrong rabbit hole.  This should
fix you up, for real this time ;).  Thanks,

Josef


diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 1cf810a..ac415cf7 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4471,7 +4471,12 @@ static void update_global_block_rsv(struct btrfs_fs_info *fs_info)
 	spin_lock(&sinfo->lock);
 	spin_lock(&block_rsv->lock);
 
-	block_rsv->size = num_bytes;
+	/*
+	 * Limit the global block rsv to 512mb, we have infrastructure in place
+	 * to throttle reservations if we start getting low on global block rsv
+	 * space.
+	 */
+	block_rsv->size = min_t(u64, num_bytes, 512 * 1024 * 1024);
 
 	num_bytes = sinfo->bytes_used + sinfo->bytes_pinned +
 		    sinfo->bytes_reserved + sinfo->bytes_readonly +
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe]

Hi Josef,

Am 26.03.2013 18:45, schrieb Josef Bacik:
>> Am 26.03.2013 16:25, schrieb Josef Bacik:
>>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
>>>> Hi,
>>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>>>>>> no - it's just mounted with mount -o noatime
>>>>>>>>
>>>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>>>>>>>
>>>>>>>
>>>>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
>>>>>>> it?  Thanks,
>>>>>>
>>>>>> It still does not fix the problem.
>>>>>>
>>>>>> The rsync output looks like this so it does not work for file a but then
>>>>>> continues on c d e, ...
>>>>>>
>>>>>> sync -av --progress /backup/ /mnt/
>>>>>> sending incremental file list
>>>>>> .etc_openvpn/ipp.txt
>>>>>>            229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
>>>>>> .etc_openvpn/openvpn-status.log
>>>>>>            360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
>>>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>>>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
>>>>>> .log/
>>>>>> .log/UcliEvt.log
>>>>>>         104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
>>>>>> .log/auth.log
>>>>>>       15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
>>>>>> .log/auth.log.1
>>>>>>       19431424  61%    7.35MB/s    0:00:01
>>>>>>
>>>>>> the dmesg output looks like this:
>>>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
>>>>>> 2, flush_state 7  dumping space info
>>>>>> [  551.323694] space_info 4 has 6439526400 free, is full
>>>>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
>>>>>> reserved=49152, may_use=6438453248, readonly=65536
>>>>>>
>>>>>
>>>>> Ok so then this is probably it, let me know if it helps.  Thanks,
>>>>
>>>> OK it now has copied a lot of files (170) without an error all were very
>>>> small.
>>>>
>>>
>>> Welp progress is good.  Throw this into the mix and go again, it's just adding
>>> some more debugging so I can make sure I'm going down the right rabbit hole.
>>> Thanks,
>>
>> Output is now:
>> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
>> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
>> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
>> [ 9587.646958] space_info 4 has 6439428096 free, is full
>> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
>> reserved=45056, may_use=6438453248, readonly=65536
>> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
>> 2, flush_state 7  dumping space info
>> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
>> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
>> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
>> [ 9587.839935] space_info 4 has 6439428096 free, is full
>> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
>> reserved=45056, may_use=6438354944, readonly=65536
>>
>
> Well then that looks like I was going down the wrong rabbit hole.  This should
> fix you up, for real this time ;).  Thanks,

Yes - this works now. Which of the patches can i drop? Do i just need 
the last one?
Is it safe to add another 18TB raid via converting it to btrfs raid0?
Will the fix be part of 3.9-rc5?

Thanks and greets,
Stefan

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote:
> Hi Josef,
> 
> Am 26.03.2013 18:45, schrieb Josef Bacik:
> >> Am 26.03.2013 16:25, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
> >>>> Hi,
> >>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>>>>>> no - it's just mounted with mount -o noatime
> >>>>>>>>
> >>>>>>>> :~# cat /proc/mounts | grep btrfs
> >>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>>>>>>>
> >>>>>>>
> >>>>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
> >>>>>>> it?  Thanks,
> >>>>>>
> >>>>>> It still does not fix the problem.
> >>>>>>
> >>>>>> The rsync output looks like this so it does not work for file a but then
> >>>>>> continues on c d e, ...
> >>>>>>
> >>>>>> sync -av --progress /backup/ /mnt/
> >>>>>> sending incremental file list
> >>>>>> .etc_openvpn/ipp.txt
> >>>>>>            229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> >>>>>> .etc_openvpn/openvpn-status.log
> >>>>>>            360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> >>>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >>>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
> >>>>>> .log/
> >>>>>> .log/UcliEvt.log
> >>>>>>         104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> >>>>>> .log/auth.log
> >>>>>>       15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> >>>>>> .log/auth.log.1
> >>>>>>       19431424  61%    7.35MB/s    0:00:01
> >>>>>>
> >>>>>> the dmesg output looks like this:
> >>>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> >>>>>> 2, flush_state 7  dumping space info
> >>>>>> [  551.323694] space_info 4 has 6439526400 free, is full
> >>>>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> >>>>>> reserved=49152, may_use=6438453248, readonly=65536
> >>>>>>
> >>>>>
> >>>>> Ok so then this is probably it, let me know if it helps.  Thanks,
> >>>>
> >>>> OK it now has copied a lot of files (170) without an error all were very
> >>>> small.
> >>>>
> >>>
> >>> Welp progress is good.  Throw this into the mix and go again, it's just adding
> >>> some more debugging so I can make sure I'm going down the right rabbit hole.
> >>> Thanks,
> >>
> >> Output is now:
> >> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
> >> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
> >> [ 9587.646958] space_info 4 has 6439428096 free, is full
> >> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
> >> reserved=45056, may_use=6438453248, readonly=65536
> >> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
> >> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
> >> [ 9587.839935] space_info 4 has 6439428096 free, is full
> >> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
> >> reserved=45056, may_use=6438354944, readonly=65536
> >>
> >
> > Well then that looks like I was going down the wrong rabbit hole.  This should
> > fix you up, for real this time ;).  Thanks,
> 
> Yes - this works now. Which of the patches can i drop? Do i just need 
> the last one?
> Is it safe to add another 18TB raid via converting it to btrfs raid0?
> Will the fix be part of 3.9-rc5?
> 

So I'll put together all of the patches that actually need to go up for this and
post them, but basically its the mutex patch, the last patch I sent you and the
one that adjusts the reservations for rename and delete.  Thanks,

Josef

Re: No space left on device (28)

[Stefan Priebe]

Hi,

but when i transfer big files i see now this one:
[20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
[20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
disables this message.
[20368.895140] rsync           D ffffffff8160f580     0 14911      1 
0x00000000
[20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 
ffff88022394f800
[20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 
0000000000012c40
[20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 
ffff8801cefce130
[20368.895169] Call Trace:
[20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
[20368.895207]  [<ffffffffa0158c75>] 
wait_current_trans.isra.32+0x95/0x100 [btrfs]
[20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
[20368.895236]  [<ffffffffa015a45d>] 
start_transaction.part.33+0x13d/0x4d0 [btrfs]
[20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
[20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
[20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 
[btrfs]
[20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
[20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
[20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
[20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
[20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
[20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
[20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
[20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
[20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
[20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b

Speed is just 100kb/s instead of 100MB/s.

Stefan

Am 26.03.2013 20:16, schrieb Josef Bacik:
> On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote:
>> Hi Josef,
>>
>> Am 26.03.2013 18:45, schrieb Josef Bacik:
>>>> Am 26.03.2013 16:25, schrieb Josef Bacik:
>>>>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
>>>>>> Hi,
>>>>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
>>>>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
>>>>>>>>>> no - it's just mounted with mount -o noatime
>>>>>>>>>>
>>>>>>>>>> :~# cat /proc/mounts | grep btrfs
>>>>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
>>>>>>>>> it?  Thanks,
>>>>>>>>
>>>>>>>> It still does not fix the problem.
>>>>>>>>
>>>>>>>> The rsync output looks like this so it does not work for file a but then
>>>>>>>> continues on c d e, ...
>>>>>>>>
>>>>>>>> sync -av --progress /backup/ /mnt/
>>>>>>>> sending incremental file list
>>>>>>>> .etc_openvpn/ipp.txt
>>>>>>>>             229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
>>>>>>>> .etc_openvpn/openvpn-status.log
>>>>>>>>             360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
>>>>>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
>>>>>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
>>>>>>>> .log/
>>>>>>>> .log/UcliEvt.log
>>>>>>>>          104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
>>>>>>>> .log/auth.log
>>>>>>>>        15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
>>>>>>>> .log/auth.log.1
>>>>>>>>        19431424  61%    7.35MB/s    0:00:01
>>>>>>>>
>>>>>>>> the dmesg output looks like this:
>>>>>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
>>>>>>>> 2, flush_state 7  dumping space info
>>>>>>>> [  551.323694] space_info 4 has 6439526400 free, is full
>>>>>>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
>>>>>>>> reserved=49152, may_use=6438453248, readonly=65536
>>>>>>>>
>>>>>>>
>>>>>>> Ok so then this is probably it, let me know if it helps.  Thanks,
>>>>>>
>>>>>> OK it now has copied a lot of files (170) without an error all were very
>>>>>> small.
>>>>>>
>>>>>
>>>>> Welp progress is good.  Throw this into the mix and go again, it's just adding
>>>>> some more debugging so I can make sure I'm going down the right rabbit hole.
>>>>> Thanks,
>>>>
>>>> Output is now:
>>>> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
>>>> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
>>>> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
>>>> [ 9587.646958] space_info 4 has 6439428096 free, is full
>>>> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
>>>> reserved=45056, may_use=6438453248, readonly=65536
>>>> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
>>>> 2, flush_state 7  dumping space info
>>>> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
>>>> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
>>>> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
>>>> [ 9587.839935] space_info 4 has 6439428096 free, is full
>>>> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
>>>> reserved=45056, may_use=6438354944, readonly=65536
>>>>
>>>
>>> Well then that looks like I was going down the wrong rabbit hole.  This should
>>> fix you up, for real this time ;).  Thanks,
>>
>> Yes - this works now. Which of the patches can i drop? Do i just need
>> the last one?
>> Is it safe to add another 18TB raid via converting it to btrfs raid0?
>> Will the fix be part of 3.9-rc5?
>>
>
> So I'll put together all of the patches that actually need to go up for this and
> post them, but basically its the mutex patch, the last patch I sent you and the
> one that adjusts the reservations for rename and delete.  Thanks,
>
> Josef
>

Re: No space left on device (28)

[Josef Bacik]

On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe wrote:
> Hi,
> 
> but when i transfer big files i see now this one:
> [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
> [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" 
> disables this message.
> [20368.895140] rsync           D ffffffff8160f580     0 14911      1 
> 0x00000000
> [20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198 
> ffff88022394f800
> [20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8 
> 0000000000012c40
> [20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88 
> ffff8801cefce130
> [20368.895169] Call Trace:
> [20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
> [20368.895207]  [<ffffffffa0158c75>] 
> wait_current_trans.isra.32+0x95/0x100 [btrfs]
> [20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
> [20368.895236]  [<ffffffffa015a45d>] 
> start_transaction.part.33+0x13d/0x4d0 [btrfs]
> [20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
> [20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
> [20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20 
> [btrfs]
> [20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
> [20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
> [20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
> [20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
> [20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
> [20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
> [20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
> [20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
> [20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
> [20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
> 
> Speed is just 100kb/s instead of 100MB/s.
> 

Hrm I wonder if 512 is too small for your case, can you add this patch to the
pile and see what dmesg says when you are having these problems?  Thanks,

Josef


diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 50767bb..d19c9f6 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -31,6 +31,7 @@
 #include "inode-map.h"
 #include "volumes.h"
 #include "dev-replace.h"
+#include "math.h"
 
 #define BTRFS_ROOT_TRANS_TAG 0
 
@@ -576,10 +577,19 @@ void btrfs_throttle(struct btrfs_root *root)
 static int should_end_transaction(struct btrfs_trans_handle *trans,
 				  struct btrfs_root *root)
 {
-	int ret;
+	struct btrfs_block_rsv *block_rsv = &root->fs_info->global_block_rsv;
+	u64 num_bytes = 0;
+	int ret = 1;
 
-	ret = btrfs_block_rsv_check(root, &root->fs_info->global_block_rsv, 5);
-	return ret ? 1 : 0;
+	spin_lock(&block_rsv->lock);
+	num_bytes = div_factor(block_rsv->size, 5);
+	if (block_rsv->reserved >= num_bytes)
+		ret = 0;
+	else
+		printk(KERN_ERR "we're pretty low, setting blocked, reserved %Lu, size %Lu, num %Lu\n",
+		       block_rsv->reserved, block_rsv->size, num_bytes);
+	spin_unlock(&block_rsv->lock);
+	return ret;
 }
 
 int btrfs_should_end_transaction(struct btrfs_trans_handle *trans,
-- 
1.7.7.6

Re: No space left on device (28)

[Stefan Priebe]

HI,
Am 26.03.2013 20:38, schrieb Josef Bacik:
> On Tue, Mar 26, 2013 at 01:22:20PM -0600, Stefan Priebe wrote:
>> Hi,
>>
>> but when i transfer big files i see now this one:
>> [20368.784736] INFO: task rsync:14911 blocked for more than 120 seconds.
>> [20368.821978] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [20368.895140] rsync           D ffffffff8160f580     0 14911      1
>> 0x00000000
>> [20368.895148]  ffff8801ca63fc78 0000000000000086 ffff8800c28f8198
>> ffff88022394f800
>> [20368.895158]  ffff8801ca63ffd8 ffff8801ca63ffd8 ffff8801ca63ffd8
>> 0000000000012c40
>> [20368.895163]  ffffffff81a11440 ffff8801c9d36340 ffff8801ca63fc88
>> ffff8801cefce130
>> [20368.895169] Call Trace:
>> [20368.895180]  [<ffffffff8151a774>] schedule+0x24/0x70
>> [20368.895207]  [<ffffffffa0158c75>]
>> wait_current_trans.isra.32+0x95/0x100 [btrfs]
>> [20368.895214]  [<ffffffff8106d4f0>] ? add_wait_queue+0x60/0x60
>> [20368.895236]  [<ffffffffa015a45d>]
>> start_transaction.part.33+0x13d/0x4d0 [btrfs]
>> [20368.895252]  [<ffffffff811420f3>] ? inode_permission+0x13/0x50
>> [20368.895271]  [<ffffffffa015a814>] start_transaction+0x24/0x30 [btrfs]
>> [20368.895287]  [<ffffffffa015aae3>] btrfs_start_transaction+0x13/0x20
>> [btrfs]
>> [20368.895302]  [<ffffffffa015b2f0>] __unlink_start_trans+0x70/0x460 [btrfs]
>> [20368.895307]  [<ffffffff8150ee3e>] ? check_acl+0x5a/0x122
>> [20368.895312]  [<ffffffff81055ff0>] ? ns_capable+0x30/0x60
>> [20368.895317]  [<ffffffff811413bd>] ? generic_permission+0xbd/0x110
>> [20368.895336]  [<ffffffffa0163f92>] btrfs_unlink+0x32/0xc0 [btrfs]
>> [20368.895341]  [<ffffffff8114186d>] vfs_unlink.part.61+0x6d/0xd0
>> [20368.895345]  [<ffffffff81143ad7>] vfs_unlink+0x37/0x50
>> [20368.895349]  [<ffffffff81143c8b>] do_unlinkat+0x19b/0x240
>> [20368.895354]  [<ffffffff81146171>] sys_unlink+0x11/0x20
>> [20368.895359]  [<ffffffff8151c2e9>] system_call_fastpath+0x16/0x1b
>>
>> Speed is just 100kb/s instead of 100MB/s.
>>
>
> Hrm I wonder if 512 is too small for your case, can you add this patch to the
> pile and see what dmesg says when you are having these problems?  Thanks,

No idea why but i can't reproduce... ;-(

Stefan

Re: No space left on device (28)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 2529 bytes --]

On Thu, 21 Mar 2013 20:42:28 +0100
Stefan Priebe <s.priebe@profihost.ag> wrote:

I might be wrong here, but doesn't this

> rsync: rename 
> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" 
> -> 
> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h": 

...try to move a file from 

  "/mnt/.software/"

to

  ".software/"

(relative to current dir)??

Maybe you end up trying to rsync your array to a small root partition or
something?

> No space left on device (28)
> 
> # btrfs filesystem df /mnt/
> Data: total=18.14TB, used=15.05TB
> System, DUP: total=8.00MB, used=1.94MB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=23.98GB, used=17.98GB
> Metadata: total=8.00MB, used=0.00
> 
> Stefan
> 
> Am 21.03.2013 20:02, schrieb Chris Mason:
> > Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
> >> Hi Chris,
> >>
> >> Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
> >>
> >>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
> >>>> Hi,
> >>>>
> >>>> while copying a 15GB file to my btrfs volume i'm getting "No space left
> >>>> on device".
> >>>>
> >>>> Some information:
> >>>>
> >>>> # btrfs filesystem df /mnt/
> >>>> Data: total=18.14TB, used=15.05TB
> >>>> System, DUP: total=8.00MB, used=1.94MB
> >>>> System: total=4.00MB, used=0.00
> >>>> Metadata, DUP: total=23.98GB, used=17.97GB
> >>>> Metadata: total=8.00MB, used=0.00
> >>>>
> >>>> # df -h
> >>>> Filesystem             Size  Used Avail Use% Mounted on
> >>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
> >>>>
> >>>> # btrfs fi show
> >>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
> >>>>         Total devices 1 FS bytes used 15.07TB
> >>>>         devid    1 size 18.19TB used 18.19TB path /dev/dm-3
> >>>>
> >>>> # btrfs fi balance start -dusage=5 /mnt/
> >>>> Done, had to relocate 0 out of 18604 chunks
> >>>>
> >>>> What wrong here? What is about the last 3TB?
> >>>
> >>> Which kernel are you running?
> >>>
> >>> -chris
> >>
> >> vanilla 3.8.3.
> >
> > Ok, with the 3.9 merge window Josef changed how we do the reservations.
> > Are you able to try a slightly more experimental kernel?
> >
> > -chris
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: No space left on device (28)

[cwillu]

On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov <rm@romanrm.ru> wrote:
> On Thu, 21 Mar 2013 20:42:28 +0100
> Stefan Priebe <s.priebe@profihost.ag> wrote:
>
> I might be wrong here, but doesn't this
>
>> rsync: rename
>> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>> ->
>> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>
> ...try to move a file from
>
>   "/mnt/.software/"
>
> to
>
>   ".software/"
>
> (relative to current dir)??

No; that's rsync giving the full path, and then the target path
relative to the command it was given.  The filename itself
(".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to
temporarily, so it can mv it over the original in an atomic fashion...

Stefan: ...which means that the actual copy succeeded, which suggests
that this is more of a metadata enospc thing.

You might try btrfs balance start -musage=5 (instead of -dusage), and
if that doesn't report any chunks balanced, try a high number until it
does.

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%

Am 22.03.2013 um 07:24 schrieb cwillu <cwillu@cwillu.com>:

> On Fri, Mar 22, 2013 at 12:13 AM, Roman Mamedov <rm@romanrm.ru> wrote:
>> On Thu, 21 Mar 2013 20:42:28 +0100
>> Stefan Priebe <s.priebe@profihost.ag> wrote:
>> 
>> I might be wrong here, but doesn't this
>> 
>>> rsync: rename
>>> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/""
>>> ->
>>> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
>> 
>> ...try to move a file from
>> 
>>  "/mnt/.software/"
>> 
>> to
>> 
>>  ".software/"
>> 
>> (relative to current dir)??
> 
> No; that's rsync giving the full path, and then the target path
> relative to the command it was given.  The filename itself
> (".c2_ae.h.WEhLGP") is a semi-random filename rsync uses to write to
> temporarily, so it can mv it over the original in an atomic fashion...
> 
> Stefan: ...which means that the actual copy succeeded, which suggests
> that this is more of a metadata enospc thing.
> 
> You might try btrfs balance start -musage=5 (instead of -dusage), and
> if that doesn't report any chunks balanced, try a high number until it
> does.

Re: No space left on device (28)

[cwillu]

On Fri, Mar 22, 2013 at 12:39 AM, Stefan Priebe - Profihost AG
<s.priebe@profihost.ag> wrote:
> Already tried with value 5 did not help ;-( and it also happens with plain cp copying a 15gb file and aborts at about 80%

You tried -musage=5?  Your original email said -dusage=5.

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

Hi,

Am 22.03.2013 07:41, schrieb cwillu:> On Fri, Mar 22, 2013 at 12:39 AM,
Stefan Priebe - Profihost AG
> <s.priebe@profihost.ag> wrote:
>> Already tried with value 5 did not help ;-( and it also happens with
>> plain cp copying a 15gb file and aborts at about 80%
>
> You tried -musage=5?  Your original email said -dusage=5.

sorry i missed the difference - but it does not work:

~# btrfs fi balance start -musage=5 /mnt/
ERROR: error during balancing '/mnt/' - No space left on device
There may be more info in syslog - try dmesg | tail

~# dmesg|tail
[ 1183.019367] device fsid fc23c4a8-a351-4c91-96a2-ee6abeffe59a devid 1
transid 6224 /dev/mapper/raid54tb1
[ 1183.019860] btrfs: disk space caching is enabled
[42719.915781] btrfs: relocating block group 4194304 flags 4
[42727.916141] btrfs: 5 enospc errors during balance

Stefan

Re: No space left on device (28)

[Stefan Priebe - Profihost AG]

No this is fine it's rsync style same happens with cp.

Am 22.03.2013 um 07:13 schrieb Roman Mamedov <rm@romanrm.ru>:

> On Thu, 21 Mar 2013 20:42:28 +0100
> Stefan Priebe <s.priebe@profihost.ag> wrote:
> 
> I might be wrong here, but doesn't this
> 
>> rsync: rename 
>> "/mnt/.software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/.c2_ae.h.WEhLGP" 
>> -> 
>> ".software/kernel/linux-3.9-rc3/drivers/infiniband/hw/amso1100/c2_ae.h":
> 
> ...try to move a file from 
> 
>  "/mnt/.software/"
> 
> to
> 
>  ".software/"
> 
> (relative to current dir)??
> 
> Maybe you end up trying to rsync your array to a small root partition or
> something?
> 
>> No space left on device (28)
>> 
>> # btrfs filesystem df /mnt/
>> Data: total=18.14TB, used=15.05TB
>> System, DUP: total=8.00MB, used=1.94MB
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=23.98GB, used=17.98GB
>> Metadata: total=8.00MB, used=0.00
>> 
>> Stefan
>> 
>> Am 21.03.2013 20:02, schrieb Chris Mason:
>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 14:35:59)
>>>> Hi Chris,
>>>> 
>>>> Am 21.03.2013 um 19:00 schrieb Chris Mason <chris.mason@fusionio.com>:
>>>> 
>>>>> Quoting Stefan Priebe - Profihost AG (2013-03-21 04:03:04)
>>>>>> Hi,
>>>>>> 
>>>>>> while copying a 15GB file to my btrfs volume i'm getting "No space left
>>>>>> on device".
>>>>>> 
>>>>>> Some information:
>>>>>> 
>>>>>> # btrfs filesystem df /mnt/
>>>>>> Data: total=18.14TB, used=15.05TB
>>>>>> System, DUP: total=8.00MB, used=1.94MB
>>>>>> System: total=4.00MB, used=0.00
>>>>>> Metadata, DUP: total=23.98GB, used=17.97GB
>>>>>> Metadata: total=8.00MB, used=0.00
>>>>>> 
>>>>>> # df -h
>>>>>> Filesystem             Size  Used Avail Use% Mounted on
>>>>>> /dev/mapper/raid54tb1   19T   16T  3,1T  84% /mnt
>>>>>> 
>>>>>> # btrfs fi show
>>>>>> Label: none  uuid: fc23c4a8-a351-4c91-96a2-ee6abeffe59a
>>>>>>        Total devices 1 FS bytes used 15.07TB
>>>>>>        devid    1 size 18.19TB used 18.19TB path /dev/dm-3
>>>>>> 
>>>>>> # btrfs fi balance start -dusage=5 /mnt/
>>>>>> Done, had to relocate 0 out of 18604 chunks
>>>>>> 
>>>>>> What wrong here? What is about the last 3TB?
>>>>> 
>>>>> Which kernel are you running?
>>>>> 
>>>>> -chris
>>>> 
>>>> vanilla 3.8.3.
>>> 
>>> Ok, with the 3.9 merge window Josef changed how we do the reservations.
>>> Are you able to try a slightly more experimental kernel?
>>> 
>>> -chris
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> -- 
> With respect,
> Roman