Re: 800 GByte free, but "no space left"

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Evert,

Du meintest am 04.12.10 zum Thema Re: 800 GByte free, but "no space lef=
t":

> On Sat, Dec 4, 2010 at 10:17 AM, Helmut Hullen <Hullen@t-online.de>
> wrote:
>> Hallo,
>>
>> I wrote am 02.12.10:
>>
>>> I use 2 disks (1.5 Tbyte and 2.0 TByte) under 1 LABEL (for my video
>>> collection, nearly alle files have more than 1 GByte):
>>
>>> Label: MM2 =A0uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
>>> =A0 =A0 =A0 Total devices 2 FS bytes used 2.38TB
>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc3
>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf2
>>
>>> ("btrfs-show" uses TiByte, it's 10% less than TByte)
>>
>>> Btrfs Btrfs v0.19
>>
>>> Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available =
Use% Mounted on
>>> /dev/sdc3 =A0 =A0 =A0 =A0 =A0 =A03400799848 2559596740 841203108 =A0=
76% /srv/MM
>>
>>> --------------------------------
>>
>>> When I add some more videos, writing gets slower and slower, and
>>> then the system refuses with "no space left ..."
>>
>> [...]

>> No help?

> I am not an expert on this by a long shot, but it looks like you
> added these two disks in raid0.

> This means that the total space cannot exceed the space of the
> smallest disk.

> ie: 1.35TB is the max you can use on any of your disks, as that is
> the size of the smallest disk. In other words, once any of the disks
> in a btrfs array runs out of space, the whole array is out of space.

> I don't know if this is intended, but it certainly would appear so.

I won't hope that this error is related to RAID0, I haven't installed =20
(as far as I know) RAID0.

My installation way:

(2-TByte-Disk)

        mkfs.btrfs /dev/sdf2
        mount /dev/sdf2 /srv/MM

(1.5-TByte-Disk)
        btrfs device add /dev/sdc3 /srv/MM
        btrfs filesystem balance /srv/MM

(and then waiting about 1 day ...)
Especially: no RAID definition.

If the smallest device defines the capacity then I should use 2*1.35 =20
TiByte, but my system tells "no space left" at about 2.4 TiByte - where=
 =20
are (at least) 300 GiByte hidden?

---------------------------------------

Kernel 2.6.35.8
btrfs-git from 20101117

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[cwillu]

On Sun, Dec 5, 2010 at 1:48 AM, Helmut Hullen <Hullen@t-online.de> wrot=
e:
> Hallo, Evert,
>
> Du meintest am 04.12.10 zum Thema Re: 800 GByte free, but "no space l=
eft":
>
>> On Sat, Dec 4, 2010 at 10:17 AM, Helmut Hullen <Hullen@t-online.de>
>> wrote:
>>> Hallo,
>>>
>>> I wrote am 02.12.10:
>>>
>>>> I use 2 disks (1.5 Tbyte and 2.0 TByte) under 1 LABEL (for my vide=
o
>>>> collection, nearly alle files have more than 1 GByte):
>>>
>>>> Label: MM2 =A0uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
>>>> =A0 =A0 =A0 Total devices 2 FS bytes used 2.38TB
>>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc3
>>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf2
>>>
>>>> ("btrfs-show" uses TiByte, it's 10% less than TByte)
>>>
>>>> Btrfs Btrfs v0.19
>>>
>>>> Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available=
 Use% Mounted on
>>>> /dev/sdc3 =A0 =A0 =A0 =A0 =A0 =A03400799848 2559596740 841203108 =A0=
76% /srv/MM
>>>
>>>> --------------------------------
>>>
>>>> When I add some more videos, writing gets slower and slower, and
>>>> then the system refuses with "no space left ..."
>>>
>>> [...]
>
>>> No help?

The weekend isn't the best time for

>> I am not an expert on this by a long shot, but it looks like you
>> added these two disks in raid0.
>
>> This means that the total space cannot exceed the space of the
>> smallest disk.
>
>> ie: 1.35TB is the max you can use on any of your disks, as that is
>> the size of the smallest disk. In other words, once any of the disks
>> in a btrfs array runs out of space, the whole array is out of space.
>
>> I don't know if this is intended, but it certainly would appear so.
>
> I won't hope that this error is related to RAID0, I haven't installed
> (as far as I know) RAID0.
>
> My installation way:
>
> (2-TByte-Disk)
>
> =A0 =A0 =A0 =A0mkfs.btrfs /dev/sdf2
> =A0 =A0 =A0 =A0mount /dev/sdf2 /srv/MM
>
> (1.5-TByte-Disk)
> =A0 =A0 =A0 =A0btrfs device add /dev/sdc3 /srv/MM
> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>
> (and then waiting about 1 day ...)
> Especially: no RAID definition.
>
> If the smallest device defines the capacity then I should use 2*1.35
> TiByte, but my system tells "no space left" at about 2.4 TiByte - whe=
re
> are (at least) 300 GiByte hidden?
>
> ---------------------------------------
>
> Kernel 2.6.35.8
> btrfs-git from 20101117
>
> Viele Gruesse!
> Helmut
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>

If it's not a raid1, and there's multiple devices, it's a raid0 (and
so available space is the sum of all drives).  Your problem however is
that metadata is raid1 by default (where everything is duplicated on
separate drives). One device has no space unallocated to any block
groups, and so if a particular metadata block group is full, there's
no place for that device's copy to end up, hence ENOSPC.

Adding another device will probably work around this, as will simply
running a balance operation (possibly, and you may need to free up
some space first anyway).
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, cwillu,

Du meintest am 05.12.10:

>>> I am not an expert on this by a long shot, but it looks like you
>>> added these two disks in raid0.

>> I won't hope that this error is related to RAID0, I haven't
>> installed (as far as I know) RAID0.
>>
>> My installation way:
>>
>> (2-TByte-Disk)
>>
>> =A0 =A0 =A0 =A0mkfs.btrfs /dev/sdf2
>> =A0 =A0 =A0 =A0mount /dev/sdf2 /srv/MM
>>
>> (1.5-TByte-Disk)
>> =A0 =A0 =A0 =A0btrfs device add /dev/sdc3 /srv/MM
>> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>>
>> (and then waiting about 1 day ...)
>> Especially: no RAID definition.

[...]

> If it's not a raid1, and there's multiple devices, it's a raid0 (and
> so available space is the sum of all drives).  Your problem however
> is that metadata is raid1 by default (where everything is duplicated
> on separate drives).

Maybe you're right. But if you're right then I have got the worst of tw=
o =20
worlds. I don't want neither RAID0 nor RAID1, I want a bundle of =20
different disks (at least partititions) which seem to be one large disk=
=2E =20
And I've hoped btrfs does this job.

> Adding another device will probably work around this, as will simply
> running a balance operation (possibly, and you may need to free up
> some space first anyway).

That could lead to the following steps:

Buy a 3 GByte disk =A0 =A0 =A0 =A0

        btrfs device add /dev/sdxy /srv/MM
   =A0 =A0 =A0btrfs filesystem balance /srv/MM

1.5 TByte disk:
        btrfs device delete /dev/sdc3 /srv/MM
   =A0 =A0 =A0btrfs filesystem balance /srv/MM

and then disconnect the 1.5 TByte disk (and hope that now the 2 TByte =20
disk sets the limits).
No nice way ...

--------------------------

Is there a way to avoid this (presumably) RAID mismatch?

By the way: working with TByte disks includes (for home users) that =20
there's no backup ...

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[cwillu]

On Sun, Dec 5, 2010 at 3:51 AM, Helmut Hullen <Hullen@t-online.de> wrot=
e:
> Hallo, cwillu,
>
> Du meintest am 05.12.10:
>
>>>> I am not an expert on this by a long shot, but it looks like you
>>>> added these two disks in raid0.
>
>>> I won't hope that this error is related to RAID0, I haven't
>>> installed (as far as I know) RAID0.
>>>
>>> My installation way:
>>>
>>> (2-TByte-Disk)
>>>
>>> =A0 =A0 =A0 =A0mkfs.btrfs /dev/sdf2
>>> =A0 =A0 =A0 =A0mount /dev/sdf2 /srv/MM
>>>
>>> (1.5-TByte-Disk)
>>> =A0 =A0 =A0 =A0btrfs device add /dev/sdc3 /srv/MM
>>> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>>>
>>> (and then waiting about 1 day ...)
>>> Especially: no RAID definition.
>
> [...]
>
>> If it's not a raid1, and there's multiple devices, it's a raid0 (and
>> so available space is the sum of all drives). =A0Your problem howeve=
r
>> is that metadata is raid1 by default (where everything is duplicated
>> on separate drives).
>
> Maybe you're right. But if you're right then I have got the worst of =
two
> worlds. I don't want neither RAID0 nor RAID1, I want a bundle of
> different disks (at least partititions) which seem to be one large di=
sk.
> And I've hoped btrfs does this job.

That's what raid0 is, and it's actually the best of both worlds:  your
metadata (which will be less than 5% of the data) is safely
duplicated, such that you always have the checksums even with a disk
gone, so you can verify that the data that you still have is good,
while not wasting space duplicating every little bit of file data,
which you may not care about that much, and which you have backed up
anyway (right? right?).

Larger files will be striped though, which is more incidental to how
data is allocated to block groups, but even so, it's generally not as
simple as saying "I don't want raid0, I just want a single big disk";
you have to figure out how to do that, and it'll generally result in
some raid0-like properties.

This will almost certainly become much more tunable in the future, but
not every feature that people want is done yet.  In fact, most of the
really cool user-visible features aren't done yet.  Btrfs is still
pretty young.

>> Adding another device will probably work around this, as will simply
>> running a balance operation (possibly, and you may need to free up
>> some space first anyway).
>
> That could lead to the following steps:
>
> Buy a 3 GByte disk
>
> =A0 =A0 =A0 =A0btrfs device add /dev/sdxy /srv/MM
> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>
> 1.5 TByte disk:
> =A0 =A0 =A0 =A0btrfs device delete /dev/sdc3 /srv/MM
> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>
> and then disconnect the 1.5 TByte disk (and hope that now the 2 TByte
> disk sets the limits).
> No nice way ...

No, just run the balance without adding another disk.  That will
probably work (although it _will_ take a while on a large filesystem).

I'm not sure that you understand how this all works though;  you might
want to re-read the wiki articles (which I believe have been freshened
up recently).

> Is there a way to avoid this (presumably) RAID mismatch?

Yes, you can specify the raid level for each when you make the
filesystem (and will eventually be able to do it with existing
filesystems).  However, as I described above, you really want metadata
to be duplicated.  Your problem is more of an unfortunate case of
everything not being tuned quite right yet.

> By the way: working with TByte disks includes (for home users) that
> there's no backup ...

Not sure why you'd think that.  It can't be the bandwidth, and if you
can't afford a second drive, there's a good case to be made that you
can't afford the data you can't afford to lose.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Evert Vorster]

On Sun, Dec 5, 2010 at 12:48 AM, Helmut Hullen <Hullen@t-online.de> wro=
te:
> Hallo, Evert,
>
> Du meintest am 04.12.10 zum Thema Re: 800 GByte free, but "no space l=
eft":
>
>> On Sat, Dec 4, 2010 at 10:17 AM, Helmut Hullen <Hullen@t-online.de>
>> wrote:
>>> Hallo,
>>>
>>> I wrote am 02.12.10:
>>>
>>>> I use 2 disks (1.5 Tbyte and 2.0 TByte) under 1 LABEL (for my vide=
o
>>>> collection, nearly alle files have more than 1 GByte):
>>>
>>>> Label: MM2 =A0uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
>>>> =A0 =A0 =A0 Total devices 2 FS bytes used 2.38TB
>>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc3
>>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf2
>>>
>>>> ("btrfs-show" uses TiByte, it's 10% less than TByte)
>>>
>>>> Btrfs Btrfs v0.19
>>>
>>>> Filesystem =A0 =A0 =A0 =A0 =A0 1K-blocks =A0 =A0 =A0Used Available=
 Use% Mounted on
>>>> /dev/sdc3 =A0 =A0 =A0 =A0 =A0 =A03400799848 2559596740 841203108 =A0=
76% /srv/MM
>>>
>>>> --------------------------------
>>>
>>>> When I add some more videos, writing gets slower and slower, and
>>>> then the system refuses with "no space left ..."
>>>
>>> [...]
>
>>> No help?
>
>> I am not an expert on this by a long shot, but it looks like you
>> added these two disks in raid0.
>
>> This means that the total space cannot exceed the space of the
>> smallest disk.
>
>> ie: 1.35TB is the max you can use on any of your disks, as that is
>> the size of the smallest disk. In other words, once any of the disks
>> in a btrfs array runs out of space, the whole array is out of space.
>
>> I don't know if this is intended, but it certainly would appear so.
>
> I won't hope that this error is related to RAID0, I haven't installed
> (as far as I know) RAID0.
>
> My installation way:
>
> (2-TByte-Disk)
>
> =A0 =A0 =A0 =A0mkfs.btrfs /dev/sdf2
> =A0 =A0 =A0 =A0mount /dev/sdf2 /srv/MM
>
> (1.5-TByte-Disk)
> =A0 =A0 =A0 =A0btrfs device add /dev/sdc3 /srv/MM
> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>
> (and then waiting about 1 day ...)
> Especially: no RAID definition.
>
> If the smallest device defines the capacity then I should use 2*1.35
> TiByte, but my system tells "no space left" at about 2.4 TiByte - whe=
re
> are (at least) 300 GiByte hidden?

>>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
>>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2

Here devid 2 is at 100%, and hence you are getting the no more space
left errors. So, the 300 TB is on the bigger disk, and not usable for
you right now.

I know of the disk mode you speak.. an old raid card of mine called it
"Just a bunch of disks" and it literally filled up the first disk
before carrying on to the second one until that was full under
windows... under UNIX it had the effect of just adding all the sectors
to each other, and stretching the file system over the disks in a
linear fashion. Most UNIX file systems writes files in the middle of
the largest contiguous free space, which meant that some files got
written on the first disk, and some on the second. As far as I know,
btrfs does not support this raid mode.

Another thing to keep in mind is that as far as I know you cannot
remove devid 1 from a btrfs volume. This is due to be fixed, but I
have no idea on the status of that.

Lastly, external USB disks are not too expensive, and together with
rsync makes a good off-line backup solution.

You could, if you really wanted to use all of two differently sized
disks in a btrfs, subdivide the disks in equal sized partitions, and
just put all of those partitions in a btrfs raid0...

ie:

Say you have a 1TB disk and 2TB disk... make 1TB partition on first
disk, and two 1TB partitions on second disk, then add all three
partitions to btrfs raid0 to make one volume of 3TB which will be
fully usable.

In your case:
1.81 Tb and 1.3 Tb your best use of space would probably be 0.6Tb
partitions.ie: 3x0.6Tb partitions on the first disk, and 2x0.6TB
partitions on the second. Then add all five partitions to a btrfs
raid. This would leave 0.1Tb of wasted space on the smaller device,
but at least you can use this partition separately.

Kind regards,
-Evert Vorster-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2976 bytes --]

On Sun, Dec 05, 2010 at 04:08:26AM -0700, Evert Vorster wrote:
> On Sun, Dec 5, 2010 at 12:48 AM, Helmut Hullen <Hullen@t-online.de> wrote:
> > Hallo, Evert,
> >
> > Du meintest am 04.12.10 zum Thema Re: 800 GByte free, but "no space left":
> >
> >> I am not an expert on this by a long shot, but it looks like you
> >> added these two disks in raid0.

   Nope -- btrfs will spread out its allocations across both disks.

> >> This means that the total space cannot exceed the space of the
> >> smallest disk.
[...]
> > Especially: no RAID definition.
> >
> > If the smallest device defines the capacity then I should use 2*1.35
> > TiByte, but my system tells "no space left" at about 2.4 TiByte - where
> > are (at least) 300 GiByte hidden?
> 
> >>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
> >>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2
> 
> Here devid 2 is at 100%, and hence you are getting the no more space
> left errors. So, the 300 TB is on the bigger disk, and not usable for
> you right now.

   I _think_ that a balance is all that's needed at this point. It
can't hurt, anyway (other than taking quite a long time).

> I know of the disk mode you speak.. an old raid card of mine called it
> "Just a bunch of disks" and it literally filled up the first disk
> before carrying on to the second one until that was full under
> windows... under UNIX it had the effect of just adding all the sectors
> to each other, and stretching the file system over the disks in a
> linear fashion. Most UNIX file systems writes files in the middle of
> the largest contiguous free space, which meant that some files got
> written on the first disk, and some on the second. As far as I know,
> btrfs does not support this raid mode.

   It does support it: that's what the "single" RAID profile in
mkfs.btrfs is. It attempts to use the disk space marginally more
intelligently than traditional linear mode, though, as it allocates
block groups (in chunks of about 1G) to each disk in turn. This isn't
the same as RAID-0, which stripes within block groups with a much
smaller stripe size.

> Another thing to keep in mind is that as far as I know you cannot
> remove devid 1 from a btrfs volume. This is due to be fixed, but I
> have no idea on the status of that.

   I've done it (I have a filesystem with IDs 7, 8, 9, 12, 13, 14).
Looks like that particular problem has been fixed.

> You could, if you really wanted to use all of two differently sized
> disks in a btrfs, subdivide the disks in equal sized partitions, and
> just put all of those partitions in a btrfs raid0...
[...]

   That would be a really bad idea, as your disks would thrash
horribly, reading stripes from different locations on the disk.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
               --- Nostalgia isn't what it used to be. ---               

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Evert,

Du meintest am 05.12.10:

> You could, if you really wanted to use all of two differently sized
> disks in a btrfs, subdivide the disks in equal sized partitions, and
> just put all of those partitions in a btrfs raid0...

> ie:

> Say you have a 1TB disk and 2TB disk... make 1TB partition on first
> disk, and two 1TB partitions on second disk, then add all three
> partitions to btrfs raid0 to make one volume of 3TB which will be
> fully usable.

I'll think about it - it's anyway time for a new disk with 2 or 3 TByte  
...

And (if your way is the best) partitioning all new disks into 1-TByte-  
partitions. Strange ... remembers me to the invention of partitioning a  
hard disk (because the size of the whole disk was to big for the old  
fashionened FAT).

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, cwillu,

Du meintest am 05.12.10:

>> Maybe you're right. But if you're right then I have got the worst of
>> two worlds. I don't want neither RAID0 nor RAID1, I want a bundle of
>> different disks (at least partititions) which seem to be one large
>> disk. And I've hoped btrfs does this job.

> That's what raid0 is, and it's actually the best of both worlds:
> your metadata (which will be less than 5% of the data) is safely
> duplicated, such that you always have the checksums even with a disk
> gone, so you can verify that the data that you still have is good,
> while not wasting space duplicating every little bit of file data,
> which you may not care about that much, and which you have backed up
> anyway (right? right?).

Is it really RAID0, or is the btrfs way only similar to RAID0? I don't =
=20
like RAID0 because I never now on which physical disk the files are. =20
That makes changing disks very risky.

[...]

> This will almost certainly become much more tunable in the future,
> but not every feature that people want is done yet.  In fact, most of
> the really cool user-visible features aren't done yet.  Btrfs is
> still pretty young.

But I still hope btrfs is smarter than RAID0 or LVM ...

[...]

>> 1.5 TByte disk:
>> =A0 =A0 =A0 =A0btrfs device delete /dev/sdc3 /srv/MM
>> =A0 =A0 =A0 =A0btrfs filesystem balance /srv/MM
>>
>> and then disconnect the 1.5 TByte disk (and hope that now the 2
>> TByte disk sets the limits).
>> No nice way ...

> No, just run the balance without adding another disk.  That will
> probably work (although it _will_ take a while on a large
> filesystem).

I'll try - perhaps it helps for some (few) weeks.

> I'm not sure that you understand how this all works though;  you
> might want to re-read the wiki articles (which I believe have been
> freshened up recently).

Beg your pardon - my major interest is that the system works. I'm glad =
=20
when I believe to understand what happens, but this feeling is an add-=20
on, no must.

>> Is there a way to avoid this (presumably) RAID mismatch?

> Yes, you can specify the raid level for each when you make the
> filesystem (and will eventually be able to do it with existing
> filesystems).  However, as I described above, you really want
> metadata to be duplicated.  Your problem is more of an unfortunate
> case of everything not being tuned quite right yet.

May be - I thought avoiding the "RAID0" definition was a good idea.

>> By the way: working with TByte disks includes (for home users) that
>> there's no backup ...

> Not sure why you'd think that.  It can't be the bandwidth, and if you
> can't afford a second drive, there's a good case to be made that you
> can't afford the data you can't afford to lose.

The data isn't really valuable - DVB videos. Most of them are copied to=
 =20
disks in a machine about 250 km away. And the TV stations repeat them o=
n =20
and on.

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 05.12.10:

>>> If the smallest device defines the capacity then I should use
>>> 2*1.35 TiByte, but my system tells "no space left" at about 2.4
>>> TiByte - where are (at least) 300 GiByte hidden?

>>>>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
>>>>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2

>> Here devid 2 is at 100%, and hence you are getting the no more space
>> left errors. So, the 300 TB is on the bigger disk, and not usable
>> for you right now.

>    I _think_ that a balance is all that's needed at this point. It
> can't hurt, anyway (other than taking quite a long time).


It's just running - tomorrow I'll know more.

        grep relocating /var/log/messages

(counting down the groups) tells the proceedings.

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Evert Vorster]

>> >>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc=
3
>> >>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf=
2
>>
>> Here devid 2 is at 100%, and hence you are getting the no more space
>> left errors. So, the 300 TB is on the bigger disk, and not usable fo=
r
>> you right now.
>
> =A0 I _think_ that a balance is all that's needed at this point. It
> can't hurt, anyway (other than taking quite a long time).
I am under the impression that as soon as one of the devices in a
btrfs is out of space, and write to that file system returns ENOSPACE
This certainly does look like it from this example, and from what I
have read elsewhere. Also, I believe that the balance operation tries
to equalize the amount of bytes used on each underlying device, rather
than try to equalize the percentage of underlying devices filled. With
exactly the same amount used on both disks, I doubt that there would
be any advantage to a balance operation.

>> I know of the disk mode you speak.. an old raid card of mine called =
it
>> "Just a bunch of disks" and it literally filled up the first disk
>> before carrying on to the second one until that was full under
>> windows... under UNIX it had the effect of just adding all the secto=
rs
>> to each other, and stretching the file system over the disks in a
>> linear fashion. Most UNIX file systems writes files in the middle of
>> the largest contiguous free space, which meant that some files got
>> written on the first disk, and some on the second. As far as I know,
>> btrfs does not support this raid mode.
>
> =A0 It does support it: that's what the "single" RAID profile in
> mkfs.btrfs is. It attempts to use the disk space marginally more
> intelligently than traditional linear mode, though, as it allocates
> block groups (in chunks of about 1G) to each disk in turn. This isn't
> the same as RAID-0, which stripes within block groups with a much
> smaller stripe size.
I stand corrected.
Which brings us to the question... if you don't specify the raid level
when creating the btrfs, what is the default? raid0? single?

>> Another thing to keep in mind is that as far as I know you cannot
>> remove devid 1 from a btrfs volume. This is due to be fixed, but I
>> have no idea on the status of that.
>
> =A0 I've done it (I have a filesystem with IDs 7, 8, 9, 12, 13, 14).
> Looks like that particular problem has been fixed.
Good to know. I'm outgrowing one of my filesystems as well... and
would like to be able to migrate to bigger disks without rebuilding
the btrfs from scratch.

>> You could, if you really wanted to use all of two differently sized
>> disks in a btrfs, subdivide the disks in equal sized partitions, and
>> just put all of those partitions in a btrfs raid0...
> [...]
>
> =A0 That would be a really bad idea, as your disks would thrash
> horribly, reading stripes from different locations on the disk.
True, it would be slower than a normal raid0... in "single" mode there
should be almost no thrashing, though.

This does warrant some testing, though. I'll see what I can do testing
wise. Unfortunately, I'm no good at coding so the best I can do is
test and report results...

:(

-Evert Vorster-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Evert,

Du meintest am 05.12.10 zum Thema Re: 800 GByte free, but "no space lef=
t":

>>>>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc=
3
>>>>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf=
2

>>> Here devid 2 is at 100%, and hence you are getting the no more
>>> space left errors. So, the 300 TB is on the bigger disk, and not
>>> usable for you right now.

>> =A0 I _think_ that a balance is all that's needed at this point. It
>> can't hurt, anyway (other than taking quite a long time).

> I am under the impression that as soon as one of the devices in a
> btrfs is out of space, and write to that file system returns ENOSPACE
> This certainly does look like it from this example, and from what I
> have read elsewhere.

Maybe - I don't know the code.

But in my case "ENOSPACE" is returned too early.

"btrfs-show" tells 2*1.35 TiByte used, but "no space left" is told when=
 =20
2.4 TiByte are used (2.38 TiByte is ok). That's 300 GiByte too early (o=
r =20
about 10% too early).

Or am I wrong in interpreting the messages?

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Evert Vorster]

On Sun, Dec 5, 2010 at 7:33 AM, Helmut Hullen <Hullen@t-online.de> wrot=
e:
> Hallo, Evert,
>
> Du meintest am 05.12.10 zum Thema Re: 800 GByte free, but "no space l=
eft":
>
>>>>>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sd=
c3
>>>>>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sd=
f2
>
I'd like to point to this output of btrfs filesystem show again...
Here devid is 100% full. This, I assume, is both data and metadata.
When you do a df, what you see is only data, and not the metadata.
That might account for the difference you see.

I did a little bit of testing on my machine....

I created two partitions, /dev/sdb6 (1Gb)  and  /dev/sdb7(2Gb)
When I created an btrfs with the following command:
mkfs.btrfs -d raid0 /dev/sdb6 /dev/sdb7

I got a filesystem that started giving ENOSPACE with only 1.6Gb full.
Running a filesystem balance did not do anything.... and it was
predictably slow...

When I created a file system with
mkfs.btrfs -d single /dev/sdb6 /dev/sdb7
I was able to fill the resulting file system to 2.7Gb before it
started throwing ENOSPACE this file system was quite a bit faster...

So, by far the simplest solution would be to re-create your file
system with "single" mode, and then you will be able to utilize all of
the space of both disks. I was using quite chunky files, so there
would be some granularity in the test results, but it clearly shows
that raid0 is limited in space to (smallest subdevice) x (number of
subdevices) whereas "single" tries to use all of the available space
on all the subdevices.

Kind regards,
-Evert Vorster-
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Evert,

Du meintest am 05.12.10:

>>>>>> =A0 =A0 =A0 devid =A0 =A02 size 1.35TB used 1.35TB path /dev/sdc=
3
>>>>>> =A0 =A0 =A0 devid =A0 =A01 size 1.81TB used 1.35TB path /dev/sdf=
2

[...]

> When I created a file system with
> mkfs.btrfs -d single /dev/sdb6 /dev/sdb7
> I was able to fill the resulting file system to 2.7Gb before it
> started throwing ENOSPACE this file system was quite a bit faster...

> So, by far the simplest solution would be to re-create your file
> system with "single" mode,

Can I add or delete hard disks/partitions to the two devices/partitions=
 =20
in your example?
I've studied the man page and the Wiki but didn't find any help.

And in my special case I have to add at least yearly new disks and from=
 =20
time to time remove the smallest disks from this bundle.

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 05.12.10:

>>>>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
>>>>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2
>>
>> Here devid 2 is at 100%, and hence you are getting the no more space
>> left errors. So, the 300 TB is on the bigger disk, and not usable
>> for you right now.

>    I _think_ that a balance is all that's needed at this point. It
> can't hurt, anyway (other than taking quite a long time).

After running "balance" it looks better:

Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
	Total devices 2 FS bytes used 2.39TB
	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
	devid    1 size 1.81TB used 1.20TB path /dev/sdf2

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo,

I wrote am 05.12.10:

>>>>>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
>>>>>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2

>>> Here devid 2 is at 100%, and hence you are getting the no more
>>> space left errors. So, the 300 TB is on the bigger disk, and not
>>> usable for you right now.

>>    I _think_ that a balance is all that's needed at this point. It
>> can't hurt, anyway (other than taking quite a long time).

> After running "balance" it looks better:

> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> 	Total devices 2 FS bytes used 2.39TB
> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2

And now:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc3            3400799848 2569770888 831028960  76% /srv/MM

But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I got "no  
space left on device". Looks like balancing has stolen about 300 GByte.

Time to buy a 3-TByte disk ...

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Brian Rogers]

On 12/05/2010 10:26 AM, Helmut Hullen wrote:
>> So, by far the simplest solution would be to re-create your file
>> system with "single" mode,
> Can I add or delete hard disks/partitions to the two devices/partitions
> in your example?

You can add, but you'll want to avoid deleting, balancing, and 
shrinking. From my testing, any of these operations will convert the 
chunks they relocate to raid0. Then, once you have any raid0 data, btrfs 
will want to allocate all new chunks as raid0 and you'll have the 
unusable space problem again.

> I've studied the man page and the Wiki but didn't find any help.
>
> And in my special case I have to add at least yearly new disks and from
> time to time remove the smallest disks from this bundle.

With the current state of btrfs, you could do this as long as you never 
reduce the total number of disks. When you want to replace an old disk 
with a new one, just go around btrfs: take the filesystem offline and 
copy the old disk's partition into a full-sized partition on the new 
disk. Then remove the old disk and bring the filesystem online again.

It will remember the old size at first, but that can be fixed with

btrfs filesystem resize <devid>:max <path>

Also, if you want metadata redundancy, you have to start your new 
filesystem with two or more disks. Otherwise, the only way to change 
metadata to raid1 is btrfs balance, which must be avoided.

Brian

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1715 bytes --]

On Mon, Dec 06, 2010 at 01:56:31AM -0800, Brian Rogers wrote:
> On 12/05/2010 10:26 AM, Helmut Hullen wrote:
> >>So, by far the simplest solution would be to re-create your file
> >>system with "single" mode,
> >Can I add or delete hard disks/partitions to the two devices/partitions
> >in your example?
> 
> You can add, but you'll want to avoid deleting, balancing, and
> shrinking. From my testing, any of these operations will convert the
> chunks they relocate to raid0. Then, once you have any raid0 data,
> btrfs will want to allocate all new chunks as raid0 and you'll have
> the unusable space problem again.

   That's a known bug. Josef even posted a patch for it (but it's
apparently not doing the right thing yet).

> >I've studied the man page and the Wiki but didn't find any help.
> >
> >And in my special case I have to add at least yearly new disks and from
> >time to time remove the smallest disks from this bundle.
> 
> With the current state of btrfs, you could do this as long as you
> never reduce the total number of disks. When you want to replace an
> old disk with a new one, just go around btrfs: take the filesystem
> offline and copy the old disk's partition into a full-sized
> partition on the new disk. Then remove the old disk and bring the
> filesystem online again.
> 
> It will remember the old size at first, but that can be fixed with
> 
> btrfs filesystem resize <devid>:max <path>

   This doesn't work on multi-volume filesystems (yet).

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- We believe in free will because we have no choice. ---        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

On Mon, Dec 06, 2010 at 08:43:00AM +0100, Helmut Hullen wrote:
> Hallo,
> 
> I wrote am 05.12.10:
> 
> >>>>>>       devid    2 size 1.35TB used 1.35TB path /dev/sdc3
> >>>>>>       devid    1 size 1.81TB used 1.35TB path /dev/sdf2
> 
> >>> Here devid 2 is at 100%, and hence you are getting the no more
> >>> space left errors. So, the 300 TB is on the bigger disk, and not
> >>> usable for you right now.
> 
> >>    I _think_ that a balance is all that's needed at this point. It
> >> can't hurt, anyway (other than taking quite a long time).
> 
> > After running "balance" it looks better:
> 
> > Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> > 	Total devices 2 FS bytes used 2.39TB
> > 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
> > 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
> 
> And now:
> 
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdc3            3400799848 2569770888 831028960  76% /srv/MM
> 
> But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I got "no  
> space left on device". Looks like balancing has stolen about 300 GByte.

   This sounds exactly like a problem I've had. What output do you get
from "btrfs fi df /srv/MM"?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- We believe in free will because we have no choice. ---        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 06.12.10:

>>> After running "balance" it looks better:

>>> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
>>> 	Total devices 2 FS bytes used 2.39TB
>>> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
>>> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
>>
>> And now:
>>
>> Filesystem           1K-blocks      Used Available Use% Mounted on
>> /dev/sdc3            3400799848 2569770888 831028960  76% /srv/MM
>>
>> But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I got
>> "no space left on device". Looks like balancing has stolen about 300
>> GByte.

>    This sounds exactly like a problem I've had. What output do you
> get from "btrfs fi df /srv/MM"?

I've just written a script for gathering the (perhaps) interesting data  
...

# btrfs filesystem show
Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
	Total devices 2 FS bytes used 2.37TB
	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
	devid    1 size 1.81TB used 1.20TB path /dev/sdf2

Btrfs Btrfs v0.19

# btrfs filesystem df /srv/MM
Data: total=2.39TB, used=2.37TB
Metadata: total=5.25GB, used=3.51GB
System: total=12.00MB, used=188.00KB

# df -t btrfs
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/sdc3    btrfs   3400799848 2547059672 853740176  75% /srv/MM

# fdisk -l
/dev/sdc3            1569      182401  1452541072+  83  Linux
/dev/sdf2             655      243201  1948258777+  83  Linux

# ----------------------------------------

I've moved about 50 Gbyte away from "srv/MM" in the meantime, before  
running the script with this output.

And I don't dare running "balance" again - maybe it reduces the  
available space again and again.

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2109 bytes --]

On Mon, Dec 06, 2010 at 01:42:00PM +0100, Helmut Hullen wrote:
> Hallo, Hugo,
> 
> Du meintest am 06.12.10:
> 
> >>> After running "balance" it looks better:
> 
> >>> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> >>> 	Total devices 2 FS bytes used 2.39TB
> >>> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
> >>> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
> >>
> >> And now:
> >>
> >> Filesystem           1K-blocks      Used Available Use% Mounted on
> >> /dev/sdc3            3400799848 2569770888 831028960  76% /srv/MM
> >>
> >> But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I got
> >> "no space left on device". Looks like balancing has stolen about 300
> >> GByte.
> 
> >    This sounds exactly like a problem I've had. What output do you
> > get from "btrfs fi df /srv/MM"?
> 
> I've just written a script for gathering the (perhaps) interesting data  
> ...
> 
> # btrfs filesystem show
> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> 	Total devices 2 FS bytes used 2.37TB
> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
> 
> Btrfs Btrfs v0.19
> 
> # btrfs filesystem df /srv/MM
> Data: total=2.39TB, used=2.37TB
> Metadata: total=5.25GB, used=3.51GB
> System: total=12.00MB, used=188.00KB

   Can you try that again with either the latest 2.6.37-rc, or with
the btrfs-unstable kernel? There's a bug in earlier versions that
breaks the reporting of RAID types, which is what I wanted to see
here.

> I've moved about 50 Gbyte away from "srv/MM" in the meantime, before  
> running the script with this output.
> 
> And I don't dare running "balance" again - maybe it reduces the  
> available space again and again.

   If you've hit the bug I think you have, then yes, it will.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- Do not meddle in the affairs of wizards, for they are subtle, ---  
                           and quick to anger.                           

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 06.12.10:

>>>> But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I
>>>> got "no space left on device". Looks like balancing has stolen
>>>> about 300 GByte.
>>
>>>    This sounds exactly like a problem I've had. What output do you
>>> get from "btrfs fi df /srv/MM"?
>>
>> I've just written a script for gathering the (perhaps) interesting
>> data ...
>>
>> # btrfs filesystem show
>> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
>> 	Total devices 2 FS bytes used 2.37TB
>> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
>> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
>>
>> Btrfs Btrfs v0.19
>>
>> # btrfs filesystem df /srv/MM
>> Data: total=2.39TB, used=2.37TB
>> Metadata: total=5.25GB, used=3.51GB
>> System: total=12.00MB, used=188.00KB

>    Can you try that again with either the latest 2.6.37-rc, or with
> the btrfs-unstable kernel? There's a bug in earlier versions that
> breaks the reporting of RAID types, which is what I wanted to see
> here.

Do you mean

  git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git

as "btrfs-unstable kernel"?

Compiling 2.6.37-rc is no big problem, it only needs som time.

Just now I'm using

Kernel 2.6.35.8
btrfs-git from 20101117

>> I've moved about 50 Gbyte away from "srv/MM" in the meantime, before
>> running the script with this output.
>>
>> And I don't dare running "balance" again - maybe it reduces the
>> available space again and again.

>    If you've hit the bug I think you have, then yes, it will.

Hmm - it can't get worse ...
If the error is related to the kernel or to the btrfs version and I try  
a newer one: can that lead to more free space?

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2492 bytes --]

On Mon, Dec 06, 2010 at 02:13:00PM +0100, Helmut Hullen wrote:
> Hallo, Hugo,
> 
> Du meintest am 06.12.10:
> 
> >>>> But after copying about 300 MByte (part of a 1.5-GByte *.mpg) I
> >>>> got "no space left on device". Looks like balancing has stolen
> >>>> about 300 GByte.
> >>
> >>>    This sounds exactly like a problem I've had. What output do you
> >>> get from "btrfs fi df /srv/MM"?
> >>
> >> I've just written a script for gathering the (perhaps) interesting
> >> data ...
> >>
> >> # btrfs filesystem show
> >> Label: 'MM2'  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> >> 	Total devices 2 FS bytes used 2.37TB
> >> 	devid    2 size 1.35TB used 1.20TB path /dev/sdc3
> >> 	devid    1 size 1.81TB used 1.20TB path /dev/sdf2
> >>
> >> Btrfs Btrfs v0.19
> >>
> >> # btrfs filesystem df /srv/MM
> >> Data: total=2.39TB, used=2.37TB
> >> Metadata: total=5.25GB, used=3.51GB
> >> System: total=12.00MB, used=188.00KB
> 
> >    Can you try that again with either the latest 2.6.37-rc, or with
> > the btrfs-unstable kernel? There's a bug in earlier versions that
> > breaks the reporting of RAID types, which is what I wanted to see
> > here.
> 
> Do you mean
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-unstable.git
> 
> as "btrfs-unstable kernel"?

   Yes. It's 2.6.36, plus the patches that Chris has sent to Linus for
inclusion into 2.6.37.

> Compiling 2.6.37-rc is no big problem, it only needs som time.
> 
> Just now I'm using
> 
> Kernel 2.6.35.8
> btrfs-git from 20101117
> 
> >> I've moved about 50 Gbyte away from "srv/MM" in the meantime, before
> >> running the script with this output.
> >>
> >> And I don't dare running "balance" again - maybe it reduces the
> >> available space again and again.
> 
> >    If you've hit the bug I think you have, then yes, it will.
> 
> Hmm - it can't get worse ...
> If the error is related to the kernel or to the btrfs version and I try  
> a newer one: can that lead to more free space?

   Not yet. I've taken the whole of December off work (using up my
leave allocation for last year), and my plan is to get myself to the
point where I can understand enough of the code to fix this particular
problem.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- "What are we going to do tonight?" "The same thing we do ---     
            every night, Pinky.  Try to take over the world!"            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 06.12.10:

>>>    Can you try that again with either the latest 2.6.37-rc, or with
>>> the btrfs-unstable kernel? There's a bug in earlier versions that
>>> breaks the reporting of RAID types, which is what I wanted to see
>>> here.

>> Compiling 2.6.37-rc is no big problem, it only needs som time.

Compiling runs ..

>>>> And I don't dare running "balance" again - maybe it reduces the
>>>> available space again and again.

>>>    If you've hit the bug I think you have, then yes, it will.

>> Hmm - it can't get worse ...
>> If the error is related to the kernel or to the btrfs version and I
>> try a newer one: can that lead to more free space?

>    Not yet. I've taken the whole of December off work (using up my
> leave allocation for last year), and my plan is to get myself to the
> point where I can understand enough of the code to fix this
> particular problem.

I love such vacancies planning ...

If/when I install 2.6.37-rc4: should I update btrfs (from the 20101117  
version)?

How can I see that changing the kernel makes things better? It's more  
and more difficult to externalize (?) btrfs directories to other disks  
...

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1254 bytes --]

   Helmut - 

On Mon, Dec 06, 2010 at 03:45:00PM +0100, Helmut Hullen wrote:
> If/when I install 2.6.37-rc4: should I update btrfs (from the 20101117  
> version)?

   I think that's the latest version.

> How can I see that changing the kernel makes things better? It's more  
> and more difficult to externalize (?) btrfs directories to other disks  
> ...

   Updating the kernel won't fix the problem I'm thinking of (sorry).
It will, however, fix the bug that stops the btrfs tool from reporting
what RAID levels you've got.

   The problem I suspect you may have (because your symptoms seem to
be the same as mine) is that there are some circumstances where the
filesystem can change RAID levels pretty much arbitrarily. Running
"btrfs fi df" with a kernel that reports RAID levels will show whether
that's the case, as you'll have more than one RAID level listed.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
  --- But people have always eaten people,  / what else is there to ---  
         eat?  / If the Juju had meant us not to eat people / he         
                     wouldn't have made us of meat.                      

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 06.12.10:

>> How can I see that changing the kernel makes things better? It's
>> more and more difficult to externalize (?) btrfs directories to
>> other disks ...

>    Updating the kernel won't fix the problem I'm thinking of (sorry).
> It will, however, fix the bug that stops the btrfs tool from
> reporting what RAID levels you've got.

>    The problem I suspect you may have (because your symptoms seem to
> be the same as mine) is that there are some circumstances where the
> filesystem can change RAID levels pretty much arbitrarily. Running
> "btrfs fi df" with a kernel that reports RAID levels will show
> whether that's the case, as you'll have more than one RAID level
> listed.

Kernel 2.6.35.8:

# btrfs filesystem df /srv/MM
Data: total=2.39TB, used=2.37TB
Metadata: total=5.25GB, used=3.51GB
System: total=12.00MB, used=188.00KB

Kernel 2.6.37-rc4:

# btrfs filesystem df /srv/MM
Data, RAID0: total=2.39TB, used=2.37TB
System, RAID1: total=8.00MB, used=188.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=4.25GB, used=3.51GB
Metadata, DUP: total=1.00GB, used=2.33MB

Hope it helps!

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1556 bytes --]

On Mon, Dec 06, 2010 at 06:13:00PM +0100, Helmut Hullen wrote:
> Hallo, Hugo,
> 
> Du meintest am 06.12.10:
> 
> >> How can I see that changing the kernel makes things better? It's
> >> more and more difficult to externalize (?) btrfs directories to
> >> other disks ...
> 
> >    Updating the kernel won't fix the problem I'm thinking of (sorry).
> > It will, however, fix the bug that stops the btrfs tool from
> > reporting what RAID levels you've got.
> 
> >    The problem I suspect you may have (because your symptoms seem to
> > be the same as mine) is that there are some circumstances where the
> > filesystem can change RAID levels pretty much arbitrarily. Running
> > "btrfs fi df" with a kernel that reports RAID levels will show
> > whether that's the case, as you'll have more than one RAID level
> > listed.

> Kernel 2.6.37-rc4:
> 
> # btrfs filesystem df /srv/MM
> Data, RAID0: total=2.39TB, used=2.37TB
> System, RAID1: total=8.00MB, used=188.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=4.25GB, used=3.51GB
> Metadata, DUP: total=1.00GB, used=2.33MB
> 
> Hope it helps!

   Yup. You've got what I've got(*). You have two different RAID types
for metadata, which shouldn't happen (but does, due to a bug).

   Hugo.

(*) My .sig fairy is clearly working overtime for appropriate
quotations. :)

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Charting the inexorable advance of Western syphilisation... ---   

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 06.12.10:

>> Kernel 2.6.37-rc4:
>>
>> # btrfs filesystem df /srv/MM
>> Data, RAID0: total=2.39TB, used=2.37TB
>> System, RAID1: total=8.00MB, used=188.00KB
>> System: total=4.00MB, used=0.00
>> Metadata, RAID1: total=4.25GB, used=3.51GB
>> Metadata, DUP: total=1.00GB, used=2.33MB
>>
>> Hope it helps!

>    Yup. You've got what I've got(*). You have two different RAID
> types for metadata, which shouldn't happen (but does, due to a bug).

Fear I right that "balancing" tries to reduce the system to something  
like RAID1?

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1380 bytes --]

   Helmut -

On Tue, Dec 07, 2010 at 06:05:00PM +0100, Helmut Hullen wrote:
> Du meintest am 06.12.10:
> 
> >> Kernel 2.6.37-rc4:
> >>
> >> # btrfs filesystem df /srv/MM
> >> Data, RAID0: total=2.39TB, used=2.37TB
> >> System, RAID1: total=8.00MB, used=188.00KB
> >> System: total=4.00MB, used=0.00
> >> Metadata, RAID1: total=4.25GB, used=3.51GB
> >> Metadata, DUP: total=1.00GB, used=2.33MB
> >>
> >> Hope it helps!
> 
> >    Yup. You've got what I've got(*). You have two different RAID
> > types for metadata, which shouldn't happen (but does, due to a bug).
> 
> Fear I right that "balancing" tries to reduce the system to something  
> like RAID1?

   It _should_ move all the data on the disk to somewhere else on the
disk, whilst honouring the RAID settings for the filesystem. However,
since it's got buggered up RAID settings now and has been using some
of the space for the wrong RAID type, the balance can't find space
with the right RAID parameters to write to, so it "runs out of space".

   (I think I got that right, anyway. I'm working off a conversation
with Chris on IRC some weeks ago, about what happened to my
filesystem).

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- Always be sincere,  whether you mean it or not. ---         

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Hugo,

Du meintest am 07.12.10:

>> Fear I right that "balancing" tries to reduce the system to
>> something like RAID1?

>    It _should_ move all the data on the disk to somewhere else on the
> disk,

[...]

>    (I think I got that right, anyway. I'm working off a conversation
> with Chris on IRC some weeks ago, about what happened to my
> filesystem).

Ok - I'll wait some days (and weeks).

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo,

I wrote am 02.12.10:

> I use 2 disks (1.5 Tbyte and 2.0 TByte) under 1 LABEL (for my video
> collection, nearly alle files have more than 1 GByte):

> Label: MM2  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
> 	Total devices 2 FS bytes used 2.38TB
> 	devid    2 size 1.35TB used 1.35TB path /dev/sdc3
> 	devid    1 size 1.81TB used 1.35TB path /dev/sdf2

> ("btrfs-show" uses TiByte, it's 10% less than TByte)

> Btrfs Btrfs v0.19

> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdc3            3400799848 2559596740 841203108  76% /srv/MM

> --------------------------------

> When I add some more videos, writing gets slower and slower, and then
> the system refuses with "no space left ..."

[...]

No help?

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Helmut Hullen]

Hallo, Mike,

Du meintest am 02.12.10:

>> Btrfs Btrfs v0.19

> "btrfs" in the kernel has been version 0.19 for a *long* time.  The
> version number there may never change.  How do you encode a feature
> mask in a version number?  Some features may be in one tree but not
> upstreamed all together and other such minutiae.

Sorry - I forgot:

Kernel 2.6.35.8
btrfs-git from 20101117

Viele Gruesse!
Helmut

Re: 800 GByte free, but "no space left"

[Mike Fedyk]

On Thu, Dec 2, 2010 at 10:23 AM, Helmut Hullen <Hullen@t-online.de> wrote:
> Btrfs Btrfs v0.19
>

"btrfs" in the kernel has been version 0.19 for a *long* time.  The
version number there may never change.  How do you encode a feature
mask in a version number?  Some features may be in one tree but not
upstreamed all together and other such minutiae.

What you need to do is use a more recent kernel than 2.6.32 if you
want to use btrfs (modulo backports, but let's not talk about that
right now).

So if you're using a kernel older than 2.6.36, then you should probably upgrade.

800 GByte free, but "no space left"

[Helmut Hullen]

Hallo,

I've new problems.

I use 2 disks (1.5 Tbyte and 2.0 TByte) under 1 LABEL (for my video  
collection, nearly alle files have more than 1 GByte):

Label: MM2  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
	Total devices 2 FS bytes used 2.38TB
	devid    2 size 1.35TB used 1.35TB path /dev/sdc3
	devid    1 size 1.81TB used 1.35TB path /dev/sdf2

("btrfs-show" uses TiByte, it's 10% less than TByte)

Btrfs Btrfs v0.19

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc3            3400799848 2559596740 841203108  76% /srv/MM

--------------------------------

When I add some more videos, writing gets slower and slower, and then  
the system refuses with "no space left ..."

Label: MM2  uuid: ad7c0668-316c-4a79-ba00-3b505b9d99b4
	Total devices 2 FS bytes used 2.40TB
	devid    2 size 1.35TB used 1.35TB path /dev/sdc3
	devid    1 size 1.81TB used 1.35TB path /dev/sdf2

Btrfs Btrfs v0.19

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sdc3            3400799848 2585340332 815459516  77% /srv/MM

---------------------------------

When I try to rename files I get "no space left ...". When I delete some  
files and then try again to rename the system doesn't really rename but  
first copies to the new name and then deletes the old file. Same  
behaviour when I try to move a file from one directory to another.

----------------------------------

Where is the bottleneck?

Viele Gruesse!
Helmut