No space left on device

No space left on device

[Jakob Truelsen]

Hello. I am experiencing "No space left on device" with a btrfs file
system, yet I cannot seem to find any exhausted resource. Could some
resource I do not know about be exhausted, or is this caused by
something else. Below is a trace of information that might be usefull,
please let me know if there is anything I can do to resolve the issue.

/Jakob

[jakobt@soda ~]$ uname -a
Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
x86_64 GNU/Linux

[jakobt@soda ~]$ btrfs --version
Btrfs v3.12

[jakobt@soda ~]$ mount
...
/dev/sda on /data type btrfs (rw,relatime,nospace_cache)

[jakobt@soda ~]$ df /data
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sdb2       76594224 49247368  23433028  68% /

[jakobt@soda ~]$ btrfs filesystem df /data
Data, single: total=1.82TiB, used=518.04GiB
System, DUP: total=8.00MiB, used=204.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=1.00GiB, used=767.70MiB
Metadata, single: total=8.00MiB, used=0.00

[jakobt@soda ~]$ touch /data/jakobt/hat
touch: cannot touch ‘/data/jakobt/hat’: No space left on device

[jakobt@soda ~]$ sudo btrfs fi balance start /data
ERROR: error during balancing '/data' - No space left on device
There may be more info in syslog - try dmesg | tail

[jakobt@soda ~]$ dmesg | grep tail -n 2
[1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
[1113507.641752] btrfs: 1866 enospc errors during balance

[jakobt@soda ~]$ sudo umount /data

[jakobt@soda ~]$ sudo btrfsck /dev/sda
...
cache and super generation don't match, space cache will be invalidated
..
found 172181366447 bytes used err is 0
total csum bytes: 418841160
total tree bytes: 805187584
total fs tree bytes: 247164928
total extent tree bytes: 26329088
btree space waste bytes: 164771401
file data blocks allocated: 561564688384
 referenced 511759908864
Btrfs v3.12

Re: No space left on device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3295 bytes --]

On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote:
> Hello. I am experiencing "No space left on device" with a btrfs file
> system, yet I cannot seem to find any exhausted resource. Could some
> resource I do not know about be exhausted, or is this caused by
> something else. Below is a trace of information that might be usefull,
> please let me know if there is anything I can do to resolve the issue.
> 
> /Jakob
> 
> [jakobt@soda ~]$ uname -a
> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
> x86_64 GNU/Linux

   Were you using this kernel when the problem happened?

> [jakobt@soda ~]$ btrfs --version
> Btrfs v3.12
> 
> [jakobt@soda ~]$ mount
> ...
> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
> 
> [jakobt@soda ~]$ df /data
> Filesystem     1K-blocks     Used Available Use% Mounted on
> /dev/sdb2       76594224 49247368  23433028  68% /
> 
> [jakobt@soda ~]$ btrfs filesystem df /data
> Data, single: total=1.82TiB, used=518.04GiB
> System, DUP: total=8.00MiB, used=204.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=1.00GiB, used=767.70MiB

   ^^^ This is your problem, most likely in conjunction with all the
space on the device being allocated. Being a copy-on-write filesystem,
btrfs needs free space to make any update. If it doesn't have that
free space, you get "No space left on device". You typically need
somewhere around 0.5-1 GiB of headroom in metadata for normal
operation, so I'm surprised you got this far. :)

   The FS should normally allocate more metadata space as it needs it,
but because (I think) your data allocation has taken up all the
available space on the device, there's no way for it to add more.

> Metadata, single: total=8.00MiB, used=0.00

> [jakobt@soda ~]$ touch /data/jakobt/hat
> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
> 
> [jakobt@soda ~]$ sudo btrfs fi balance start /data
> ERROR: error during balancing '/data' - No space left on device
> There may be more info in syslog - try dmesg | tail

   Try:

btrfs balance start -dusage=0 /data

which should go looking for entirely unused block groups and reclaim
those. (If you don't use the -dusage= parameter, it will try to
balance everything, which takes a long time).

> [jakobt@soda ~]$ dmesg | grep tail -n 2
> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
> [1113507.641752] btrfs: 1866 enospc errors during balance

   Although, that said... it looks like it's tried every block group
and failed with each one, so my suggestion above may not work in this
instance.

   Let us know what happens with the balance command above anyway
(dmesg output is useful information at this point). If that doesn't
help, then we'll probably need to take a metadata image and throw it
in josef's direction, where he will start crying at having to deal
with enospc problems again. :)

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- If you're not part of the solution, you're part ---         
                           of the precipiate.                            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: No space left on device

[Leonidas Spyropoulos]

On 12/02/2014 09:51, Jakob Truelsen wrote:
> Hello. I am experiencing "No space left on device" with a btrfs file
> system, yet I cannot seem to find any exhausted resource. Could some
> resource I do not know about be exhausted, or is this caused by
> something else. Below is a trace of information that might be usefull,
> please let me know if there is anything I can do to resolve the issue.
>
> /Jakob
>
> [jakobt@soda ~]$ uname -a
> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
> x86_64 GNU/Linux
>
> [jakobt@soda ~]$ btrfs --version
> Btrfs v3.12
>
> [jakobt@soda ~]$ mount
> ...
> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
>
> [jakobt@soda ~]$ df /data
> Filesystem     1K-blocks     Used Available Use% Mounted on
> /dev/sdb2       76594224 49247368  23433028  68% /
>
> [jakobt@soda ~]$ btrfs filesystem df /data
> Data, single: total=1.82TiB, used=518.04GiB
> System, DUP: total=8.00MiB, used=204.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=1.00GiB, used=767.70MiB
> Metadata, single: total=8.00MiB, used=0.00
>
> [jakobt@soda ~]$ touch /data/jakobt/hat
> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
>
> [jakobt@soda ~]$ sudo btrfs fi balance start /data
> ERROR: error during balancing '/data' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> [jakobt@soda ~]$ dmesg | grep tail -n 2
> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
> [1113507.641752] btrfs: 1866 enospc errors during balance
>
> [jakobt@soda ~]$ sudo umount /data
>
> [jakobt@soda ~]$ sudo btrfsck /dev/sda
> ...
> cache and super generation don't match, space cache will be invalidated
> ..
> found 172181366447 bytes used err is 0
> total csum bytes: 418841160
> total tree bytes: 805187584
> total fs tree bytes: 247164928
> total extent tree bytes: 26329088
> btree space waste bytes: 164771401
> file data blocks allocated: 561564688384
>   referenced 511759908864
> Btrfs v3.12
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Hello Jacob,

Have you tried balancing just the data/metadata chunks only?

Regards,
Leonidas

Re: No space left on device

[Jakob Truelsen]

Hi and thanks for the quick reply. Have remounted the filesystem with
enospc_debug, and run the rebalance you suggested, with the trance
below. So perhaps the next step is for me to figure out how to take a
metadata image and send it to josef (perhaps with a box of tissues)

/Jakob


[jakobt@soda ~]$ mount
...
/dev/sda on /data type btrfs (rw,relatime,nospace_cache,enospc_debug)

[jakobt@soda ~]$ sudo  btrfs balance start -dusage=0 /data
ERROR: error during balancing '/data' - No space left on device
There may be more info in syslog - try dmesg | tail

[jakobt@soda ~]$ touch /data/jakobt/monkey
touch: cannot touch ‘/data/jakobt/monkey’: No space left on device

[jakobt@soda ~]$ dmesg | tail -n3
[1117530.870965] btrfs: device label Data devid 1 transid 47091 /dev/sda
[1117573.087580] btrfs: 426 enospc errors during balance
[1117642.002437] btrfs: 426 enospc errors during balance


On Wed, Feb 12, 2014 at 11:26 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote:
>> Hello. I am experiencing "No space left on device" with a btrfs file
>> system, yet I cannot seem to find any exhausted resource. Could some
>> resource I do not know about be exhausted, or is this caused by
>> something else. Below is a trace of information that might be usefull,
>> please let me know if there is anything I can do to resolve the issue.
>>
>> /Jakob
>>
>> [jakobt@soda ~]$ uname -a
>> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
>> x86_64 GNU/Linux
>
>    Were you using this kernel when the problem happened?
>
>> [jakobt@soda ~]$ btrfs --version
>> Btrfs v3.12
>>
>> [jakobt@soda ~]$ mount
>> ...
>> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
>>
>> [jakobt@soda ~]$ df /data
>> Filesystem     1K-blocks     Used Available Use% Mounted on
>> /dev/sdb2       76594224 49247368  23433028  68% /
>>
>> [jakobt@soda ~]$ btrfs filesystem df /data
>> Data, single: total=1.82TiB, used=518.04GiB
>> System, DUP: total=8.00MiB, used=204.00KiB
>> System, single: total=4.00MiB, used=0.00
>> Metadata, DUP: total=1.00GiB, used=767.70MiB
>
>    ^^^ This is your problem, most likely in conjunction with all the
> space on the device being allocated. Being a copy-on-write filesystem,
> btrfs needs free space to make any update. If it doesn't have that
> free space, you get "No space left on device". You typically need
> somewhere around 0.5-1 GiB of headroom in metadata for normal
> operation, so I'm surprised you got this far. :)
>
>    The FS should normally allocate more metadata space as it needs it,
> but because (I think) your data allocation has taken up all the
> available space on the device, there's no way for it to add more.
>
>> Metadata, single: total=8.00MiB, used=0.00
>
>> [jakobt@soda ~]$ touch /data/jakobt/hat
>> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
>>
>> [jakobt@soda ~]$ sudo btrfs fi balance start /data
>> ERROR: error during balancing '/data' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>
>    Try:
>
> btrfs balance start -dusage=0 /data
>
> which should go looking for entirely unused block groups and reclaim
> those. (If you don't use the -dusage= parameter, it will try to
> balance everything, which takes a long time).
>
>> [jakobt@soda ~]$ dmesg | grep tail -n 2
>> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
>> [1113507.641752] btrfs: 1866 enospc errors during balance
>
>    Although, that said... it looks like it's tried every block group
> and failed with each one, so my suggestion above may not work in this
> instance.
>
>    Let us know what happens with the balance command above anyway
> (dmesg output is useful information at this point). If that doesn't
> help, then we'll probably need to take a metadata image and throw it
> in josef's direction, where he will start crying at having to deal
> with enospc problems again. :)
>
>    Hugo.
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>          --- If you're not part of the solution, you're part ---
>                            of the precipiate.

Re: No space left on device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 5136 bytes --]

On Wed, Feb 12, 2014 at 11:45:34AM +0100, Jakob Truelsen wrote:
> Hi and thanks for the quick reply. Have remounted the filesystem with
> enospc_debug, and run the rebalance you suggested, with the trance
> below. So perhaps the next step is for me to figure out how to take a
> metadata image and send it to josef (perhaps with a box of tissues)

   You can get the metadata image with:

btrfs-image -c9 -t4 /dev/sda image-file-to-create

   It will probably be a couple of hundred megabytes in size (your
metadata size, compressed). It will contain filenames and xattrs, so
if those are sensitive, you may want to use -s to hide that
information.

   After that, probably the best place to make sure it's not lost is
to open an issue on bugzilla.kernel.org, making sure that you set the
Component to btrfs.

   Hugo.

> /Jakob
> 
> 
> [jakobt@soda ~]$ mount
> ...
> /dev/sda on /data type btrfs (rw,relatime,nospace_cache,enospc_debug)
> 
> [jakobt@soda ~]$ sudo  btrfs balance start -dusage=0 /data
> ERROR: error during balancing '/data' - No space left on device
> There may be more info in syslog - try dmesg | tail
> 
> [jakobt@soda ~]$ touch /data/jakobt/monkey
> touch: cannot touch ‘/data/jakobt/monkey’: No space left on device
> 
> [jakobt@soda ~]$ dmesg | tail -n3
> [1117530.870965] btrfs: device label Data devid 1 transid 47091 /dev/sda
> [1117573.087580] btrfs: 426 enospc errors during balance
> [1117642.002437] btrfs: 426 enospc errors during balance

> On Wed, Feb 12, 2014 at 11:26 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> > On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote:
> >> Hello. I am experiencing "No space left on device" with a btrfs file
> >> system, yet I cannot seem to find any exhausted resource. Could some
> >> resource I do not know about be exhausted, or is this caused by
> >> something else. Below is a trace of information that might be usefull,
> >> please let me know if there is anything I can do to resolve the issue.
> >>
> >> /Jakob
> >>
> >> [jakobt@soda ~]$ uname -a
> >> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
> >> x86_64 GNU/Linux
> >
> >    Were you using this kernel when the problem happened?
> >
> >> [jakobt@soda ~]$ btrfs --version
> >> Btrfs v3.12
> >>
> >> [jakobt@soda ~]$ mount
> >> ...
> >> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
> >>
> >> [jakobt@soda ~]$ df /data
> >> Filesystem     1K-blocks     Used Available Use% Mounted on
> >> /dev/sdb2       76594224 49247368  23433028  68% /
> >>
> >> [jakobt@soda ~]$ btrfs filesystem df /data
> >> Data, single: total=1.82TiB, used=518.04GiB
> >> System, DUP: total=8.00MiB, used=204.00KiB
> >> System, single: total=4.00MiB, used=0.00
> >> Metadata, DUP: total=1.00GiB, used=767.70MiB
> >
> >    ^^^ This is your problem, most likely in conjunction with all the
> > space on the device being allocated. Being a copy-on-write filesystem,
> > btrfs needs free space to make any update. If it doesn't have that
> > free space, you get "No space left on device". You typically need
> > somewhere around 0.5-1 GiB of headroom in metadata for normal
> > operation, so I'm surprised you got this far. :)
> >
> >    The FS should normally allocate more metadata space as it needs it,
> > but because (I think) your data allocation has taken up all the
> > available space on the device, there's no way for it to add more.
> >
> >> Metadata, single: total=8.00MiB, used=0.00
> >
> >> [jakobt@soda ~]$ touch /data/jakobt/hat
> >> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
> >>
> >> [jakobt@soda ~]$ sudo btrfs fi balance start /data
> >> ERROR: error during balancing '/data' - No space left on device
> >> There may be more info in syslog - try dmesg | tail
> >
> >    Try:
> >
> > btrfs balance start -dusage=0 /data
> >
> > which should go looking for entirely unused block groups and reclaim
> > those. (If you don't use the -dusage= parameter, it will try to
> > balance everything, which takes a long time).
> >
> >> [jakobt@soda ~]$ dmesg | grep tail -n 2
> >> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
> >> [1113507.641752] btrfs: 1866 enospc errors during balance
> >
> >    Although, that said... it looks like it's tried every block group
> > and failed with each one, so my suggestion above may not work in this
> > instance.
> >
> >    Let us know what happens with the balance command above anyway
> > (dmesg output is useful information at this point). If that doesn't
> > help, then we'll probably need to take a metadata image and throw it
> > in josef's direction, where he will start crying at having to deal
> > with enospc problems again. :)
> >
> >    Hugo.
> >

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- If you're not part of the solution, you're part ---         
                           of the precipiate.                            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: no space left on device.

[Kenneth Johansson]

On 11/02/2012 04:54 PM, Kyle Gates wrote:
>> So I have ended up in a state where I can't delete files with rm.
>>
>> the error I get is no space on device. however I'm not even close to empty.
>> /dev/sdb1 38G 27G 9.5G 75%
>> there is about 800k files/dirs in this filesystem
>>
>> extra strange is that I can in another directory create and delete files.
>>
>> So I tried pretty much all I could google my way to but problem
>> persisted. So I decided to do a backup and a format. But when the backup
>> was done I tried one more time and now it was possible to delete the
>> directory and all content?
>>
>> using the 3.5 kernel in ubuntu 12.10. Is this a known issue ? is it
>> fixed in later kernels?
>>
>> fsck /btrfs scrub and kernel log. nothing indicate any problem of any kind.
>>
> First let's see the output of:
> btrfs fi df /mountpoint
>
> You're probably way over allocated in metadata so a balance should help:
> btrfs bal start -m /mountpoint
> or omit the -m option to run a full balance. 		
I had to use the disk for work so I could not be stuck in that situation 
and thus had to nuke the disk.
maybe I can recreate the state I put back mostly the same data again 
minus some stuff I did not really need.

So under what circumstance can this actually happen ? why could I 
remove/add files in one directory and not another? its the same 
partition. And also the filesystem was just a few days old and no 
snapshots or anything, well I do use lzo compression but other than that 
default values was used to create it.

Re: no space left on device.

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1468 bytes --]

On Fri, Nov 02, 2012 at 10:54:47AM -0500, Kyle Gates wrote:
> > So I have ended up in a state where I can't delete files with rm.
> >
> > the error I get is no space on device. however I'm not even close to empty.
> > /dev/sdb1 38G 27G 9.5G 75%
> > there is about 800k files/dirs in this filesystem
> >
> > extra strange is that I can in another directory create and delete files.
> >
> > So I tried pretty much all I could google my way to but problem
> > persisted. So I decided to do a backup and a format. But when the backup
> > was done I tried one more time and now it was possible to delete the
> > directory and all content?
> >
> > using the 3.5 kernel in ubuntu 12.10. Is this a known issue ? is it
> > fixed in later kernels?
> >
> > fsck /btrfs scrub and kernel log. nothing indicate any problem of any kind.
> >
> 
> First let's see the output of:
> btrfs fi df /mountpoint
> 
> You're probably way over allocated in metadata so a balance should help:
> btrfs bal start -m /mountpoint
> or omit the -m option to run a full balance.

   Or, better, -musage=5 (or 1), which will do even less work.

... but let's see the btrfs fi df output first. Could you also add the
output of "btrfs fi show" (no parameters), please?

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
            --- You're never alone with a rubber duck... ---             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

RE: no space left on device.

[Kyle Gates]

> So I have ended up in a state where I can't delete files with rm.
>
> the error I get is no space on device. however I'm not even close to empty.
> /dev/sdb1 38G 27G 9.5G 75%
> there is about 800k files/dirs in this filesystem
>
> extra strange is that I can in another directory create and delete files.
>
> So I tried pretty much all I could google my way to but problem
> persisted. So I decided to do a backup and a format. But when the backup
> was done I tried one more time and now it was possible to delete the
> directory and all content?
>
> using the 3.5 kernel in ubuntu 12.10. Is this a known issue ? is it
> fixed in later kernels?
>
> fsck /btrfs scrub and kernel log. nothing indicate any problem of any kind.
>

First let's see the output of:
btrfs fi df /mountpoint

You're probably way over allocated in metadata so a balance should help:
btrfs bal start -m /mountpoint
or omit the -m option to run a full balance. 		 	   		  

no space left on device.

[Kenneth Johansson]

So I have ended up in a state where I can't delete files with rm.

the error I get is no space on device. however I'm not even close to empty.
/dev/sdb1 38G 27G 9.5G 75%
there is about 800k files/dirs in this filesystem

extra strange is that I can in another directory create and delete files.

So I tried pretty much all I could google my way to but problem 
persisted. So I decided to do a backup and a format. But when the backup 
was done I tried one more time and now it was possible to delete the 
directory and all content?

using the 3.5 kernel in ubuntu 12.10. Is this a known issue ? is it 
fixed in later kernels?

fsck /btrfs scrub and kernel log. nothing indicate any problem of any kind.

Re: no space left on device

[Martin Steigerwald]

Am Freitag, 3. August 2012 schrieb Mark Marshall:
> Hi,

Hi Mark,

> I am new to btrfs, and just installed a new system with SLED 11 SP2 a
> few days ago.

Now you have a nice SLED 11 SP 2 with official BTRFS support from SUSE. 
Did you consider to ask SUSE support? ;)

> However the system seems to be in a real sad state now, saying there
> is no free space left on the device even though there is about 8GB
> left on the / filesystem.
> 
> A defragment works sometimes but then it goes back to the original
> state a day or two later. Other times the command just won't respond.
> 
> I've had this happen with 2-3 systems now, is there a solution? I
> can't keep going around to users machines and defragmenting all the
> time !

First read and understand

https://btrfs.wiki.kernel.org/index.php/FAQ#Aaargh.21_My_filesystem_is_full.2C_and_I.27ve_put_almost_nothing_into_it.21

plus the following FAQs in there.

Then come back with more information.

Possible a btrfs balance will work to make some trees smaller, but this 
depends on the current state of your BTRFS. The information you provided 
is not nearly enough for an exact diagnosis. A defragment likely won´t 
help tough.

Thanks,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: no space left on device

[Chris Samuel]

Hi Mark,

On 08/03/2012 08:05 PM, Mark Marshall wrote:

> I am new to btrfs, and just installed a new system with SLED 11 SP2 a
> few days ago.

Looking at the release notes for SLED11 SP2 it appears to ship with an 
ancient kernel (in btrfs terms), 3.0.10.  You will want to upgrade to 
*at least* a 3.2.x kernel (I use Kubuntu 12.04 on my work laptop with 
btrfs /home and had no issues with its 3.2.x), though it's likely you'll 
want to go straight to the current 3.5.x kernel.

Best of luck!
Chris

no space left on device

[Mark Marshall]

Hi,

I am new to btrfs, and just installed a new system with SLED 11 SP2 a
few days ago.

However the system seems to be in a real sad state now, saying there
is no free space left on the device even though there is about 8GB
left on the / filesystem.

A defragment works sometimes but then it goes back to the original
state a day or two later. Other times the command just won't respond.

I've had this happen with 2-3 systems now, is there a solution? I
can't keep going around to users machines and defragmenting all the
time !

Thanks,

Mark

Re: No space left on device

[André Øien Langvand]

[-- Attachment #1.1: Type: text/plain, Size: 6994 bytes --]

Thank you both for informative replies, much appreciated.

I ended up doing the tedious job of shuffling data around to get a more
reasonable distance to ENOSPC, as there was no way I could free up the
needed contiguous free space with my dataset.


Thanks,

André

On 20 June 2012 08:07, Dave Chinner <david@fromorbit.com> wrote:

> On Tue, Jun 19, 2012 at 07:36:24AM -0500, Geoffrey Wehrman wrote:
> > On Tue, Jun 19, 2012 at 02:05:34PM +0200, André Øien Langvand wrote:
> > | Hi,
> > |
> > | I know there are quite a few posts regarding similar issues around,
> but I
> > | can't seem to find a solution or at least an answer to why this is
> > | happening in my case, so I thought I'd try the mailing list and I hope
> > | that's okay.
> > |
> > | We have 2 file servers with identical hardware and identical
> configuration
> > | (Dell R610's, H800 controllers, MD1200 DAS, RAID-5) set up with rsync
> to
> > | mirror the contents. The content is music in several formats (from PCM
> WAV
> > | to 64kbit AAC previews), which means file sizes of about 1 - 40mb. Both
> > | systems running SLES 11 SP1. Same kernel (2.6.32.59-0.3-default), same
> > | xfsprogs version (xfsprogs-3.1.1-0.1.36).
> > |
> > | My example partition on the source now has 9.9G (of 9.1T) available
> space
> > | and still doesn't report the drive as full. On the destination,
> however, it
> > | wont allow me to use any of the remaining 51G. This is obviously a
> problem
> > | when trying to do mirroring.
> > |
> > | Both file systems have been mounted with inode64 option since first
> mount,
> > | there are plenty of inodes available and I've also verified that there
> are
> > | noe sparse files (find -type f -printf "%S\t%p\n" 2>/dev/null | gawk
> '{if
> > | ($1 < 1.0) print $1 $2}'), just in case.
> > |
> > | I have tried repairing (xfs_repair), defragmenting (xfs_fsr) and alter
> > | imaxpct without any luck. Rsync is run like this: # ionice -c3 rsync
> -rv
> > | --size-only --progress --delete-before --inplace.
> > |
> > |
> > | More detailed information on source file system:
> > |
> > | # df -k | grep sdg1
> > | /dev/sdg1            9762777052 9752457156  10319896 100%
> /content/raid31
> > |
> > | # df -i | grep sdg1
> > | /dev/sdg1            7471884 2311914 5159970   31% /content/raid31
> > |
> > | # xfs_info /dev/sdg1
> > | meta-data=/dev/sdg1              isize=2048   agcount=10,
> agsize=268435424
> > | blks
> > |          =                       sectsz=512   attr=2
> > | data     =                       bsize=4096   blocks=2441215991,
> imaxpct=5
> > |          =                       sunit=16     swidth=80 blks
> > | naming   =version 2              bsize=4096   ascii-ci=0
> > | log      =internal               bsize=4096   blocks=521728, version=2
> > |          =                       sectsz=512   sunit=16 blks,
> lazy-count=1
> > | realtime =none                   extsz=4096   blocks=0, rtextents=0
> > |
> > | # xfs_db -r "-c freesp -s" /dev/sdg1
> > |    from      to extents  blocks    pct
> > |       1       1   69981   69981   2.99
> > |       2       3  246574  559149  23.86
> > |       4       7  315038 1707929  72.88
> > |       8      15     561 6374 0.27
> > | total free extents 632154
> > | total free blocks 2343433
> > | average free extent size 3.70706
> > |
> > |
> > |
> > | More detailed information on destination file system:
> > |
> > | # df -k | grep sdj1
> > | /dev/sdj1            9762777052 9710148076  52628976 100%
> /content/sg08/vd08
> > |
> > | # df -i | grep sdj1
> > | /dev/sdj1            28622264 2307776 26314488    9% /content/sg08/vd08
> > |
> > | # xfs_info /dev/sdj1
> > | meta-data=/dev/sdj1              isize=2048   agcount=10,
> agsize=268435424
> > | blks
> > |          =                       sectsz=512   attr=2
> > | data     =                       bsize=4096   blocks=2441215991,
> imaxpct=5
> > |          =                       sunit=16     swidth=80 blks
> > | naming   =version 2              bsize=4096   ascii-ci=0
> > | log      =internal               bsize=4096   blocks=521728, version=2
> > |          =                       sectsz=512   sunit=16 blks,
> lazy-count=1
> > | realtime =none                   extsz=4096   blocks=0, rtextents=0
> > |
> > | # xfs_db -r "-c freesp -s" /dev/sdj1
> > |    from      to extents  blocks    pct
> > |       1       1   81761   81761   0.62
> > |       2       3  530258 1147719   8.73
> > |       4       7  675864 3551039  27.01
> > |       8      15  743089 8363043  63.62
> > |      16      31     102    1972   0.02
> > | total free extents 2031074
> > | total free blocks 13145534
> > | average free extent size 6.47221
> > |
> > |
> > | I would be grateful if anyone could shed some light on why this is
> > | happening or maybe even provide a solution.
> >
> > You are using 2 KiB inodes, so an inode cluster (64 inodes) requires
> > 128 KiB of contiguous space on disk.  The freesp output above shows that
> > the largest possible contiguous free space chunk available is 31 * 4 KiB
> > or 4 KiB short of 128 KiB.  You don't have enough contiguous space to
> > create a new inode cluster, and your existing inodes are likely all
> > used.  This can be verified using xfs_db:
> >       xfs_db -r -c "sb" -c "p ifree" /dev/sdj1
> >
> > xfs_fsr does not defragment free space, it only makes the problem worse.
> > A possible solution:
> >   1.  mount the filesystem with the ikeep mount option
> >   2.  delete a few large files to free up some contiguous space
>
> large -contiguous- files. It's likely any files written recently
> will be as fragmented as the free space....
>
> >   3.  create a few thousand files to "preallocate" inodes
> >   4.  delete the newly created files
>
> That will work for a while, but it's really just a temporary
> workaround until those "preallocated" inodes are exhausted. Normally
> to recover from this situation you need to free 15-20% of the disk
> space to allow sufficiently large contiguous free space extents to
> reform naturally and allow the allocator to work at full efficiency
> again....
>
> > The ikeep mount option will prevent the space for inodes from being
> > reused for other purposes.
>
> The problem with using ikeep is that the remaining empty inode
> chunks prevent free space from defragmenting itself fully as you
> remove files from the filesystem.
>
> Realistically, I think the problem is that you are running your
> filesystems at near ENOSPC for extended periods of time. That is
> guaranteed to fragment free space and any files that are written
> when the filesytem is in this condition. As Geoffrey has said -
> xfs_fsr will not fix your problems - only changing the way you use
> your storage will prevent the problem from occurring again.
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> david@fromorbit.com
>

[-- Attachment #1.2: Type: text/html, Size: 8600 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device

[Dave Chinner]

On Tue, Jun 19, 2012 at 07:36:24AM -0500, Geoffrey Wehrman wrote:
> On Tue, Jun 19, 2012 at 02:05:34PM +0200, André Øien Langvand wrote:
> | Hi,
> | 
> | I know there are quite a few posts regarding similar issues around, but I
> | can't seem to find a solution or at least an answer to why this is
> | happening in my case, so I thought I'd try the mailing list and I hope
> | that's okay.
> | 
> | We have 2 file servers with identical hardware and identical configuration
> | (Dell R610's, H800 controllers, MD1200 DAS, RAID-5) set up with rsync to
> | mirror the contents. The content is music in several formats (from PCM WAV
> | to 64kbit AAC previews), which means file sizes of about 1 - 40mb. Both
> | systems running SLES 11 SP1. Same kernel (2.6.32.59-0.3-default), same
> | xfsprogs version (xfsprogs-3.1.1-0.1.36).
> | 
> | My example partition on the source now has 9.9G (of 9.1T) available space
> | and still doesn't report the drive as full. On the destination, however, it
> | wont allow me to use any of the remaining 51G. This is obviously a problem
> | when trying to do mirroring.
> | 
> | Both file systems have been mounted with inode64 option since first mount,
> | there are plenty of inodes available and I've also verified that there are
> | noe sparse files (find -type f -printf "%S\t%p\n" 2>/dev/null | gawk '{if
> | ($1 < 1.0) print $1 $2}'), just in case.
> | 
> | I have tried repairing (xfs_repair), defragmenting (xfs_fsr) and alter
> | imaxpct without any luck. Rsync is run like this: # ionice -c3 rsync -rv
> | --size-only --progress --delete-before --inplace.
> | 
> | 
> | More detailed information on source file system:
> | 
> | # df -k | grep sdg1
> | /dev/sdg1            9762777052 9752457156  10319896 100% /content/raid31
> | 
> | # df -i | grep sdg1
> | /dev/sdg1            7471884 2311914 5159970   31% /content/raid31
> | 
> | # xfs_info /dev/sdg1
> | meta-data=/dev/sdg1              isize=2048   agcount=10, agsize=268435424
> | blks
> |          =                       sectsz=512   attr=2
> | data     =                       bsize=4096   blocks=2441215991, imaxpct=5
> |          =                       sunit=16     swidth=80 blks
> | naming   =version 2              bsize=4096   ascii-ci=0
> | log      =internal               bsize=4096   blocks=521728, version=2
> |          =                       sectsz=512   sunit=16 blks, lazy-count=1
> | realtime =none                   extsz=4096   blocks=0, rtextents=0
> | 
> | # xfs_db -r "-c freesp -s" /dev/sdg1
> |    from      to extents  blocks    pct
> |       1       1   69981   69981   2.99
> |       2       3  246574  559149  23.86
> |       4       7  315038 1707929  72.88
> |       8      15     561    6374   0.27
> | total free extents 632154
> | total free blocks 2343433
> | average free extent size 3.70706
> | 
> | 
> | 
> | More detailed information on destination file system:
> | 
> | # df -k | grep sdj1
> | /dev/sdj1            9762777052 9710148076  52628976 100% /content/sg08/vd08
> | 
> | # df -i | grep sdj1
> | /dev/sdj1            28622264 2307776 26314488    9% /content/sg08/vd08
> | 
> | # xfs_info /dev/sdj1
> | meta-data=/dev/sdj1              isize=2048   agcount=10, agsize=268435424
> | blks
> |          =                       sectsz=512   attr=2
> | data     =                       bsize=4096   blocks=2441215991, imaxpct=5
> |          =                       sunit=16     swidth=80 blks
> | naming   =version 2              bsize=4096   ascii-ci=0
> | log      =internal               bsize=4096   blocks=521728, version=2
> |          =                       sectsz=512   sunit=16 blks, lazy-count=1
> | realtime =none                   extsz=4096   blocks=0, rtextents=0
> | 
> | # xfs_db -r "-c freesp -s" /dev/sdj1
> |    from      to extents  blocks    pct
> |       1       1   81761   81761   0.62
> |       2       3  530258 1147719   8.73
> |       4       7  675864 3551039  27.01
> |       8      15  743089 8363043  63.62
> |      16      31     102    1972   0.02
> | total free extents 2031074
> | total free blocks 13145534
> | average free extent size 6.47221
> | 
> | 
> | I would be grateful if anyone could shed some light on why this is
> | happening or maybe even provide a solution.
> 
> You are using 2 KiB inodes, so an inode cluster (64 inodes) requires
> 128 KiB of contiguous space on disk.  The freesp output above shows that
> the largest possible contiguous free space chunk available is 31 * 4 KiB
> or 4 KiB short of 128 KiB.  You don't have enough contiguous space to
> create a new inode cluster, and your existing inodes are likely all
> used.  This can be verified using xfs_db: 
> 	xfs_db -r -c "sb" -c "p ifree" /dev/sdj1
> 
> xfs_fsr does not defragment free space, it only makes the problem worse.
> A possible solution:
>   1.  mount the filesystem with the ikeep mount option
>   2.  delete a few large files to free up some contiguous space

large -contiguous- files. It's likely any files written recently
will be as fragmented as the free space....

>   3.  create a few thousand files to "preallocate" inodes
>   4.  delete the newly created files

That will work for a while, but it's really just a temporary
workaround until those "preallocated" inodes are exhausted. Normally
to recover from this situation you need to free 15-20% of the disk
space to allow sufficiently large contiguous free space extents to
reform naturally and allow the allocator to work at full efficiency
again....

> The ikeep mount option will prevent the space for inodes from being
> reused for other purposes.

The problem with using ikeep is that the remaining empty inode
chunks prevent free space from defragmenting itself fully as you
remove files from the filesystem.

Realistically, I think the problem is that you are running your
filesystems at near ENOSPC for extended periods of time. That is
guaranteed to fragment free space and any files that are written
when the filesytem is in this condition. As Geoffrey has said -
xfs_fsr will not fix your problems - only changing the way you use
your storage will prevent the problem from occurring again.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device

[Geoffrey Wehrman]

On Tue, Jun 19, 2012 at 02:05:34PM +0200, André Øien Langvand wrote:
| Hi,
| 
| I know there are quite a few posts regarding similar issues around, but I
| can't seem to find a solution or at least an answer to why this is
| happening in my case, so I thought I'd try the mailing list and I hope
| that's okay.
| 
| We have 2 file servers with identical hardware and identical configuration
| (Dell R610's, H800 controllers, MD1200 DAS, RAID-5) set up with rsync to
| mirror the contents. The content is music in several formats (from PCM WAV
| to 64kbit AAC previews), which means file sizes of about 1 - 40mb. Both
| systems running SLES 11 SP1. Same kernel (2.6.32.59-0.3-default), same
| xfsprogs version (xfsprogs-3.1.1-0.1.36).
| 
| My example partition on the source now has 9.9G (of 9.1T) available space
| and still doesn't report the drive as full. On the destination, however, it
| wont allow me to use any of the remaining 51G. This is obviously a problem
| when trying to do mirroring.
| 
| Both file systems have been mounted with inode64 option since first mount,
| there are plenty of inodes available and I've also verified that there are
| noe sparse files (find -type f -printf "%S\t%p\n" 2>/dev/null | gawk '{if
| ($1 < 1.0) print $1 $2}'), just in case.
| 
| I have tried repairing (xfs_repair), defragmenting (xfs_fsr) and alter
| imaxpct without any luck. Rsync is run like this: # ionice -c3 rsync -rv
| --size-only --progress --delete-before --inplace.
| 
| 
| More detailed information on source file system:
| 
| # df -k | grep sdg1
| /dev/sdg1            9762777052 9752457156  10319896 100% /content/raid31
| 
| # df -i | grep sdg1
| /dev/sdg1            7471884 2311914 5159970   31% /content/raid31
| 
| # xfs_info /dev/sdg1
| meta-data=/dev/sdg1              isize=2048   agcount=10, agsize=268435424
| blks
|          =                       sectsz=512   attr=2
| data     =                       bsize=4096   blocks=2441215991, imaxpct=5
|          =                       sunit=16     swidth=80 blks
| naming   =version 2              bsize=4096   ascii-ci=0
| log      =internal               bsize=4096   blocks=521728, version=2
|          =                       sectsz=512   sunit=16 blks, lazy-count=1
| realtime =none                   extsz=4096   blocks=0, rtextents=0
| 
| # xfs_db -r "-c freesp -s" /dev/sdg1
|    from      to extents  blocks    pct
|       1       1   69981   69981   2.99
|       2       3  246574  559149  23.86
|       4       7  315038 1707929  72.88
|       8      15     561    6374   0.27
| total free extents 632154
| total free blocks 2343433
| average free extent size 3.70706
| 
| 
| 
| More detailed information on destination file system:
| 
| # df -k | grep sdj1
| /dev/sdj1            9762777052 9710148076  52628976 100% /content/sg08/vd08
| 
| # df -i | grep sdj1
| /dev/sdj1            28622264 2307776 26314488    9% /content/sg08/vd08
| 
| # xfs_info /dev/sdj1
| meta-data=/dev/sdj1              isize=2048   agcount=10, agsize=268435424
| blks
|          =                       sectsz=512   attr=2
| data     =                       bsize=4096   blocks=2441215991, imaxpct=5
|          =                       sunit=16     swidth=80 blks
| naming   =version 2              bsize=4096   ascii-ci=0
| log      =internal               bsize=4096   blocks=521728, version=2
|          =                       sectsz=512   sunit=16 blks, lazy-count=1
| realtime =none                   extsz=4096   blocks=0, rtextents=0
| 
| # xfs_db -r "-c freesp -s" /dev/sdj1
|    from      to extents  blocks    pct
|       1       1   81761   81761   0.62
|       2       3  530258 1147719   8.73
|       4       7  675864 3551039  27.01
|       8      15  743089 8363043  63.62
|      16      31     102    1972   0.02
| total free extents 2031074
| total free blocks 13145534
| average free extent size 6.47221
| 
| 
| I would be grateful if anyone could shed some light on why this is
| happening or maybe even provide a solution.

You are using 2 KiB inodes, so an inode cluster (64 inodes) requires
128 KiB of contiguous space on disk.  The freesp output above shows that
the largest possible contiguous free space chunk available is 31 * 4 KiB
or 4 KiB short of 128 KiB.  You don't have enough contiguous space to
create a new inode cluster, and your existing inodes are likely all
used.  This can be verified using xfs_db: 
	xfs_db -r -c "sb" -c "p ifree" /dev/sdj1

xfs_fsr does not defragment free space, it only makes the problem worse.
A possible solution:
  1.  mount the filesystem with the ikeep mount option
  2.  delete a few large files to free up some contiguous space
  3.  create a few thousand files to "preallocate" inodes
  4.  delete the newly created files
The ikeep mount option will prevent the space for inodes from being
reused for other purposes.


-- 
Geoffrey Wehrman
SGI Building 10                             Office: (651)683-5496
2750 Blue Water Road                           Fax: (651)683-5098
Eagan, MN 55121                             E-mail: gwehrman@sgi.com
	  http://www.sgi.com/products/storage/software/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

No space left on device

[André Øien Langvand]

[-- Attachment #1.1: Type: text/plain, Size: 3868 bytes --]

Hi,

I know there are quite a few posts regarding similar issues around, but I
can't seem to find a solution or at least an answer to why this is
happening in my case, so I thought I'd try the mailing list and I hope
that's okay.

We have 2 file servers with identical hardware and identical configuration
(Dell R610's, H800 controllers, MD1200 DAS, RAID-5) set up with rsync to
mirror the contents. The content is music in several formats (from PCM WAV
to 64kbit AAC previews), which means file sizes of about 1 - 40mb. Both
systems running SLES 11 SP1. Same kernel (2.6.32.59-0.3-default), same
xfsprogs version (xfsprogs-3.1.1-0.1.36).

My example partition on the source now has 9.9G (of 9.1T) available space
and still doesn't report the drive as full. On the destination, however, it
wont allow me to use any of the remaining 51G. This is obviously a problem
when trying to do mirroring.

Both file systems have been mounted with inode64 option since first mount,
there are plenty of inodes available and I've also verified that there are
noe sparse files (find -type f -printf "%S\t%p\n" 2>/dev/null | gawk '{if
($1 < 1.0) print $1 $2}'), just in case.

I have tried repairing (xfs_repair), defragmenting (xfs_fsr) and alter
imaxpct without any luck. Rsync is run like this: # ionice -c3 rsync -rv
--size-only --progress --delete-before --inplace.


More detailed information on source file system:

# df -k | grep sdg1
/dev/sdg1            9762777052 9752457156  10319896 100% /content/raid31

# df -i | grep sdg1
/dev/sdg1            7471884 2311914 5159970   31% /content/raid31

# xfs_info /dev/sdg1
meta-data=/dev/sdg1              isize=2048   agcount=10, agsize=268435424
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2441215991, imaxpct=5
         =                       sunit=16     swidth=80 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# xfs_db -r "-c freesp -s" /dev/sdg1
   from      to extents  blocks    pct
      1       1   69981   69981   2.99
      2       3  246574  559149  23.86
      4       7  315038 1707929  72.88
      8      15     561    6374   0.27
total free extents 632154
total free blocks 2343433
average free extent size 3.70706



More detailed information on destination file system:

# df -k | grep sdj1
/dev/sdj1            9762777052 9710148076  52628976 100% /content/sg08/vd08

# df -i | grep sdj1
/dev/sdj1            28622264 2307776 26314488    9% /content/sg08/vd08

# xfs_info /dev/sdj1
meta-data=/dev/sdj1              isize=2048   agcount=10, agsize=268435424
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2441215991, imaxpct=5
         =                       sunit=16     swidth=80 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# xfs_db -r "-c freesp -s" /dev/sdj1
   from      to extents  blocks    pct
      1       1   81761   81761   0.62
      2       3  530258 1147719   8.73
      4       7  675864 3551039  27.01
      8      15  743089 8363043  63.62
     16      31     102    1972   0.02
total free extents 2031074
total free blocks 13145534
average free extent size 6.47221


I would be grateful if anyone could shed some light on why this is
happening or maybe even provide a solution.


Thank you,

André

[-- Attachment #1.2: Type: text/html, Size: 5266 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: no space left on device

[Erik Logtenberg]

>> That patch is released on this list on january 26th, called "Fix
>> balance panic". If you experience the same, you could try applying
>> this patch too. There is no (pre-built) kernel yet with this patch
>> already applied.
> 
> Maybe it will help. But my actual problem is: I cannot mount the btrfs  
> system (it leads to "kernel panic"). And starting "btrfs filesystem  
> balance ..." requires a mounted btrfs system.

Yeah sorry, I hadn't noticed that. No, that patch will certainly not fix
a filesystem you can't mount to begin with. Can't help you out.

 - Erik

Re: no space left on device

[Helmut Hullen]

Hallo, Erik,

Du meintest am 08.02.11:

[...]

> Fortunately Zheng Yan coded a patch, which I
> applied to 2.6.38-rc2, so that I could finally run - and finish - the
> balance operation.
> Additionally this version of btrfs is more clever in not wasting the
> free space to begin with, hence making future rebalances for
> reclaiming free space no longer necessary.

That patch isn't yet included in 2.6.38-rc4.

Viele Gruesse!
Helmut

Re: no space left on device

[Helmut Hullen]

Hallo, Erik,

Du meintest am 08.02.11:

>>> But the next step is to update kernel and btrfs-progs from git.

>> Maybe that doesn't help now. I'm working with kernel 2.6.37 and
>> kernel 2.6.38-rc2, and I've got big problems.

[...]

> Fortunately Zheng Yan coded a patch, which I
> applied to 2.6.38-rc2, so that I could finally run - and finish - the
> balance operation.

[...]

> That patch is released on this list on january 26th, called "Fix
> balance panic". If you experience the same, you could try applying
> this patch too. There is no (pre-built) kernel yet with this patch
> already applied.

Maybe it will help. But my actual problem is: I cannot mount the btrfs  
system (it leads to "kernel panic"). And starting "btrfs filesystem  
balance ..." requires a mounted btrfs system.

I'll see ...

Viele Gruesse!
Helmut

Re: no space left on device

[Erik Logtenberg]

>> But the next step is to update kernel and btrfs-progs from git.
> 
> Maybe that doesn't help now. I'm working with kernel 2.6.37 and kernel  
> 2.6.38-rc2, and I've got big problems.

I had to install at least 2.6.37 to have a kernel with an advanced
enough balance feature to actually reclaim the free space and report it
correctly too.
Still, I had so much kernel crashes during this balance operation that
even after numerous retries I still hadn't once completed it (on a FS of
merely 300GB). Fortunately Zheng Yan coded a patch, which I applied to
2.6.38-rc2, so that I could finally run - and finish - the balance
operation.
Additionally this version of btrfs is more clever in not wasting the
free space to begin with, hence making future rebalances for reclaiming
free space no longer necessary.

That patch is released on this list on january 26th, called "Fix balance
panic". If you experience the same, you could try applying this patch
too. There is no (pre-built) kernel yet with this patch already applied.

  - Erik.

Re: no space left on device

[Helmut Hullen]

Hallo, Leonidas,

Du meintest am 08.02.11:

[...]

>> I have to be blunt, blaming your problems on debian isn't terribly
>> classy. =A0The "oooo, shiny!" reflex is your fault, not debian's.

> Well you are right it is my problem and yeah I wanted to test the
> "new shiny" Debian 6 with officially btrfs supported.

It's my problem too - I have to restore about 1.5 TByte somehow ...

> For the moment I apt-get clean to get some space since my / was
> updating the KDE (so it had a lot of cache files there)
> Reserved something like 300Mb and then vlextend the /

> But the next step is to update kernel and btrfs-progs from git.

Maybe that doesn't help now. I'm working with kernel 2.6.37 and kernel =
=20
2.6.38-rc2, and I've got big problems.

Viele Gruesse!
Helmut
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Leonidas Spyropoulos]

On 8 February 2011 09:31, cwillu <cwillu@cwillu.com> wrote:
> On Tue, Feb 8, 2011 at 3:00 AM, Leonidas Spyropoulos
> <artafinde@gmail.com> wrote:
>> On Feb 8, 2011 12:09 AM, "C Anthony Risinger" <anthony@extof.me> wro=
te:
>>>
>>> On Mon, Feb 7, 2011 at 3:21 PM, Leonidas Spyropoulos
>>> <artafinde@gmail.com> wrote:
>>> > Hey all,
>>> >
>>> > I run into no space left on device on a virtualbox
>>> >
>>> > After installing Debian 6 on a virtual machine
>>> > I tried installing the KDE desktop
>>> >
>>> > The system HDD is 8Gb
>>> > Both root (/) and /home are btrfs
>>> > over LVM.
>>> >
>>> > While installing the packages I run into:
>>> >
>>> > no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_us=
ed, 0
>>> > bytes_reserved,
>>> > 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
>>> >
>>> > df shows only 74% used space on /
>>> >
>>> > kernel used: stock debian 6 2.6.32-5-686
>>> >
>>> > At the moment I cannot access it with normal boot, only recovery =
mode.
>>> >
>>> > I can provide whatever info you would like as long as you think o=
f a
>>> > way to load the normal system and not the recovery mode.
>>>
>>> IIRC .32 has all sorts of ENOSPC problems; I think this was serious=
ly
>>> tackled in kernels > .32... this kernel was only declared ready for
>>> "early adopters", with an "expect issues" disclaimer.
>>>
>>> The btrfs-tools in squeeze is probably so old you may not even have
>>> the `btrfs` binary, but I don't run debian so I'm not sure there...
>>> not really a solution probably for you, but I wouldn't run that ker=
nel
>>> if using btrfs.
>>>
>>> C Anthony
>>
>> Hey all,
>>
>> Thanks for all the answers.
>>
>> The problem is that I cannot login to the system.only recovery mode =
works,
>> and there btrfs command is not there as you imagined.
>>
>> I will try though ssh but I don't think it's installed by default an=
d I
>> cannot install it.
>>
>> So the next step is try from recovery console of debian live cd, whi=
ch still
>> has the really old tools...
>>
>> I think this is quite some serious issue but generally all debian's =
fault
>> adopting a btrfs file system support on a 2.6.32 kernel and without
>> btrfs-progs on some decent version.
>>
>> I'll update when possible.
>> Please throw any other alternatives my way anyone.
>
> I have to be blunt, blaming your problems on debian isn't terribly
> classy. =A0The "oooo, shiny!" reflex is your fault, not debian's.

Well you are right it is my problem and yeah I wanted to test the "new
shiny" Debian 6 with
officially btrfs supported.

=46or the moment I apt-get clean to get some space since my / was
updating the KDE (so it had a lot of cache files there)
Reserved something like 300Mb and then vlextend the /

But the next step is to update kernel and btrfs-progs from git.

Thanks for answers

>
> Download and install a prebuilt 2.6.35 or later kernel into your /boo=
t
> via a livecd or whatever, unpack and add the btrfs command to the
> initramfs for that kernel, boot up into that initramfs with the kerne=
l
> option "break=3Dpremount", and fix the rootfs from the busybox prompt=
=2E

The above seems unknown to me. Could you elaborate a bit please?

>
> Alternatively, an ubuntu natty alpha livecd has a 2.6.38 kernel, and
> you can install mostly up-to-date btrfs tools into that environment.
> I'm sure debian has something similar available.

I was wondering how to fix this without liveCD.

>
> --Carey Underwood
>

--=20
Caution: breathing may be hazardous to your health.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[cwillu]

On Tue, Feb 8, 2011 at 3:00 AM, Leonidas Spyropoulos
<artafinde@gmail.com> wrote:
> On Feb 8, 2011 12:09 AM, "C Anthony Risinger" <anthony@extof.me> wrote:
>>
>> On Mon, Feb 7, 2011 at 3:21 PM, Leonidas Spyropoulos
>> <artafinde@gmail.com> wrote:
>> > Hey all,
>> >
>> > I run into no space left on device on a virtualbox
>> >
>> > After installing Debian 6 on a virtual machine
>> > I tried installing the KDE desktop
>> >
>> > The system HDD is 8Gb
>> > Both root (/) and /home are btrfs
>> > over LVM.
>> >
>> > While installing the packages I run into:
>> >
>> > no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_used, 0
>> > bytes_reserved,
>> > 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
>> >
>> > df shows only 74% used space on /
>> >
>> > kernel used: stock debian 6 2.6.32-5-686
>> >
>> > At the moment I cannot access it with normal boot, only recovery mode.
>> >
>> > I can provide whatever info you would like as long as you think of a
>> > way to load the normal system and not the recovery mode.
>>
>> IIRC .32 has all sorts of ENOSPC problems; I think this was seriously
>> tackled in kernels > .32... this kernel was only declared ready for
>> "early adopters", with an "expect issues" disclaimer.
>>
>> The btrfs-tools in squeeze is probably so old you may not even have
>> the `btrfs` binary, but I don't run debian so I'm not sure there...
>> not really a solution probably for you, but I wouldn't run that kernel
>> if using btrfs.
>>
>> C Anthony
>
> Hey all,
>
> Thanks for all the answers.
>
> The problem is that I cannot login to the system.only recovery mode works,
> and there btrfs command is not there as you imagined.
>
> I will try though ssh but I don't think it's installed by default and I
> cannot install it.
>
> So the next step is try from recovery console of debian live cd, which still
> has the really old tools...
>
> I think this is quite some serious issue but generally all debian's fault
> adopting a btrfs file system support on a 2.6.32 kernel and without
> btrfs-progs on some decent version.
>
> I'll update when possible.
> Please throw any other alternatives my way anyone.

I have to be blunt, blaming your problems on debian isn't terribly
classy.  The "oooo, shiny!" reflex is your fault, not debian's.

Download and install a prebuilt 2.6.35 or later kernel into your /boot
via a livecd or whatever, unpack and add the btrfs command to the
initramfs for that kernel, boot up into that initramfs with the kernel
option "break=premount", and fix the rootfs from the busybox prompt.

Alternatively, an ubuntu natty alpha livecd has a 2.6.38 kernel, and
you can install mostly up-to-date btrfs tools into that environment.
I'm sure debian has something similar available.

--Carey Underwood

Re: no space left on device

[Leonidas Spyropoulos]

On Feb 8, 2011 12:09 AM, "C Anthony Risinger" <anthony@extof.me> wrote:
>
> On Mon, Feb 7, 2011 at 3:21 PM, Leonidas Spyropoulos
> <artafinde@gmail.com> wrote:
> > Hey all,
> >
> > I run into no space left on device on a virtualbox
> >
> > After installing Debian 6 on a virtual machine
> > I tried installing the KDE desktop
> >
> > The system HDD is 8Gb
> > Both root (/) and /home are btrfs
> > over LVM.
> >
> > While installing the packages I run into:
> >
> > no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_used, 0
> > bytes_reserved,
> > 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
> >
> > df shows only 74% used space on /
> >
> > kernel used: stock debian 6 2.6.32-5-686
> >
> > At the moment I cannot access it with normal boot, only recovery mode.
> >
> > I can provide whatever info you would like as long as you think of a
> > way to load the normal system and not the recovery mode.
>
> IIRC .32 has all sorts of ENOSPC problems; I think this was seriously
> tackled in kernels > .32... this kernel was only declared ready for
> "early adopters", with an "expect issues" disclaimer.
>
> The btrfs-tools in squeeze is probably so old you may not even have
> the `btrfs` binary, but I don't run debian so I'm not sure there...
> not really a solution probably for you, but I wouldn't run that kernel
> if using btrfs.
>
> C Anthony

Hey all,

Thanks for all the answers.

The problem is that I cannot login to the system.only recovery mode works,
and there btrfs command is not there as you imagined.

I will try though ssh but I don't think it's installed by default and I
cannot install it.

So the next step is try from recovery console of debian live cd, which still
has the really old tools...

I think this is quite some serious issue but generally all debian's fault
adopting a btrfs file system support on a 2.6.32 kernel and without
btrfs-progs on some decent version.

I'll update when possible.
Please throw any other alternatives my way anyone.

Thanks,
Leonidas


-- 
Caution: breathing may be hazardous to your health.

Re: no space left on device

[C Anthony Risinger]

On Mon, Feb 7, 2011 at 3:21 PM, Leonidas Spyropoulos
<artafinde@gmail.com> wrote:
> Hey all,
>
> I run into no space left on device on a virtualbox
>
> After installing Debian 6 on a virtual machine
> I tried installing the KDE desktop
>
> The system HDD is 8Gb
> Both root (/) and /home are btrfs
> over LVM.
>
> While installing the packages I run into:
>
> no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_used, 0
> bytes_reserved,
> 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
>
> df shows only 74% used space on /
>
> kernel used: stock debian 6 2.6.32-5-686
>
> At the moment I cannot access it with normal boot, only recovery mode.
>
> I can provide whatever info you would like as long as you think of a
> way to load the normal system and not the recovery mode.

IIRC .32 has all sorts of ENOSPC problems; I think this was seriously
tackled in kernels > .32... this kernel was only declared ready for
"early adopters", with an "expect issues" disclaimer.

The btrfs-tools in squeeze is probably so old you may not even have
the `btrfs` binary, but I don't run debian so I'm not sure there...
not really a solution probably for you, but I wouldn't run that kernel
if using btrfs.

C Anthony

Re: no space left on device

[Robert G.]

Hi,

I had a similar problem on my Debian (squeeze) about a half year ago.
I've described that on this mailing list. That was a main reason to
migrate from brtfs (-o ssd) to ext4.=20

I know it's a pain for my SSD but I want to revert this fail system
when it will be more stable. Thanks God it was added to Debian 6 and
Ubuntu 10.10. Now there will be even more alpha/beta testers.
And some problems will be noticed more frequently.

When will you change status from experimental to stable in the kernel?

Regards
Robert.

Dnia 2011-02-07, o godz. 22:27:14
Erik Logtenberg <erik@logtenberg.eu> napisa=C5=82(a):

> Hi Leonidas,
>=20
> Please check this:
>=20
> btrfs fi df /home
>=20
> If this shows much of your space used by metadata then please use:
>=20
> btrfs fi balance /home
>=20
> Note that this can take a long (>1 day) time to complete on a big FS.
>=20
>  - Erik
>=20
>=20
>=20
> On 02/07/2011 10:21 PM, Leonidas Spyropoulos wrote:
> > Hey all,
> >=20
> > I run into no space left on device on a virtualbox
> >=20
> > After installing Debian 6 on a virtual machine
> > I tried installing the KDE desktop
> >=20
> > The system HDD is 8Gb
> > Both root (/) and /home are btrfs
> > over LVM.
> >=20
> > While installing the packages I run into:
> >=20
> > no space left, need 4096, 4096 dealloc bytes, 1776283648
> > bytes_used, 0 bytes_reserved,
> > 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
> >=20
> > df shows only 74% used space on /
> >=20
> > kernel used: stock debian 6 2.6.32-5-686
> >=20
> > At the moment I cannot access it with normal boot, only recovery
> > mode.
> >=20
> > I can provide whatever info you would like as long as you think of =
a
> > way to load the normal system and not the recovery mode.
> >=20
> >=20
> >=20
> >=20
>=20
> --
> To unsubscribe from this list: send the line "unsubscribe
> linux-btrfs" in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Erik Logtenberg]

Hi Leonidas,

Please check this:

btrfs fi df /home

If this shows much of your space used by metadata then please use:

btrfs fi balance /home

Note that this can take a long (>1 day) time to complete on a big FS.

 - Erik



On 02/07/2011 10:21 PM, Leonidas Spyropoulos wrote:
> Hey all,
> 
> I run into no space left on device on a virtualbox
> 
> After installing Debian 6 on a virtual machine
> I tried installing the KDE desktop
> 
> The system HDD is 8Gb
> Both root (/) and /home are btrfs
> over LVM.
> 
> While installing the packages I run into:
> 
> no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_used, 0
> bytes_reserved,
> 0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total
> 
> df shows only 74% used space on /
> 
> kernel used: stock debian 6 2.6.32-5-686
> 
> At the moment I cannot access it with normal boot, only recovery mode.
> 
> I can provide whatever info you would like as long as you think of a
> way to load the normal system and not the recovery mode.
> 
> 
> 
> 

no space left on device

[Leonidas Spyropoulos]

Hey all,

I run into no space left on device on a virtualbox

After installing Debian 6 on a virtual machine
I tried installing the KDE desktop

The system HDD is 8Gb
Both root (/) and /home are btrfs
over LVM.

While installing the packages I run into:

no space left, need 4096, 4096 dealloc bytes, 1776283648 bytes_used, 0
bytes_reserved,
0 bytes_pinned, 0 bytes_readonly, 0 may use 1776287744 total

df shows only 74% used space on /

kernel used: stock debian 6 2.6.32-5-686

At the moment I cannot access it with normal boot, only recovery mode.

I can provide whatever info you would like as long as you think of a
way to load the normal system and not the recovery mode.




-- 
Caution: breathing may be hazardous to your health.

Re: No space left on device

[Turbo Fredriksson]

[please keep all communication on/to the list for posterity]

On 27 sep 2010, at 18.26, Turbo Fredriksson wrote:

> On 27 sep 2010, at 18.06, Geoffrey Wehrman wrote:
> 
>> | > or to a much more recent kernel, and the problem will go
>> | > away.
>> | 
>> | How recent?
>> 
>> Based on the following FAQ, 2.6.35 or later.
>> http://xfs.org/index.php/XFS_FAQ#Q:_Can_I_just_try_the_inode64_option_to_see_if_it_helps_me.3F
> 
> That only stated that you can/could switch back
> and forth between 32/64bit. Not that it was/is
> solved - or solve my problem.
> 
> But thanx. I'll try a later kernel then.
> 
> 
> Ok, so 2.6.35.6 is the very latest!! I'm running
> 2.6.32, which is ancient... I had expected a much
> higher minor version... Thanx!

Ok, I'm now at 2.6.35.6, AMD64 compiled, which let me
create more files - using the inode64 mount option
and/or (did both at the same time :) moving what I
think was the first 5Gb data.

Unfortunately, I'm still using a 32bit user land
(can't upgrade at the moment).


celia:~# uname -a
Linux celia 2.6.35.6 #1 SMP Sat Oct 2 09:12:38 UTC 2010 x86_64 GNU/Linux
celia:~# file `which xfs_db`
/usr/sbin/xfs_db: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped
celia:~# xfs_info /share
meta-data=/dev/mapper/movies-movies isize=256    agcount=340, agsize=6104768 blks
        =                       sectsz=512   attr=1
data    =                       bsize=4096   blocks=2075610112, imaxpct=10
        =                       sunit=0      swidth=0 blks
naming  =version 2              bsize=4096   ascii-ci=0
log     =internal               bsize=4096   blocks=32768, version=1
        =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime=none                   extsz=65536  blocks=0, rtextents=0


The big question is now, what do I do if/when (soon)
I have to grow the FS?!

On 24 sep 2010, at 17.23, Geoffrey Wehrman wrote:

> On Fri, Sep 24, 2010 at 04:07:23PM +0200, Turbo Fredriksson wrote:
> | The worst thing is that every time I grow
> | the FS, I have to reboot into a 32bit kernel:
> | 
> | xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument
> 
> This may be becuase your are running 32bit userland.

On 24 sep 2010, at 23.17, Geoffrey Wehrman wrote:

> On Fri, Sep 24, 2010 at 10:46:47PM +0200, Turbo Fredriksson wrote:
> | On 24 sep 2010, at 17.23, Geoffrey Wehrman wrote:
> | > Once you have 64bit inodes, it is difficult to go back to 32bit inodes.
> | 
> | Figured that.. But will the growfs stop working?
> 
> The 64 bit growfs should work.
> 
> | I assume it won't help just recompiling the XFS
> | binaries since my libs et all is/will be 32bit?.
> 
> Correct.  You must have a 64 bit application/libraries/kernel.

Will my current kernel, but compiled as 32bit, work
without upgrading to a 64bit user land? Or is 2.6.35.6
also fixing the XFS_IOC_FSGROWFSDATA problem I've been
getting on 2.6.32/64bit?

But I guess I can't use that older kernel any more,
since I'm now using 64bit inodes.

The FAQ mentioned above, do say that I'm supposed to
go back and forth, but ... I'm a coward! (Actually,
I used to call myself 'careful', but I guess it's
really the same thing - also, I don't care any more :).


It doesn't seem reasonable that I could create a bigger
(i.e. 64bit) inode table - and files in that table,
switch back to a kernel that only understands 32bit
inode tables (correctly and/or 'is buggy'), grow the
FS, and then go back to using 64bit inodes and still
have all those files created in the 64bit inode table...

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device

[Dave Chinner]

On Fri, Sep 24, 2010 at 04:07:23PM +0200, Turbo Fredriksson wrote:
> I started with a few hundred gig on my FS
> (on top of LVM which in turn is on top a bunch
> of MD devices).
> 
> Slowly increasing the size with pvmove/{vg,lv}extend
> etc. Yesterday, I added more disks yet again and
> now have about 8Tb on one FS... ~1Tb free. But still
> get the 'No space left on device'.
> 
> 
> I've been looking through the 'Net and it
> must be because I'm out of inodes (and that
> the data is on the first 1Tb etc)...
> 
> I'm running on AMD 64bit (native kernel, but
> Debian Lenny 32bit userland - 2.6.2[4-6] kernel).
....
> The worst thing is that every time I grow
> the FS, I have to reboot into a 32bit kernel:
> 
> xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument

The compat ioctl handlers for 32bit userspace to 64 bit kernel we
not included until a later kernel. Either upgrade to a 64 bit
userspace or to a much more recent kernel, and the problem will go
away.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: No space left on device

[Geoffrey Wehrman]

On Fri, Sep 24, 2010 at 04:07:23PM +0200, Turbo Fredriksson wrote:
| I started with a few hundred gig on my FS
| (on top of LVM which in turn is on top a bunch
| of MD devices).
| 
| Slowly increasing the size with pvmove/{vg,lv}extend
| etc. Yesterday, I added more disks yet again and
| now have about 8Tb on one FS... ~1Tb free. But still
| get the 'No space left on device'.
| 
| 
| I've been looking through the 'Net and it
| must be because I'm out of inodes (and that
| the data is on the first 1Tb etc)...

You may may be able to workaround the the problem by increasing imaxpct
on the filesystem.
See http://oss.sgi.com/archives/xfs/2010-09/msg00295.html

| I'm running on AMD 64bit (native kernel, but
| Debian Lenny 32bit userland - 2.6.2[4-6] kernel).
| 
| 
| 
| Now, if I use (I'll try to upgrade to a newer
| kernel so I can reverse) the inode64, what
| impact will this have? I guess I still have
| to move my firstly created 1Tb of-disk and
| then back again?
| 
| 
| The worst thing is that every time I grow
| the FS, I have to reboot into a 32bit kernel:
| 
| xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument

This may be becuase your are running 32bit userland.

| and then back to the 64bit kernel... How will
| this (system/FS) react if I go 64bit inodes?!
| 
| 
| And how safe is this!? I just don't have the
| possibility to do a full backup of all this
| data, not even temporarily... I _might_ be
| able to do this in a few weeks (work is ordering
| a bunch of LaCie 8Tb USB/eSATA storage boxes
| which I might be able to load for a few days
| 'to do some stress testing on' :)
| 
| 
| References:
| http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_receive_No_space_left_on_device_after_xfs_growfs.3F
| http://xfs.org/index.php/XFS_FAQ#Q:_Can_I_just_try_the_inode64_option_to_see_if_it_helps_me.3F
| http://www.doc.ic.ac.uk/~dcw/xfs_16tb/

Once you have 64bit inodes, it is difficult to go back to 32bit inodes.
The filesystem will stay together, but you may not be able to access
your files with 64bit inodes from 32bit applications.


-- 
Geoffrey Wehrman

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

No space left on device

[Turbo Fredriksson]

I started with a few hundred gig on my FS
(on top of LVM which in turn is on top a bunch
of MD devices).

Slowly increasing the size with pvmove/{vg,lv}extend
etc. Yesterday, I added more disks yet again and
now have about 8Tb on one FS... ~1Tb free. But still
get the 'No space left on device'.


I've been looking through the 'Net and it
must be because I'm out of inodes (and that
the data is on the first 1Tb etc)...

I'm running on AMD 64bit (native kernel, but
Debian Lenny 32bit userland - 2.6.2[4-6] kernel).



Now, if I use (I'll try to upgrade to a newer
kernel so I can reverse) the inode64, what
impact will this have? I guess I still have
to move my firstly created 1Tb of-disk and
then back again?


The worst thing is that every time I grow
the FS, I have to reboot into a 32bit kernel:

xfs_growfs: XFS_IOC_FSGROWFSDATA xfsctl failed: Invalid argument

and then back to the 64bit kernel... How will
this (system/FS) react if I go 64bit inodes?!


And how safe is this!? I just don't have the
possibility to do a full backup of all this
data, not even temporarily... I _might_ be
able to do this in a few weeks (work is ordering
a bunch of LaCie 8Tb USB/eSATA storage boxes
which I might be able to load for a few days
'to do some stress testing on' :)


References:
http://xfs.org/index.php/XFS_FAQ#Q:_Why_do_I_receive_No_space_left_on_device_after_xfs_growfs.3F
http://xfs.org/index.php/XFS_FAQ#Q:_Can_I_just_try_the_inode64_option_to_see_if_it_helps_me.3F
http://www.doc.ic.ac.uk/~dcw/xfs_16tb/

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: no space left on device

[Lubos Kolouch]

I have another example with 2.6.35-r1 and latest btrfs-progs

# btrfs filesystem show
Label: none  uuid: 46c7a8b4-ca13-4b4f-a695-6a0e31112411
	Total devices 2 FS bytes used 1.23TB
	devid    1 size 896.39GB used 896.39GB path /dev/dm-0
	devid    2 size 896.39GB used 896.25GB path /dev/dm-1

# btrfs filesystem df /home
Data: total=3D1.74TB, used=3D1.23TB
Metadata: total=3D3.01GB, used=3D2.18GB
System: total=3D12.00MB, used=3D200.00KB

So there should be plenty of space left, but I am getting messages
that the device is out of free space.

Anything I can try to debug this?

Thank you

Lubos


Lubos Kolouch, Fri, 30 Jul 2010 15:09:59 +0000:

> correction - btrfs-cache processes
>=20
> Lubos Kolouch, Fri, 30 Jul 2010 15:02:18 +0000:
>=20
>> With 2.6.34 it basically "works", but time to time the load goes up =
to
>> 50 and lot of btrfs-cleaner processess are spawned... the system is
>> responsive though, except for the filesystem... after an hour or so =
the
>> processes finish and the load goes back to normal
>>=20
>> Lubos
>>=20
>>> On Fri, Jul 30, 2010 at 3:26 PM, Leonidas Spyropoulos
>>> <artafinde@gmail.com> wrote:
>>>> I use the latest btrfs-tools and 2.6.34-020634-generic kernel for
>>>> lucid. Got no issues like that, is it happening with 2.6.34 kernel=
 as
>>>> well?
>>>>
>>>> On Fri, Jul 30, 2010 at 1:27 PM, Lubos Kolouch
>>>> <lubos.kolouch@gmail.com> wrote:
>>>>>
>>>>> Yes, I pulled today's btrfs-progs.
>>>>>
>>>>> Lubos
>>>>>
>>>>> On P=C3=A1, =C4=8Dec 30, 2010, 13:03:10, Leonidas Spyropoulos wro=
te:
>>>>> > Have you updated with latest btrfs-tools?
>>>>> >
>>>>> > On Fri, Jul 30, 2010 at 6:31 AM, Lubos Kolouch
>>>>> > <lubos.kolouch@gmail.com>wrote:
>>>>> >
>>>>> > > Hello,
>>>>> > >
>>>>> > > kerner 2.6.35-rc6
>>>>> > >
>>>>> > > btrfs filesystem df /home
>>>>> > > Data: total=3D1.68TB, used=3D987.62GB
>>>>> > > Metadata: total=3D56.01GB, used=3D48.16GB System: total=3D12.=
00MB,
>>>>> > > used=3D200.00KB
>>>>> > >
>>>>> > > touch: cannot touch `/home/x': No space left on device
>>>>> > >
>>>>> > > Any ideas what I should do?
>>>>> > >
>>>>> > > Thank you
>>>>> > >
>>>>> > > Lubos
>>>>> > >


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Lubos Kolouch]

correction - btrfs-cache processes

Lubos Kolouch, Fri, 30 Jul 2010 15:02:18 +0000:

> With 2.6.34 it basically "works", but time to time the load goes up t=
o
> 50 and lot of btrfs-cleaner processess are spawned... the system is
> responsive though, except for the filesystem... after an hour or so t=
he
> processes finish and the load goes back to normal
>=20
> Lubos
>=20
>> On Fri, Jul 30, 2010 at 3:26 PM, Leonidas Spyropoulos
>> <artafinde@gmail.com> wrote:
>>> I use the latest btrfs-tools and 2.6.34-020634-generic kernel for
>>> lucid. Got no issues like that, is it happening with 2.6.34 kernel =
as
>>> well?
>>>
>>> On Fri, Jul 30, 2010 at 1:27 PM, Lubos Kolouch
>>> <lubos.kolouch@gmail.com> wrote:
>>>>
>>>> Yes, I pulled today's btrfs-progs.
>>>>
>>>> Lubos
>>>>
>>>> On P=C3=A1, =C4=8Dec 30, 2010, 13:03:10, Leonidas Spyropoulos wrot=
e:
>>>> > Have you updated with latest btrfs-tools?
>>>> >
>>>> > On Fri, Jul 30, 2010 at 6:31 AM, Lubos Kolouch
>>>> > <lubos.kolouch@gmail.com>wrote:
>>>> >
>>>> > > Hello,
>>>> > >
>>>> > > kerner 2.6.35-rc6
>>>> > >
>>>> > > btrfs filesystem df /home
>>>> > > Data: total=3D1.68TB, used=3D987.62GB
>>>> > > Metadata: total=3D56.01GB, used=3D48.16GB System: total=3D12.0=
0MB,
>>>> > > used=3D200.00KB
>>>> > >
>>>> > > touch: cannot touch `/home/x': No space left on device
>>>> > >
>>>> > > Any ideas what I should do?
>>>> > >
>>>> > > Thank you
>>>> > >
>>>> > > Lubos
>>>> > >


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Lubos Kolouch]

With 2.6.34 it basically "works", but time to time the load goes up to =
50
and lot of btrfs-cleaner processess are spawned...=20
the system is responsive though, except for the filesystem... after an=20
hour or so the processes finish and the load goes back to normal

Lubos

> On Fri, Jul 30, 2010 at 3:26 PM, Leonidas Spyropoulos
> <artafinde@gmail.com> wrote:
>> I use the latest btrfs-tools and 2.6.34-020634-generic kernel for
>> lucid. Got no issues like that, is it happening with 2.6.34 kernel a=
s
>> well?
>>
>> On Fri, Jul 30, 2010 at 1:27 PM, Lubos Kolouch
>> <lubos.kolouch@gmail.com> wrote:
>>>
>>> Yes, I pulled today's btrfs-progs.
>>>
>>> Lubos
>>>
>>> On P=C3=A1, =C4=8Dec 30, 2010, 13:03:10, Leonidas Spyropoulos wrote=
:
>>> > Have you updated with latest btrfs-tools?
>>> >
>>> > On Fri, Jul 30, 2010 at 6:31 AM, Lubos Kolouch
>>> > <lubos.kolouch@gmail.com>wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > kerner 2.6.35-rc6
>>> > >
>>> > > btrfs filesystem df /home
>>> > > Data: total=3D1.68TB, used=3D987.62GB
>>> > > Metadata: total=3D56.01GB, used=3D48.16GB System: total=3D12.00=
MB,
>>> > > used=3D200.00KB
>>> > >
>>> > > touch: cannot touch `/home/x': No space left on device
>>> > >
>>> > > Any ideas what I should do?
>>> > >
>>> > > Thank you
>>> > >
>>> > > Lubos
>>> > >

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Leonidas Spyropoulos]

Sorry for the HTML replies.. I just read the mail delivery failure and
turned it to plain text.
You can check the quoted text for my answers.

On Fri, Jul 30, 2010 at 3:26 PM, Leonidas Spyropoulos
<artafinde@gmail.com> wrote:
> I use the latest btrfs-tools and 2.6.34-020634-generic kernel for luc=
id.
> Got no issues like that, is it happening with 2.6.34 kernel as well?
>
> On Fri, Jul 30, 2010 at 1:27 PM, Lubos Kolouch <lubos.kolouch@gmail.c=
om>
> wrote:
>>
>> Yes, I pulled today's btrfs-progs.
>>
>> Lubos
>>
>> On P=C3=A1, =C4=8Dec 30, 2010, 13:03:10, Leonidas Spyropoulos wrote:
>> > Have you updated with latest btrfs-tools?
>> >
>> > On Fri, Jul 30, 2010 at 6:31 AM, Lubos Kolouch
>> > <lubos.kolouch@gmail.com>wrote:
>> >
>> > > Hello,
>> > >
>> > > kerner 2.6.35-rc6
>> > >
>> > > btrfs filesystem df /home
>> > > Data: total=3D1.68TB, used=3D987.62GB
>> > > Metadata: total=3D56.01GB, used=3D48.16GB
>> > > System: total=3D12.00MB, used=3D200.00KB
>> > >
>> > > touch: cannot touch `/home/x': No space left on device
>> > >
>> > > Any ideas what I should do?
>> > >
>> > > Thank you
>> > >
>> > > Lubos
>> > >
>> > > --
>> > > To unsubscribe from this list: send the line "unsubscribe linux-=
btrfs"
>> > > in
>> > > the body of a message to majordomo@vger.kernel.org
>> > > More majordomo info at =C2=A0http://vger.kernel.org/majordomo-in=
fo.html
>> > >
>> >
>> >
>> >
>> > --
>> > Caution: breathing may be hazardous to your health.
>
>
>
> --
> Caution: breathing may be hazardous to your health.
>



--=20
Caution: breathing may be hazardous to your health.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: no space left on device

[Lubos Kolouch]

Yes, I pulled today's btrfs-progs.=20

Lubos

On P=C3=A1, =C4=8Dec 30, 2010, 13:03:10, Leonidas Spyropoulos wrote:
> Have you updated with latest btrfs-tools?
>=20
> On Fri, Jul 30, 2010 at 6:31 AM, Lubos Kolouch <lubos.kolouch@gmail.c=
om>wrote:
>=20
> > Hello,
> >
> > kerner 2.6.35-rc6
> >
> > btrfs filesystem df /home
> > Data: total=3D1.68TB, used=3D987.62GB
> > Metadata: total=3D56.01GB, used=3D48.16GB
> > System: total=3D12.00MB, used=3D200.00KB
> >
> > touch: cannot touch `/home/x': No space left on device
> >
> > Any ideas what I should do?
> >
> > Thank you
> >
> > Lubos
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btr=
fs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
>=20
>=20
>=20
> --=20
> Caution: breathing may be hazardous to your health.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

no space left on device

[Lubos Kolouch]

Hello,

kerner 2.6.35-rc6

btrfs filesystem df /home 
Data: total=1.68TB, used=987.62GB
Metadata: total=56.01GB, used=48.16GB
System: total=12.00MB, used=200.00KB

touch: cannot touch `/home/x': No space left on device

Any ideas what I should do?

Thank you

Lubos

Re: No space left on device

[dcg]

El Mon, 08 Dec 2008 10:46:25 +0100, Gabor MICSKO <gabor.micsko@gmail.co=
m> escribi=C3=B3:

> # df -h
> [...]
> /dev/sdb1              75G   61G   15G  81% /media/coolermaster
>=20
> It seems like i have 15GB free space on the btrfs formatted partition=
,
> but when i try to create a file of 4GB on the btrfs filesystem i get =
the
> following error:


btrfs has an (artificial) limit of 85% of the space which won't be lift=
ed until
the allocation code is mature. So you can't use the 100% for now...that
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No space left on device

[Gabor MICSKO]

hi,

I have an 80GB external HDD with one btrfs partition. Latest kernel
module and btrfs-progs from "unstable" git repos.

# mount
[...]
/dev/sdb1 on /media/coolermaster type btrfs (rw)

# df -h
[...]
/dev/sdb1              75G   61G   15G  81% /media/coolermaster

It seems like i have 15GB free space on the btrfs formatted partition,
but when i try to create a file of 4GB on the btrfs filesystem i get the
following error:

root@alderaan:/media/coolermaster/tmp# dd if=/dev/zero of=foobar.img
bs=1024 count=4000K
dd: writing `foobar.img': No space left on device
3372894+0 records in
3372893+0 records out
3453842432 bytes (3.5 GB) copied, 321.087 s, 10.8 MB/s

syslog:

Dec  8 10:35:46 alderaan kernel: [30716.434747] device fsid
46489b7e62f254e0-e306291dd7724d8f devid 1 transid 283 /dev/sdb1
Dec  8 10:35:50 alderaan kernel: [30720.457774] space info full 36

Am i missing something (e.g. reserved space for root, etc.) obvious
here?

Thanks,

--
mg

Re: No space left on device

[Sebastian]

It's a 512K AMD flash with 8 erase blocks on an IBM Walnut board. The ROM
Monitor program which I use for booting takes up the last 2 blocks, and I'm
trying to use the first 6 blocks for JFFS.
I've attached a previous email below that gives more details.

Sebastian



Hi,

I'm trying to use JFFS2 on a 512K AMD (AM29F040B) flash chip (8 erase
blocks) installed on the IBM Walnut board. I was able to create a JFFS2
image with mkfs.jffs2, erase the flash, and then copy the image to the flash
( about 200 bytes). I can mount the mtdblock and browse the files on the
JFFS2 file system, but I cannot make any changes to the existing files or
add new ones. Everytime I try to modify it I get the 'No space left on
device' error. I had a quick look through the JFFS2 code and I see that
there has to be at least 5 free erase blocks (+ data) for the FS to function
properly ... but I have 8! I tried changing JFFS2_RESERVED_BLOCKS_BASE from
3 to 1 (which gives a total of 3 blocks for minimum requirement) and it
still didn't work. Am I missing something here? Do I have to pad my image
with 0xFF and make it 512K before I copy it? Is there another way to make
the image and copy it to the flash ... etc?

Regards,
Sebastian Vaitus


>
> sebastian.vaitus@masstechgroup.com said:
> > Erase at 0x00010000 finished, but state != MTD_ERASE_DONE. State is
> > 0x0 instead.
>
> What hardware?
>
> --
> dwmw2
>
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
>

Re: No space left on device

[David Woodhouse]

sebastian.vaitus@masstechgroup.com said:
> Erase at 0x00010000 finished, but state != MTD_ERASE_DONE. State is
> 0x0 instead. 

What hardware?

--
dwmw2

Re: No space left on device

[Sebastian]

[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]

David,

I have logged the output for the problem described in a previous email
(attached below). I am using 6 erase blocks this time instead of 8 (which
should still be enough for the 280 byte JFFS2 image). From what I saw in the
logs, I think that JFFS is only able to use the first block out of the 6 due
to some formatting or erasing problem with the rest of 5 blocks. Please have
a  look at the attached log and maybe you'll be able to easily identify the
problem..

Regards,
Sebastian

>
> sebastian.vaitus@masstechgroup.com said:
> >  I was able to create a JFFS2 image with mkfs.jffs2, erase the flash,
> > and then copy the image to the flash ( about 200 bytes). I can mount
> > the mtdblock and browse the files on the JFFS2 file system, but I
> > cannot make any changes to the existing files or add new ones.
> > Everytime I try to modify it I get the 'No space left on device'
> > error.
>
> Please compile with CONFIG_JFFS2_FS_DEBUG=1, set the console loglevel to 9
> so that you see all the KERN_DEBUG messages, and log the output over a
> serial console while you reproduce this and also run 'df'.
>
> --
> dwmw2
>
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
>

[-- Attachment #2: jffs2_clean --]
[-- Type: application/octet-stream, Size: 10013 bytes --]


*** Make the jffs2 image

root@192.0.0.123:/jffs2#
root@192.0.0.123:/jffs2# mkfs.jffs2 -d ./imgdir -o jffs2.img
root@192.0.0.123:/jffs2# ls
imgdir	jffs2.img
root@192.0.0.123:/jffs2# ls -l
total 8
drwxr-xr-x    2 root     root         4096 Mar 27  2002 imgdir
-rw-r--r--    1 root     root          280 Mar 27  2002 jffs2.img

*** Copy the image to flash

root@192.0.0.123:/jffs2# cd ..
root@192.0.0.123:/# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00060000 00010000 "Flash File System"
mtd1: 00020000 00010000 "ROM Monitor"
mtd2: 00080000 00010000 "Whole Flash"
root@192.0.0.123:/# eraseall /dev/mtd0
MTD_open
MTD_ioctl
MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 0 --  0 % complete.MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 10000 -- 16 % complete.MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 20000 -- 33 % complete.MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 30000 -- 50 % complete.MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 40000 -- 66 % complete.MTD_ioctl
Erasing 64 Kibytodo: 1 0 0 0
te @ 50000 -- 83 % complete.MTD_close
Erased 384 Kibyte @ 0 -- 100% complete.
root@192.0.0.123:/# cp /jffs2/jffs2.img /dev/mtd0
MTD_open
MTD_write
MTD_close

*** Mount the JFFS2 file system

root@192.0.0.123:/# cd mnt/flash
root@192.0.0.123:/mnt/flash# ls
root@192.0.0.123:/mnt/flash# cd ..
root@192.0.0.123:/mnt# mount -t jffs2 /dev/mtdblock0 /mnt/flash
mtdblock_open
ok
jffs2: read_super for device 1f:00
jffs2_scan_eraseblock(): Scanning block at 0x0
jffs2_scan_inode_node(): Node at 0x00000000
jffs2_get_ino_cache(): ino 1
jffs2_get_ino_cache found 00000000 for ino 1
Allocated inocache at c778c2d4
jffs2_add_ino_cache: Add c778c2d4 (ino #1)
Node is ino #1, version 1. Range 0x0-0x0
jffs2_scan_dirent_node(): Node at 0x00000044
jffs2_get_ino_cache(): ino 1
jffs2_get_ino_cache found c778c2d4 for ino 1
jffs2_add_fd_to_list( c70124c0, c70124a0 (->00000000))
Dirent "test" (hash 0x01c4ddc5, ino #2
jffs2_scan_inode_node(): Node at 0x00000070
jffs2_get_ino_cache(): ino 2
jffs2_get_ino_cache found 00000000 for ino 2
Allocated inocache at c778c2c0
jffs2_add_ino_cache: Add c778c2c0 (ino #2)
Node is ino #2, version 1. Range 0x0-0x1d
jffs2_scan_inode_node(): Node at 0x000000d0
jffs2_get_ino_cache(): ino 2
jffs2_get_ino_cache found c778c2c0 for ino 2
Node is ino #2, version 2. Range 0x1d-0x1f
Found empty flash at 0x118
Empty flash detected from 0x00000118 to 0x00010000
Block at 0x00000000: free 0x0000fee8, dirty 0x00000000, used 0x00000118
jffs2_scan_eraseblock(): Scanning block at 0x10000
Found empty flash at 0x10000
Empty flash detected from 0x00010000 to 0x00020000
Block at 0x00010000: free 0x00010000, dirty 0x00000000, used 0x00000000
JFFS2: Erase block at 0x00010000 is not formatted. It will be erased
jffs2_scan_eraseblock(): Scanning block at 0x20000
Found empty flash at 0x20000
Empty flash detected from 0x00020000 to 0x00030000
Block at 0x00020000: free 0x00010000, dirty 0x00000000, used 0x00000000
JFFS2: Erase block at 0x00020000 is not formatted. It will be erased
jffs2_scan_eraseblock(): Scanning block at 0x30000
Found empty flash at 0x30000
Empty flash detected from 0x00030000 to 0x00040000
Block at 0x00030000: free 0x00010000, dirty 0x00000000, used 0x00000000
JFFS2: Erase block at 0x00030000 is not formatted. It will be erased
jffs2_scan_eraseblock(): Scanning block at 0x40000
Found empty flash at 0x40000
Empty flash detected from 0x00040000 to 0x00050000
Block at 0x00040000: free 0x00010000, dirty 0x00000000, used 0x00000000
JFFS2: Erase block at 0x00040000 is not formatted. It will be erased
jffs2_scan_eraseblock(): Scanning block at 0x50000
Found empty flash at 0x50000
Empty flash detected from 0x00050000 to 0x00060000
Block at 0x00050000: free 0x00010000, dirty 0x00000000, used 0x00000000
JFFS2: Erase block at 0x00050000 is not formatted. It will be erased
Scanned flash completely
Pass 1: ino #1
jffs2_build_inode building inode #1
jffs2_get_ino_cache(): ino 2
jffs2_get_ino_cache found c778c2c0 for ino 2
Increased nlink for child "test" (ino #2)
Pass 1: ino #2
jffs2_build_inode building inode #2
adding node 0000-001d @0x00000070 on flash, newfrag *c778b2d4
adding node 001d-001f @0x000000d0 on flash, newfrag *c778b2c0
j_a_f_d_t_f: skipping frag 0x0000-0x001d; phys 0x00000070 (*c778b2d4->00000000)
Pass 1 complete
Pass 2 (re)starting
Pass 2: ino #1, nlink 1, ic c778c2d4, nodes c778f360
Pass 2: ino #2, nlink 1, ic c778c2c0, nodes c778f340
Pass 2 complete
Pass 3: ino #1, ic c778c2d4, nodes c778f360
Pass 3: ino #2, ic c778c2c0, nodes c778f340
Pass 3 complete
jffs2_read_super(): Getting root inode
jffs2_read_inode(): inode->i_ino == 1
getting inocache
jffs2_get_ino_cache(): ino 1
jffs2_get_ino_cache found c778c2d4 for ino 1
jffs2_read_inode(): Got inocache at c778c2d4
jffs2_read_inode(): ino #1 nlink is 1
jffs2_get_inode_nodes(): ino #1
Node at 00000044 is a dirent node
Adding fd "test", ino #2
jffs2_add_fd_to_list( c70124e0, c7429c1c (->00000000))
Dirent "test" (hash 0x01c4ddc5, ino #2
Node at 00000000 is a data node
version 1, highest_version now 2
dnode @00000000: ver 1, offset 0000, dsize 0000
metadata @00000000: ver 1
jffs2_read_inode() returning
jffs2_read_super(): d_alloc_root()
JFFS2: Garbage collect thread is pid 186
thread_should_wake(): nr_free_blocks 0, nr_erasing_blocks 5, dirty_size 0x0
jffs2_garbage_collect_thread sleeping...
thread_should_wake(): nr_free_blocks 0, nr_erasing_blocks 5, dirty_size 0x0
Starting erase of pending block 0x00050000
Freeing all node refs for eraseblock offset 0x00050000
todo: 1 0 0 0
Erase at 0x00050000 finished, but state != MTD_ERASE_DONE. State is 0x0 instead.
Starting erase of pending block 0x00040000
Freeing all node refs for eraseblock offset 0x00040000
todo: 1 0 0 0
Erase at 0x00040000 finished, but state != MTD_ERASE_DONE. State is 0x0 instead.
Starting erase of pending block 0x00030000
Freeing all node refs for eraseblock offset 0x00030000
todo: 1 0 0 0
Erase at 0x00030000 finished, but state != MTD_ERASE_DONE. State is 0x0 instead.
Starting erase of pending block 0x00020000
Freeing all node refs for eraseblock offset 0x00020000
todo: 1 0 0 0
Erase at 0x00020000 finished, but state != MTD_ERASE_DONE. State is 0x0 instead.
Starting erase of pending block 0x00010000
Freeing all node refs for eraseblock offset 0x00010000
todo: 1 0 0 0
Erase at 0x00010000 finished, but state != MTD_ERASE_DONE. State is 0x0 instead.
jffs2_erase_pending_blocks completed
root@192.0.0.123:/mnt# thread_should_wake(): nr_free_blocks 0, nr_erasing_blocks 0, dirty_size 0x0
jffs2_erase_pending_blocks completed

*** List files on JFFS2 drive

root@192.0.0.123:/mnt# cd flash
root@192.0.0.123:/mnt/flash# ls
jffs2_readdir() for dir_i #1
Dirent 0: ".", ino #1
Dirent 1: "..", ino #1
Dirent 2: "test", ino #2, type 8
jffs2_readdir() for dir_i #1
Skipping dirent: "test", ino #2, type 8, because curofs 2 < offset 3
test

*** Attempt to rename the file under JFFS2

root@192.0.0.123:/mnt/flash# mv test test1
jffs2_lookup()
jffs2_lookup()
jffs2_read_inode(): inode->i_ino == 2
getting inocache
jffs2_get_ino_cache(): ino 2
jffs2_get_ino_cache found c778c2c0 for ino 2
jffs2_read_inode(): Got inocache at c778c2c0
jffs2_read_inode(): ino #2 nlink is 1
jffs2_get_inode_nodes(): ino #2
Node at 000000d0 is a data node
version 2, highest_version now 2
dnode @000000d0: ver 2, offset 001d, dsize 0002
Node at 00000070 is a data node
version 1, highest_version now 2
dnode @00000070: ver 1, offset 0000, dsize 001d
jffs2_add_full_dnode_to_inode(ino #2, f c6fd8b10, fn c778d360)
adding node 0000-001d @0x00000070 on flash, newfrag *c778b2c0
frag 0000-001d: 0x00000070 on flash (*c778b2c0->00000000)
jffs2_add_full_dnode_to_inode(ino #2, f c6fd8b10, fn c778d350)
adding node 001d-001f @0x000000d0 on flash, newfrag *c778b2d4
j_a_f_d_t_f: skipping frag 0x0000-0x001d; phys 0x00000070 (*c778b2c0->00000000)
frag 0000-001d: 0x00000070 on flash (*c778b2c0->c778b2d4)
frag 001d-001f: 0x000000d0 on flash (*c778b2d4->00000000)
Truncating fraglist to 0x0000001f bytes
jffs2_read_inode() returning
jffs2_reserve_space(): Requested 0x30 bytes
jffs2_reserve_space(): alloc sem got
Short on space, but total dirty size 0x00000000 < sector size 0x00010000, so -ENOSPC
jffs2_create()
jffs2_reserve_space(): Requested 0x44 bytes
jffs2_reserve_space(): alloc sem got
Short on space, but total dirty size 0x00000000 < sector size 0x00010000, so -ENOSPC
jffs2_create(): reserved 0xc7aef000 bytes
mv: cannot create regular file `test1': No space left on device

*** Attempt to change permissions on file

root@192.0.0.123:/mnt/flash# chmod 777 test
jffs2_setattr(): ino #2
jffs2_reserve_space(): Requested 0x44 bytes
jffs2_reserve_space(): alloc sem got
Short on space, but total dirty size 0x00000000 < sector size 0x00010000, so -ENOSPC
chmod: changing permissions of `test': No space left on device

*** Attempt to create a directory

root@192.0.0.123:/mnt/flash# mkdir test123
jffs2_lookup()
jffs2_reserve_space(): Requested 0x44 bytes
jffs2_reserve_space(): alloc sem got
Short on space, but total dirty size 0x00000000 < sector size 0x00010000, so -ENOSPC
mkdir: cannot create directory `test123': No space left on device

*** DF

root@192.0.0.123:/mnt/flash# df
Filesystem      STATFS:
flash_size: 00060000
used_size: 00000118
dirty_size: 00000000
free_size: 0000fee8
erasing_size: 00000000
bad_size: 00050000
sector_size: 00010000
nextblock: 0x00000000
gcblock: NULL
clean_list: empty
dirty_list: empty
erasing_list: empty
erase_pending_list: empty
free_list: empty
bad_list: 00010000
bad_list: 00020000
bad_list: 00030000
bad_list: 00040000
bad_list: 00050000
bad_used_list: empty
     1k-blocks      Used Available Use% Mounted on
/dev/root              2869272   1721132   1002384  64% /
/dev/mtdblock0             384       384         0 100% /mnt/flash

Re: No space left on device

[David Woodhouse]

sebastian.vaitus@masstechgroup.com said:
>  I was able to create a JFFS2 image with mkfs.jffs2, erase the flash,
> and then copy the image to the flash ( about 200 bytes). I can mount
> the mtdblock and browse the files on the JFFS2 file system, but I
> cannot make any changes to the existing files or add new ones.
> Everytime I try to modify it I get the 'No space left on device'
> error.

Please compile with CONFIG_JFFS2_FS_DEBUG=1, set the console loglevel to 9 
so that you see all the KERN_DEBUG messages, and log the output over a 
serial console while you reproduce this and also run 'df'. 

--
dwmw2

No space left on device

[Sebastian]

Hi,

I'm trying to use JFFS2 on a 512K AMD (AM29F040B) flash chip (8 erase
blocks) installed on the IBM Walnut board. I was able to create a JFFS2
image with mkfs.jffs2, erase the flash, and then copy the image to the flash
( about 200 bytes). I can mount the mtdblock and browse the files on the
JFFS2 file system, but I cannot make any changes to the existing files or
add new ones. Everytime I try to modify it I get the 'No space left on
device' error. I had a quick look through the JFFS2 code and I see that
there has to be at least 5 free erase blocks (+ data) for the FS to function
properly ... but I have 8! I tried changing JFFS2_RESERVED_BLOCKS_BASE from
3 to 1 (which gives a total of 3 blocks for minimum requirement) and it
still didn't work. Am I missing something here? Do I have to pad my image
with 0xFF and make it 512K before I copy it? Is there another way to make
the image and copy it to the flash ... etc?

Regards,
Sebastian Vaitus