Crash in __btrfs_reserve_extent

Crash in __btrfs_reserve_extent

[Tore Anderson]

Hi guys,

I just had my Fedora 11 workstation (kernel version
2.6.29.4-167.fc11.i686.PAE) crash hard.  The following was printed to a
logged-in SSH session:

------------[ cut here ]------------
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum
Process pdflush (pid: 11675, ti=d4944000 task=ee516500 task.ti=d4944000)
Stack:
 c2920960 00000024 00000000 00000000 00001000 00000000 00000000 f6d72660
 f5c8a150 00001000 00000000 a55cf000 d4945ccb d4945c50 d4945c48 f8a0d70e
Call Trace:
 [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs]
 [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs]
 [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs]
 [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs]
 [<c0565323>] ? __lookup_tag+0x89/0xe3
 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e
 [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs]
 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs]
 [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs]
 [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs]
 [<c04852bc>] ? do_writepages+0x25/0x39
 [<c04bea15>] ? __writeback_single_inode+0x15c/0x27a
 [<c0667d88>] ? dm_any_congested+0x32/0x3d
 [<f8a15c3b>] ? btrfs_congested_fn+0x38/0x66 [btrfs]
 [<c04bee90>] ? generic_sync_sb_inodes+0x1d9/0x2f6
 [<c04bf156>] ? writeback_inodes+0x82/0xca
 [<c048598c>] ? background_writeout+0x7b/0xa7
 [<c04860c2>] ? pdflush+0x130/0x1dc
 [<c0485911>] ? background_writeout+0x0/0xa7
 [<c0485f92>] ? pdflush+0x0/0x1dc
 [<c0446fc8>] ? kthread+0x41/0x65
 [<c0446f87>] ? kthread+0x0/0x65
 [<c0409dbf>] ? kernel_thread_helper+0x7/0x10
Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83
EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:d4945ba4
------------[ cut here ]------------
invalid opcode: 0000 [#2] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum
Process qemu-kvm (pid: 11670, ti=f0c6a000 task=d9c89940 task.ti=f0c6a000)
Stack:
 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d04630
 f5c8a150 00001000 00000000 05f82000 f0c6bc67 f0c6bbec f0c6bbe4 f8a0d70e
Call Trace:
 [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs]
 [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs]
 [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs]
 [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs]
 [<c0565323>] ? __lookup_tag+0x89/0xe3
 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e
 [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs]
 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs]
 [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs]
 [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs]
 [<f8a2d0be>] ? btrfs_fdatawrite_range+0x67/0x6f [btrfs]
 [<f8a23245>] ? btrfs_file_write+0x440/0x627 [btrfs]
 [<c05365ef>] ? security_file_permission+0x14/0x16
 [<c04a8b59>] ? rw_verify_area+0x9a/0xbb
 [<f8a22e05>] ? btrfs_file_write+0x0/0x627 [btrfs]
 [<c04a9229>] ? vfs_write+0x95/0xf4
 [<c04a92e0>] ? sys_pwrite64+0x58/0x70
 [<c040945f>] ? sysenter_do_call+0x12/0x34
Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83
EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f0c6bb40
------------[ cut here ]------------
invalid opcode: 0000 [#3] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1a.7/usb1/1-2/1-2.1/1-2.1.1/devnum
Process qemu-kvm (pid: 11674, ti=c583c000 task=d9c8f1a0 task.ti=c583c000)
Stack:
 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d04ed0
 f5c8a150 00001000 00000000 03fbd000 c583dc67 c583dbec c583dbe4 f8a0d70e
Call Trace:
 [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs]
 [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs]
 [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs]
 [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs]
 [<c0565323>] ? __lookup_tag+0x89/0xe3
 [<c05653ec>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e
 [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs]
 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs]
 [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs]
 [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs]
 [<f8a2d0be>] ? btrfs_fdatawrite_range+0x67/0x6f [btrfs]
 [<f8a23245>] ? btrfs_file_write+0x440/0x627 [btrfs]
 [<c05365ef>] ? security_file_permission+0x14/0x16
 [<c04a8b59>] ? rw_verify_area+0x9a/0xbb
 [<f8a22e05>] ? btrfs_file_write+0x0/0x627 [btrfs]
 [<c04a9229>] ? vfs_write+0x95/0xf4
 [<c04a92e0>] ? sys_pwrite64+0x58/0x70
 [<c040945f>] ? sysenter_do_call+0x12/0x34
Code: e8 17 46 a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 18 d0 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83
EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:c583db40

I have btrfs on my root filesystem and at the time of the crash I taring
together some files from a NFS filesystem onto it.  There was plenty of
free space on the btrfs filesystem.

I wasn't able to Google up another report with a similar-looking crash,
so I thought you might be interested.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

* Tore Anderson

> I have btrfs on my root filesystem and at the time of the crash I
> taring together some files from a NFS filesystem onto it.  There was
> plenty of free space on the btrfs filesystem.

Some more info - the file system is acting really strange after a
reboot.  A lot of messages like these are printed to /var/log/messages:

no space left, need 4096, 0 delalloc bytes, 16577376256 bytes_used, 0
bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 69816320 may
use16647192576 total

Many commands gives the error message "no space left on device".
Curiously enough /var/log/messages is stored on the very same file
system, and new lines appears in it, so it appears those writes are
avoiding the ENOSPC somehow.

Anyway, there's plenty of free space on the file system:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_echo-lv_root  228G   16G  212G   7% /

It has never been resized, by the way.  I found that this issue had been
mentioned on the mailing list by Hugo Mills (Cc-ed) before after all -
apologies for posting a dupe:

http://thread.gmane.org/gmane.comp.file-systems.btrfs/2694

Is this bug fixed in later versions of btrfs or is it still unresolved,
I wonder?  If the latter, I'll be glad to help out with any debugging
info you might need before I try to fix it (or re-install).

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 10:53:24AM +0200, Tore Anderson wrote:
> * Tore Anderson
> 
> > I have btrfs on my root filesystem and at the time of the crash I
> > taring together some files from a NFS filesystem onto it.  There was
> > plenty of free space on the btrfs filesystem.
> 
> Some more info - the file system is acting really strange after a
> reboot.  A lot of messages like these are printed to /var/log/messages:
> 
> no space left, need 4096, 0 delalloc bytes, 16577376256 bytes_used, 0
> bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 69816320 may
> use16647192576 total
> 
> Many commands gives the error message "no space left on device".
> Curiously enough /var/log/messages is stored on the very same file
> system, and new lines appears in it, so it appears those writes are
> avoiding the ENOSPC somehow.
> 
> Anyway, there's plenty of free space on the file system:
> 
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/mapper/vg_echo-lv_root  228G   16G  212G   7% /
> 
> It has never been resized, by the way.  I found that this issue had been
> mentioned on the mailing list by Hugo Mills (Cc-ed) before after all -
> apologies for posting a dupe:
> 
> http://thread.gmane.org/gmane.comp.file-systems.btrfs/2694
> 
> Is this bug fixed in later versions of btrfs or is it still unresolved,
> I wonder?  If the latter, I'll be glad to help out with any debugging
> info you might need before I try to fix it (or re-install).
>

This is one of the gotchas of btrfs, there is not proper ENOSPC handling, just a
few things in place that are a bit conservative to make sure you don't panic the
box.  Btrfs has seperate zones used for data and metadata, and these chunks are
allocated in 1gb chunks, so  you have 212gb of space thats allowed for data use,
and 16gb thats allowed for metadata use.  By default every 12 (or it may be 8, i
forget) chunks we allocate for data, we allocate 1 for metadata, which ends up
with like 8% of the disk being used for metadata.

Now in your case you can run btrfsctl -b and it will re-balance the space on the
drive, and it may give you more space back.  It could also possible panic the
box, so make sure you are all backed up :).  Thanks,

Josef 

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

Hello Josef,

* Josef Bacik

> This is one of the gotchas of btrfs, there is not proper ENOSPC
> handling, just a few things in place that are a bit conservative to
> make sure you don't panic the box.  Btrfs has seperate zones used for
> data and metadata, and these chunks are allocated in 1gb chunks, so
> you have 212gb of space thats allowed for data use, and 16gb thats
> allowed for metadata use.  By default every 12 (or it may be 8, i 
> forget) chunks we allocate for data, we allocate 1 for metadata,
> which ends up with like 8% of the disk being used for metadata.

Hmm, okay.  I was aware of the fact that it didn't handle the disk
filling up too well, but I didn't know that would cause an issue so long
before the disk has gone full?  I mean, according to df my there's 16 GB
of files on my disk (something I verified with du), while according to
you there should be 212 GB available for that in total.

Has metadata and data _both_ been stored in the 16 GB zone reserved for
metadata, for some reason?  It would make sense that I ran into ENOSPC
in that case, since the metadata-reserved zone now is indeed completely
full.  If I understand you correctly, though, the data is stored in the
212 GB large zone, not the 16 GB large metadata zone - but if that's the
case I don't understand how I could have hit ENOSPC?

> Now in your case you can run btrfsctl -b and it will re-balance the
> space on the drive, and it may give you more space back.  It could
> also possible panic the box, so make sure you are all backed up :).

Thanks for the tipe, but the "-b" option doesn't seem to be present in
btrfsctl (built from a day-fresh git checkout)...?

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 04:15:21PM +0200, Tore Anderson wrote:
> Hello Josef,
> 
> * Josef Bacik
> 
> > This is one of the gotchas of btrfs, there is not proper ENOSPC
> > handling, just a few things in place that are a bit conservative to
> > make sure you don't panic the box.  Btrfs has seperate zones used for
> > data and metadata, and these chunks are allocated in 1gb chunks, so
> > you have 212gb of space thats allowed for data use, and 16gb thats
> > allowed for metadata use.  By default every 12 (or it may be 8, i 
> > forget) chunks we allocate for data, we allocate 1 for metadata,
> > which ends up with like 8% of the disk being used for metadata.
> 
> Hmm, okay.  I was aware of the fact that it didn't handle the disk
> filling up too well, but I didn't know that would cause an issue so long
> before the disk has gone full?  I mean, according to df my there's 16 GB
> of files on my disk (something I verified with du), while according to
> you there should be 212 GB available for that in total.
> 
> Has metadata and data _both_ been stored in the 16 GB zone reserved for
> metadata, for some reason?  It would make sense that I ran into ENOSPC
> in that case, since the metadata-reserved zone now is indeed completely
> full.  If I understand you correctly, though, the data is stored in the
> 212 GB large zone, not the 16 GB large metadata zone - but if that's the
> case I don't understand how I could have hit ENOSPC?
>

Ooookay, now that I'm awake lets try this again :).  You are right, I thought it
was 212GB used, not 16GB used.  Something has gone horribly wrong, btrfs seems
to think thats all the space it can use.  The command you need to use is
btrfs-vol -b.  Please try that and see if it behaves better.  Before you run
that please do a btrfs-show and post the output, I'd like to see how big the fs
thinks its supposed to be.  Thanks,

Josef

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

* Josef Bacik

> Ooookay, now that I'm awake lets try this again :).  You are right, I
> thought it was 212GB used, not 16GB used.  Something has gone
> horribly wrong, btrfs seems to think thats all the space it can use.
> The command you need to use is btrfs-vol -b.  Please try that and see
> if it behaves better.  Before you run that please do a btrfs-show and
> post the output, I'd like to see how big the fs thinks its supposed
> to be.

It doesn't print much of anything, I'm afraid:

root@echo:~/misc/btrfs-progs-unstable# ./btrfs-show /dev/mapper/vg_echo-lv_root  
failed to read /dev/sr0
failed to read /dev/sdf
failed to read /dev/sde
failed to read /dev/sdd
failed to read /dev/sdc
failed to read /dev/sdb
Btrfs Btrfs v0.19

Same output with btrfs-progs-0.19.  The devices it's complaining about
is a USB memory card reader and my optical drive, if I disconnect
those from the SCSI layer the output is simply "Btrfs Btrfs v0.19".

I tried to run btrfsck earlier today, but it only gave me an error
message.  Perhaps that just made things worse?

I've got a btrfs-image dump that I took right after the problem showed
for the first time (before the fsck), by the way.  It's available at
<http://greed.fud.no/hosed_rootfs.btrfsimage.Z> in case you have any
use for it.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 07:41:49PM +0200, Tore Anderson wrote:
> * Josef Bacik
> 
> > Ooookay, now that I'm awake lets try this again :).  You are right, I
> > thought it was 212GB used, not 16GB used.  Something has gone
> > horribly wrong, btrfs seems to think thats all the space it can use.
> > The command you need to use is btrfs-vol -b.  Please try that and see
> > if it behaves better.  Before you run that please do a btrfs-show and
> > post the output, I'd like to see how big the fs thinks its supposed
> > to be.
> 
> It doesn't print much of anything, I'm afraid:
> 
> root@echo:~/misc/btrfs-progs-unstable# ./btrfs-show /dev/mapper/vg_echo-lv_root  
> failed to read /dev/sr0
> failed to read /dev/sdf
> failed to read /dev/sde
> failed to read /dev/sdd
> failed to read /dev/sdc
> failed to read /dev/sdb
> Btrfs Btrfs v0.19
> 
> Same output with btrfs-progs-0.19.  The devices it's complaining about
> is a USB memory card reader and my optical drive, if I disconnect
> those from the SCSI layer the output is simply "Btrfs Btrfs v0.19".
> 
> I tried to run btrfsck earlier today, but it only gave me an error
> message.  Perhaps that just made things worse?
> 
> I've got a btrfs-image dump that I took right after the problem showed
> for the first time (before the fsck), by the way.  It's available at
> <http://greed.fud.no/hosed_rootfs.btrfsimage.Z> in case you have any
> use for it.
> 

What kernel are you running?  I need to get a look at what code you are on.
Thanks for the image, I'm having some trouble getting it to work, but I will let
you know what I come up with.

Josef

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

* Josef Bacik

> What kernel are you running?  I need to get a look at what code you
> are on. Thanks for the image, I'm having some trouble getting it to
> work, but I will let you know what I come up with.

I was running 2.6.29.4-167.fc11.i686.PAE at the time of the crash, and
when I've been having problems.  I see that there's a new F11 kernel
out - I'll try to upgrade to that one and will let you know if there's
any change (I doubt it - there's no mention of btrfs-related changes).

Here's the output from btrfs-show, just figured out that I shouldn't
have supplied any arguments (thanks to Robert F=F6rster):

Label: none  uuid: 94762921-dbc0-4faf-a4e5-f5acdaf305a8
	Total devices 1 FS bytes used 15.82GB
	devid    1 size 227.53GB used 227.53GB path /dev/root

I just tried to make a new image with btrfs-image from
btrfs-progs-unstable (using "btrfs-image -c 9 /dev/vg_echo/lv_root
image2.Z").
Unlike when making the first image, the file system was mounted.  The
image is at <http://greed.fud.no/image2.Z> - is that any better?

Best regards,
--=20
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 08:14:05PM +0200, Tore Anderson wrote:
> * Josef Bacik
>=20
> > What kernel are you running?  I need to get a look at what code you
> > are on. Thanks for the image, I'm having some trouble getting it to
> > work, but I will let you know what I come up with.
>=20
> I was running 2.6.29.4-167.fc11.i686.PAE at the time of the crash, an=
d
> when I've been having problems.  I see that there's a new F11 kernel
> out - I'll try to upgrade to that one and will let you know if there'=
s
> any change (I doubt it - there's no mention of btrfs-related changes)=
=2E
>=20
> Here's the output from btrfs-show, just figured out that I shouldn't
> have supplied any arguments (thanks to Robert F=F6rster):
>=20
> Label: none  uuid: 94762921-dbc0-4faf-a4e5-f5acdaf305a8
> 	Total devices 1 FS bytes used 15.82GB
> 	devid    1 size 227.53GB used 227.53GB path /dev/root
>=20
> I just tried to make a new image with btrfs-image from
> btrfs-progs-unstable (using "btrfs-image -c 9 /dev/vg_echo/lv_root
> image2.Z").
> Unlike when making the first image, the file system was mounted.  The
> image is at <http://greed.fud.no/image2.Z> - is that any better?
>=20

Ok good news is, btrfs-vol -b will fix your problem.  Somehow most of y=
our disk has
been allocated to use metadata.  So did you have a whole bunch of stuff=
 on this
disk and then delete it all?  Because that would put you in that situat=
ion.  If
you have not then there is likely a bug in the metadata ratio stuff tha=
t needs
to be fixed.  Thankfully btrfs-vol -b will move everything around so yo=
u are
good to go, so if it is a bug that will keep you going until I can find=
 and fix
the problem.  btrfs-vol -b will take a while to run, so don't worry if =
it sits
there forever, you can run dmesg to watch its progress.  Let me know if=
 you had
a bunch of stuff on this disk that you deleted, cause if there was good=
 reason
for it to allocate all this metadata space then thats fine, but if not =
I need to
start looking at a root cause.  Thanks,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

* Josef Bacik

> Ok good news is, btrfs-vol -b will fix your problem.  Somehow most of
> your disk has been allocated to use metadata.  So did you have a
> whole bunch of stuff on this disk and then delete it all?  Because
> that would put you in that situation.  If you have not then there is
> likely a bug in the metadata ratio stuff that needs to be fixed.

As far as I know there hasn't been a lot of stuff on the file system,
I'm afraid.  The file system was created by the Fedora 11 installer,
and it has just been used as the system drive (I've got /home on NFS).

btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn.
I've was able to get output from dmesg before my SSH sessions started
hanging - maybe you can make anything out of it?  Anyway, right now I
have no more remote access to the box so any further debugging will
have to wait until tomorrow morning.

btrfs relocating chunk 129952120832
btrfs relocating block group 129952120832 flags 1
btrfs allocation failed flags 1, wanted 4096
space_info has 4096 free, is full
space_info total=16647192576, pinned=0, delalloc=57856000, may_use=1691648, used=16645496832
block group 12582912 has 8388608 bytes, 8384512 used 0 pinned 4096 reserved
0 blocks of free space at or bigger than bytes is
block group 1103101952 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 3250585600 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 4324327424 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 5398069248 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 6471811072 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 7545552896 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 30094131200 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 31167873024 has 1073741824 bytes, 1072078848 used 0 pinned 1662976 reserved
0 blocks of free space at or bigger than bytes is
block group 32241614848 has 1073741824 bytes, 1073721344 used 0 pinned 20480 reserved
0 blocks of free space at or bigger than bytes is
block group 40831549440 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 46200258560 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 47274000384 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserve
0 blocks of free space at or bigger than bytes is
block group 49421484032 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 58011418624 has 1073741824 bytes, 1073737728 used 0 pinned 4096 reserved
0 blocks of free space at or bigger than bytes is
block group 128878379008 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 129952120832 has 532676608 bytes, 532672512 used 0 pinned 0 reserved
entry offset 130484793344, bytes 4096
1 blocks of free space at or bigger than bytes is
------------[ cut here ]------------
kernel BUG at fs/btrfs/extent-tree.c:2905!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings
Modules linked in: nls_utf8 cifs nfs lockd nfs_acl auth_rpcgss ipt_MASQUERADE iptable_nat nf_nat rfcomm sco bridge stp llc bnep l2cap sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq dm_multipath kvm_intel kvm uinput snd_hda_codec_realtek snd_hda_intel firewire_ohci ppdev snd_hda_codec firewire_core i2c_i801 snd_hwdep snd_pcm snd_timer crc_itu_t btusb snd soundcore snd_page_alloc usb_storage parport_pc floppy parport bluetooth sky2 asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support serio_raw pata_jmicron ata_generic pata_acpi btrfs zlib_deflate libcrc32c nouveau drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]  

Pid: 30, comm: pdflush Not tainted (2.6.29.6-213.fc11.i686.PAE #1) P5K-VM
EIP: 0060:[<f8a0d543>] EFLAGS: 00010257 CPU: 1
EIP is at __btrfs_reserve_extent+0x339/0x347 [btrfs]
EAX: f5c8aadc EBX: f5c8aa50 ECX: 00000000 EDX: 00000001
ESI: f5c8aadc EDI: f58c786c EBP: f5c03bfc ESP: f5c03ba4
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process pdflush (pid: 30, ti=f5c02000 task=f5c08000 task.ti=f5c02000)
Stack:
 fffff000 00000000 00000000 f5c03ccb ffffffff ffffffff c1c00000 0000000d
 c3296380 00000024 00000000 00000000 00001000 00000000 00000000 f4d09690
 f5c8aad0 00001000 00000000 0019b000 f5c03ccb f5c03c50 f5c03c48 f8a0d70e
Call Trace:
 [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs]
 [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs]
 [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs]
 [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs]
 [<c0564dfb>] ? __lookup_tag+0x89/0xe3
 [<c0564ec4>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e
 [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs]
 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs]
 [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs]
 [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs]
 [<c0484bb8>] ? do_writepages+0x25/0x39
 [<c04be369>] ? __writeback_single_inode+0x140/0x241
 [<c0667944>] ? dm_any_congested+0x32/0x3d
 [<f8a15c3b>] ? btrfs_congested_fn+0x38/0x66 [btrfs]
 [<c04be7c7>] ? generic_sync_sb_inodes+0x1d9/0x2f6
 [<c04bea8d>] ? writeback_inodes+0x82/0xca
 [<c0485288>] ? background_writeout+0x7b/0xa7
 [<c04859be>] ? pdflush+0x130/0x1dc
 [<c048520d>] ? background_writeout+0x0/0xa7
 [<c048588e>] ? pdflush+0x0/0x1dc  
 [<c04468bc>] ? kthread+0x41/0x65  
 [<c044687b>] ? kthread+0x0/0x65   
 [<c0409dbf>] ? kernel_thread_helper+0x7/0x10
Code: e8 ef 3e a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 0c c9 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83
EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f5c03ba4
---[ end trace 209348013c0e69ac ]---
btrfs allocation failed flags 1, wanted 4096
space_info has 4096 free, is full  
space_info total=16647192576, pinned=0, delalloc=132501504, may_use=1691648, used=16645500928
block group 12582912 has 8388608 bytes, 8388608 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 1103101952 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 3250585600 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 4324327424 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 5398069248 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 6471811072 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 7545552896 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 30094131200 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 31167873024 has 1073741824 bytes, 1072078848 used 0 pinned 1662976 reserved
0 blocks of free space at or bigger than bytes is
block group 32241614848 has 1073741824 bytes, 1073721344 used 0 pinned 20480 reserved
0 blocks of free space at or bigger than bytes is
block group 40831549440 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 46200258560 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 47274000384 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 49421484032 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 58011418624 has 1073741824 bytes, 1073737728 used 0 pinned 4096 reserved
0 blocks of free space at or bigger than bytes is
block group 128878379008 has 1073741824 bytes, 1073741824 used 0 pinned 0 reserved
0 blocks of free space at or bigger than bytes is
block group 129952120832 has 532676608 bytes, 532672512 used 0 pinned 0 reserved
entry offset 130484793344, bytes 4096
1 blocks of free space at or bigger than bytes is
------------[ cut here ]------------
kernel BUG at fs/btrfs/extent-tree.c:2905!
invalid opcode: 0000 [#2] SMP
last sysfs file: /sys/devices/system/cpu/sched_mc_power_savings
Modules linked in: nls_utf8 cifs nfs lockd nfs_acl auth_rpcgss ipt_MASQUERADE iptable_nat nf_nat rfcomm sco bridge stp llc bnep l2cap sunrpc ip6t_REJECT nf_conntrack_ipv6 ip6table_filter ip6_tables ipv6 cpufreq_ondemand acpi_cpufreq dm_multipath kvm_intel kvm uinput snd_hda_codec_realtek snd_hda_intel firewire_ohci ppdev snd_hda_codec firewire_core i2c_i801 snd_hwdep snd_pcm snd_timer crc_itu_t btusb snd soundcore snd_page_alloc usb_storage parport_pc floppy parport bluetooth sky2 asus_atk0110 hwmon iTCO_wdt iTCO_vendor_support serio_raw pata_jmicron ata_generic pata_acpi btrfs zlib_deflate libcrc32c nouveau drm i2c_algo_bit i2c_core [last unloaded: scsi_wait_scan]  

Pid: 2656, comm: btrfs-vol Tainted: G      D    (2.6.29.6-213.fc11.i686.PAE #1) P5K-VM
EIP: 0060:[<f8a0d543>] EFLAGS: 00210257 CPU: 1
EIP is at __btrfs_reserve_extent+0x339/0x347 [btrfs]
EAX: f5c8aadc EBX: f5c8aa50 ECX: 00000000 EDX: 00000001
ESI: f5c8aadc EDI: f58c786c EBP: f09cb7e4 ESP: f09cb78c
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
Process btrfs-vol (pid: 2656, ti=f09ca000 task=f0995860 task.ti=f09ca000)
Stack:
 fffff000 00000000 00000000 f09cb8b3 ffffffff ffffffff 00000000 00000000
 00000000 00000024 00000000 00000000 00001000 00000000 00000000 f4d096f0
 f5c8aad0 00001000 00000000 000aa000 f09cb8b3 f09cb838 f09cb830 f8a0d70e
Call Trace:
 [<f8a0d70e>] ? btrfs_reserve_extent+0x40/0x64 [btrfs]
 [<f8a1e0fe>] ? cow_file_range+0x258/0x41b [btrfs]
 [<f8a1ea16>] ? run_delalloc_range+0xb0/0x31f [btrfs]
 [<f8a32c19>] ? __extent_writepage+0x1f6/0x7bd [btrfs]
 [<c0564dc7>] ? __lookup_tag+0x55/0xe3
 [<c0564ec4>] ? radix_tree_gang_lookup_tag_slot+0x6f/0x8e
 [<f8a3352a>] ? extent_write_cache_pages.clone.0+0x10c/0x1ec [btrfs]
 [<c0716989>] ? _spin_lock_irq+0x21/0x25
 [<c043dac2>] ? run_timer_softirq+0x1ae/0x1c0
 [<f8a33714>] ? extent_writepages+0x3f/0x53 [btrfs]
 [<f8a1c869>] ? btrfs_get_extent+0x0/0x927 [btrfs]
 [<f8a1c735>] ? btrfs_writepages+0x20/0x25 [btrfs]
 [<c0484bb8>] ? do_writepages+0x25/0x39
 [<c04be369>] ? __writeback_single_inode+0x140/0x241
 [<c056459d>] ? prop_fraction_single+0x35/0x55
 [<c04be7c7>] ? generic_sync_sb_inodes+0x1d9/0x2f6
 [<c04bea8d>] ? writeback_inodes+0x82/0xca
 [<c0485464>] ? balance_dirty_pages_ratelimited_nr+0x137/0x23a
 [<f8a0ad09>] ? relocate_inode_pages+0x2fa/0x305 [btrfs]
 [<f8a0ae69>] ? relocate_data_extent+0x155/0x173 [btrfs]
 [<f8a165cf>] ? btrfs_read_fs_root_no_name+0x77/0x100 [btrfs]
 [<f8a0fb0e>] ? relocate_one_extent+0x181/0x371 [btrfs]
 [<f8a18e57>] ? btrfs_end_transaction+0xf/0x11 [btrfs]
 [<f8a0a26f>] ? __alloc_chunk_for_shrink+0x10d/0x125 [btrfs]
 [<f8a0ff52>] ? btrfs_relocate_block_group+0x254/0x3dc [btrfs]
 [<f8a36105>] ? btrfs_relocate_chunk+0x53/0x419 [btrfs]
 [<f8a2e89e>] ? map_private_extent_buffer+0x96/0xb8 [btrfs]
 [<f8a2e90f>] ? map_extent_buffer+0x4f/0x7f [btrfs]
 [<c0425a4b>] ? kunmap_atomic+0x6e/0x7c
 [<f8a2e32f>] ? unmap_extent_buffer+0x11/0x13 [btrfs]
 [<f8a274c1>] ? btrfs_dev_extent_chunk_offset+0xa2/0xae [btrfs]
 [<f8a366b3>] ? btrfs_shrink_device+0x1e8/0x29c [btrfs]
 [<f8a3692a>] ? btrfs_balance+0xec/0x243 [btrfs]
 [<c053947f>] ? avc_has_perm+0x41/0x4e
 [<f8a3a8e1>] ? btrfs_ioctl+0x72c/0x837 [btrfs]
 [<f8a3a1b5>] ? btrfs_ioctl+0x0/0x837 [btrfs]
 [<c04b2623>] ? vfs_ioctl+0x1d/0x76
 [<c04b2f16>] ? do_vfs_ioctl+0x480/0x4ba
 [<c053b04c>] ? selinux_file_ioctl+0x43/0x46
 [<c04b2f96>] ? sys_ioctl+0x46/0x66
 [<c040945f>] ? sysenter_do_call+0x12/0x34
Code: e8 ef 3e a1 c7 90 8b 9b 80 00 00 00 83 c3 80 8b 83 80 00 00 00 0f 18 00 90 8d 83 80 00 00 00 39 45 e8 75 99 89 f0 e8 0c c9 a3 c7 <0f> 0b eb fe 8d 65 f4 31 c0 5b 5e 5f 5d c3 55 89 e5 57 56 53 83
EIP: [<f8a0d543>] __btrfs_reserve_extent+0x339/0x347 [btrfs] SS:ESP 0068:f09cb78c
---[ end trace 209348013c0e69ad ]---

-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Chris Mason]

On Thu, Aug 06, 2009 at 08:49:39PM +0200, Tore Anderson wrote:
> * Josef Bacik
> 
> > Ok good news is, btrfs-vol -b will fix your problem.  Somehow most of
> > your disk has been allocated to use metadata.  So did you have a
> > whole bunch of stuff on this disk and then delete it all?  Because
> > that would put you in that situation.  If you have not then there is
> > likely a bug in the metadata ratio stuff that needs to be fixed.
> 
> As far as I know there hasn't been a lot of stuff on the file system,
> I'm afraid.  The file system was created by the Fedora 11 installer,
> and it has just been used as the system drive (I've got /home on NFS).
> 
> btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn.
> I've was able to get output from dmesg before my SSH sessions started
> hanging - maybe you can make anything out of it?  Anyway, right now I
> have no more remote access to the box so any further debugging will
> have to wait until tomorrow morning.
> 
> btrfs relocating chunk 129952120832
> btrfs relocating block group 129952120832 flags 1

Ugh, it hit the data group first, which it can't relocate because there
is nowhere to put the blocks.

-chris

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 08:49:39PM +0200, Tore Anderson wrote:
> * Josef Bacik
> 
> > Ok good news is, btrfs-vol -b will fix your problem.  Somehow most of
> > your disk has been allocated to use metadata.  So did you have a
> > whole bunch of stuff on this disk and then delete it all?  Because
> > that would put you in that situation.  If you have not then there is
> > likely a bug in the metadata ratio stuff that needs to be fixed.
> 
> As far as I know there hasn't been a lot of stuff on the file system,
> I'm afraid.  The file system was created by the Fedora 11 installer,
> and it has just been used as the system drive (I've got /home on NFS).
> 
> btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn.
> I've was able to get output from dmesg before my SSH sessions started
> hanging - maybe you can make anything out of it?  Anyway, right now I
> have no more remote access to the box so any further debugging will
> have to wait until tomorrow morning.
> 

Hrm yeah I was afraid of that.  So you will have to try and free up some data so
the balancer as enough room to move things around.  Also one thing that _may_
help is mounting with nocow for a little bit, so any writing (like logging and
such) just over-writes stuff instead of needing a new extent, until btrfs-vol -b
can finish, and then you can go back to the normal mount options.  I still can't
figure out how this happened, but one thing to do for now would be to mount with
metadata_ratio=100.  This will make you more likely to panic in the event that
you run out of metadata, but its not that likely to happen, especially with as
much disk space as you have.  Sorry about all of this, hopefully a better
solution will be coming down in a few months.  Thanks,

Josef

Re: Crash in __btrfs_reserve_extent

[Chris Mason]

On Thu, Aug 06, 2009 at 03:01:18PM -0400, Josef Bacik wrote:
> On Thu, Aug 06, 2009 at 08:49:39PM +0200, Tore Anderson wrote:
> > * Josef Bacik
> > 
> > > Ok good news is, btrfs-vol -b will fix your problem.  Somehow most of
> > > your disk has been allocated to use metadata.  So did you have a
> > > whole bunch of stuff on this disk and then delete it all?  Because
> > > that would put you in that situation.  If you have not then there is
> > > likely a bug in the metadata ratio stuff that needs to be fixed.
> > 
> > As far as I know there hasn't been a lot of stuff on the file system,
> > I'm afraid.  The file system was created by the Fedora 11 installer,
> > and it has just been used as the system drive (I've got /home on NFS).
> > 
> > btrfs-vol -b / (from btrfs-progs-0.19) made my system crash and burn.
> > I've was able to get output from dmesg before my SSH sessions started
> > hanging - maybe you can make anything out of it?  Anyway, right now I
> > have no more remote access to the box so any further debugging will
> > have to wait until tomorrow morning.
> > 
> 
> Hrm yeah I was afraid of that.  So you will have to try and free up some data so
> the balancer as enough room to move things around.  Also one thing that _may_
> help is mounting with nocow for a little bit, so any writing (like logging and
> such) just over-writes stuff instead of needing a new extent, until btrfs-vol -b
> can finish, and then you can go back to the normal mount options.  I still can't
> figure out how this happened, but one thing to do for now would be to mount with
> metadata_ratio=100.  This will make you more likely to panic in the event that
> you run out of metadata, but its not that likely to happen, especially with as
> much disk space as you have.  Sorry about all of this, hopefully a better
> solution will be coming down in a few months.  Thanks,

We can code a fix for the balancer problem quickly, but you might have a
hard time recompiling it and getting it on that box.

If this data is critical we'll write something up.

-chris

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

Hi,

* Josef Bacik

>> Sorry about all of this, hopefully a better solution will be
>> coming down in a few months.

Don't be - the only reason why I'm running btrfs at this point is to
test and hopefully help out improving it by reporting the problems I'm
running into.  :-)

Thanks for the suggestions!

* Chris Mason

> We can code a fix for the balancer problem quickly, but you might
> have a hard time recompiling it and getting it on that box.
> 
> If this data is critical we'll write something up.

There's no critical data on this filesystem, it's only used for my
Fedora installation.  Won't take me long to re-install from scratch if
that's what it takes.  All og my important data (home directory) is on
NFS.  Using NFS I can easily get new versions of btrfs-progs onto the
box with the hosed btrfs, too.

So if you meant to code up the fix as a band-aid only for me, don't
bother (but thanks for the offer)!  On the other hand, if you wanted to
get such a fix into btrfs-progs anyway I'll be happy to test it out for you.

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Robert Förster]

QW0gRG9ubmVyc3RhZywgNi4gQXVndXN0IDIwMDkgMTc6MzY6MTAgc2NocmllYiBSb2JlcnQgRvZy
c3RlcjoKPiBBbSBEb25uZXJzdGFnLCA2LiBBdWd1c3QgMjAwOSAxNzoyMDo1OCBzY2hyaWViIEpv
c2VmIEJhY2lrOgo+ID4gT24gVGh1LCBBdWcgMDYsIDIwMDkgYXQgMDQ6NTc6NTBQTSArMDIwMCwg
Um9iZXJ0IEb2cnN0ZXIgd3JvdGU6Cj4gPiA+IEhlbGxvLAo+ID4gPgo+ID4gPiBpIHRob3VnaHQg
aSdkIGp1bXAgaW4gc2luY2UgaSBoYXZlIHRoZSBzYW1lIGlzc3VlOgo+ID4gPiBidHJmcy1zaG93
IG91dHB1dDoKPiA+ID4KPiA+ID4gTGFiZWw6IG5vbmUgIHV1aWQ6IDdmNGYzYmNiLTdiYTEtNDlj
NS04OTk2LWU4OTdiNTQ4M2M3Ywo+ID4gPiAgICAgICAgIFRvdGFsIGRldmljZXMgMSBGUyBieXRl
cyB1c2VkIDUuMzJHQgo+ID4gPiAgICAgICAgIGRldmlkICAgIDEgc2l6ZSA1MS4yMkdCIHVzZWQg
NTEuMjJHQiBwYXRoIC9kZXYvc2RiMQo+ID4gPgo+ID4gPiBkZiAtaCBvdXRwdXQ6Cj4gPiA+Cj4g
PiA+IEZpbGVzeXN0ZW0gICAgICAgICAgICBTaXplICBVc2VkIEF2YWlsIFVzZSUgTW91bnRlZCBv
bgo+ID4gPiByb290ZnMgICAgICAgICAgICAgICAgIDUyRyAgNSw0RyAgIDQ2RyAgMTElIC8KPiA+
Cj4gPiBBbHJpZ2h0eSwgc28gYnRyZnMtc2hvdyBzYXlzIHlvdSBoYXZlIHVzZWQgdXAgYWxsIG9m
IHlvdXIgZGlzaywgYnV0IGZvcgo+ID4gc29tZSByZWFzb24gc3RhdGZzIGlzIHNheWluZyB5b3Ug
aGF2ZW50LiAgQ2FuIHlvdSBjZCBpbnRvIC8gYW5kIHJ1bgo+ID4KPiA+IGR1IC1oIC1zCj4gPgo+
ID4gYW5kIHRoZW4gd2FpdCBhIHZlcnkgbG9uZyB0aW1lIGFuZCB0aGVuIHRlbGwgbWUgd2hhdCBp
dCBjb21lcyBiYWNrIHdpdGgKPiA+IDopLiBUaGF0IHdpbGwgdGVsbCB1cyBmb3Igc3VyZSB3aGF0
IHRoZSBoZWxsIGlzIGdvaW5nIG9uLiAgVGhhbmsgeW91LAo+ID4KPiA+IEpvc2VmCj4KPiA0MEcg
ICAgIC4KaSdkIHJ1biBidHJmcy12b2wgLWIgeWVzdGVyZGF5LCBoZXJlIGlzIHdoYXQgaGFwcGVu
ZDoKCkF1ZyAwNiAyMjozMzoxMyBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91
cCAzNTQ2Mjg0MDMyMCBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozMzoxNiBba2VybmVsXSBidHJm
czogZm91bmQgMTkgZXh0ZW50cyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAg
ICAgICAgICAgICAtIExhc3Qgb3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozMzoxOSBba2VybmVsXSBidHJmczogcmVsb2Nh
dGluZyBibG9jayBncm91cCAzNDM4OTA5ODQ5NiBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozMzoy
MyBba2VybmVsXSBidHJmczogZm91bmQgMTQgZXh0ZW50cyAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgCiAgICAgICAgICAgICAgICAtIExhc3Qgb3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0g
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozMzoyOSBba2VybmVs
XSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAzMzMxNTM1NjY3MiBmbGFncyAxICAgICAg
CkF1ZyAwNiAyMjozMzozMCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAz
MjI0MTYxNDg0OCBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozMzozMCBba2VybmVsXSBidHJmczog
cmVsb2NhdGluZyBibG9jayBncm91cCAzMTE2Nzg3MzAyNCBmbGFncyAxICAgICAgCkF1ZyAwNiAy
MjozMzozMiBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAzMDA5NDEzMTIw
MCBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozMzozMiBba2VybmVsXSBidHJmczogcmVsb2NhdGlu
ZyBibG9jayBncm91cCAyOTAyMDM4OTM3NiBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozMzozNCBb
a2VybmVsXSBidHJmczogZm91bmQgNDcgZXh0ZW50cyAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgCiAgICAgICAgICAgICAgICAtIExhc3Qgb3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozMzozOSBba2VybmVsXSBi
dHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAyNzk0NjY0NzU1MiBmbGFncyAzNiAgICAgCkF1
ZyAwNiAyMjozNTowNSBba2VybmVsXSBidHJmczogZm91bmQgOTAwIGV4dGVudHMgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozNTowNSBba2VybmVsXSBidHJmczogcmVs
b2NhdGluZyBibG9jayBncm91cCAyNjg3MjkwNTcyOCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjoz
NToxMSBba2VybmVsXSBidHJmczogZm91bmQgMjYgZXh0ZW50cyAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgCkF1ZyAwNiAyMjozNToxMSBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBi
bG9jayBncm91cCAyNTc5OTE2MzkwNCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNjoyNyBba2Vy
bmVsXSBidHJmczogZm91bmQgNjE5NCBleHRlbnRzICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgCkF1ZyAwNiAyMjozNjoyNyBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91
cCAyNDcyNTQyMjA4MCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNjo0MyBba2VybmVsXSBidHJm
czogZm91bmQgMzIxIGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAw
NiAyMjozNjo0MyBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAyMzY1MTY4
MDI1NiBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNzowOCBba2VybmVsXSBidHJmczogZm91bmQg
MTk2MyBleHRlbnRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozNzow
OCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAyMjU3NzkzODQzMiBmbGFn
cyAxICAgICAgCkF1ZyAwNiAyMjozNzowOCBba2VybmVsXSBidHJmczogZm91bmQgMiBleHRlbnRz
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAtIExhc3Qg
b3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
CkF1ZyAwNiAyMjozNzowOSBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAy
MTUwNDE5NjYwOCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNzoxNCBba2VybmVsXSBidHJmczog
Zm91bmQgMTg4IGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAy
MjozNzoxNCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAyMDQzMDQ1NDc4
NCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNzoyMCBba2VybmVsXSBidHJmczogZm91bmQgMTYz
IGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjozNzoyMCBb
a2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxOTM1NjcxMjk2MCBmbGFncyAx
ICAgICAgCkF1ZyAwNiAyMjozNzoyMCBba2VybmVsXSBidHJmczogZm91bmQgMiBleHRlbnRzICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAtIExhc3Qgb3V0
cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1
ZyAwNiAyMjozNzoyMSBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxODI4
Mjk3MTEzNiBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozNzoyOCBba2VybmVsXSBidHJmczogZm91
bmQgMjIwIGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjoz
NzoyOCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxNzIwOTIyOTMxMiBm
bGFncyAxICAgICAgCkF1ZyAwNiAyMjozNzozMSBba2VybmVsXSBidHJmczogZm91bmQgNyBleHRl
bnRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAtIExh
c3Qgb3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgCkF1ZyAwNiAyMjozNzozNCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91
cCAxNjEzNTQ4NzQ4OCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozODowMyBba2VybmVsXSBidHJm
czogZm91bmQgNzAyIGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAw
NiAyMjozODowMyBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxNTA2MTc0
NTY2NCBmbGFncyAxICAgICAgCkF1ZyAwNiAyMjozODowNyBba2VybmVsXSBidHJmczogZm91bmQg
MzQ2IGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAg
ICAtIExhc3Qgb3V0cHV0IHJlcGVhdGVkIHR3aWNlIC0gICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgCkF1ZyAwNiAyMjozODoxMyBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9j
ayBncm91cCAxMzk4ODAwMzg0MCBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozODo1MiBba2VybmVs
XSBidHJmczogZm91bmQgMTIzNyBleHRlbnRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
CkF1ZyAwNiAyMjozODo1MiBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAx
MjkxNDI2MjAxNiBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjozOTo0MCBba2VybmVsXSBidHJmczog
Zm91bmQgMTU3MiBleHRlbnRzICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAy
MjozOTo0MCBba2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxMTg0MDUyMDE5
MiBmbGFncyAzNiAgICAgCkF1ZyAwNiAyMjo0MDoxMSBba2VybmVsXSBidHJmczogZm91bmQgNzk2
IGV4dGVudHMgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxMSBb
a2VybmVsXSBidHJmczogcmVsb2NhdGluZyBibG9jayBncm91cCAxMDc2Njc3ODM2OCBmbGFncyAx
ICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAtLS0tLS0tLS0tLS1bIGN1dCBoZXJlIF0t
LS0tLS0tLS0tLS0gICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBX
QVJOSU5HOiBhdCBmcy9idHJmcy9yZWxvY2F0aW9uLmM6MzUxOSAKYnRyZnNfcmVsb2NhdGVfYmxv
Y2tfZ3JvdXArMHgyMDUvMHgzMDYoKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBIYXJkd2FyZSBu
YW1lOiBEZWxsIERYUDA1MSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0
MDoxOSBba2VybmVsXSBNb2R1bGVzIGxpbmtlZCBpbjogbnZpZGlhKFApIGkyY19pODAxICAgICAg
ICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBQaWQ6IDY3NywgY29tbTogYnRy
ZnMtdm9sIFRhaW50ZWQ6IFAgICAgICAgICAgIAoyLjYuMzEtcmM1LUJlYXRyaXggIzEgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBDYWxsIFRyYWNlOiAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxm
ZmZmZmZmZjgxMjU2YTA3Pl0gPyAKYnRyZnNfcmVsb2NhdGVfYmxvY2tfZ3JvdXArMHgyMDUvMHgz
MDYgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgx
MDQ2NGYxPl0gPyB3YXJuX3Nsb3dwYXRoX2NvbW1vbisweDdhLzB4ZDcgCkF1ZyAwNiAyMjo0MDox
OSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMjU2YTA3Pl0gPyAKYnRyZnNfcmVsb2NhdGVfYmxvY2tf
Z3JvdXArMHgyMDUvMHgzMDYgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVs
XSAgWzxmZmZmZmZmZjgxMjNlNzliPl0gPyBidHJmc19yZWxvY2F0ZV9jaHVuaysweDU5LzB4NTUx
CkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMjM0NzY2Pl0gPyBtYXBfZXh0
ZW50X2J1ZmZlcisweDhjLzB4ZjAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZm
ZmZmZjgxMjI4Y2I4Pl0gPyBidHJmc19pdGVtX29mZnNldCsweGU0LzB4ZWIgICAgCkF1ZyAwNiAy
Mjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMjNmMmMzPl0gPyBidHJmc19iYWxhbmNlKzB4
MjA5LzB4Mjg4ICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMjQz
Yzc1Pl0gPyBidHJmc19pb2N0bCsweDYwNC8weGJkZiAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBb
a2VybmVsXSAgWzxmZmZmZmZmZjgxMGQ0OTIyPl0gPyB2ZnNfaW9jdGwrMHgzNS8weGE4ICAgICAg
ICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMGQ0ZGFlPl0gPyBk
b192ZnNfaW9jdGwrMHgzNzcvMHg1NDEgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAg
WzxmZmZmZmZmZjgxMGQ1MDBkPl0gPyBzeXNfaW9jdGwrMHg5NS8weDllICAgICAgICAgICAgCkF1
ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMDBiZjZiPl0gPyBzeXN0ZW1fY2Fs
bF9mYXN0cGF0aCsweDE2LzB4MWIgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSAtLS1bIGVuZCB0
cmFjZSA3YzIyM2M5MTNlMmM1ZDNiIF0tLS0gICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0
MDoxOSBba2VybmVsXSBLZXJuZWwgQlVHIGF0IGZmZmZmZmZmODEyM2VhZjIgW3ZlcmJvc2UgZGVi
dWcgaW5mbyAKdW5hdmFpbGFibGVdICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2Vy
bmVsXSBDUFUgMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBNb2R1bGVzIGxpbmtlZCBpbjogbnZpZGlhKFAp
IGkyY19pODAxICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBQaWQ6
IDY3NywgY29tbTogYnRyZnMtdm9sIFRhaW50ZWQ6IFAgICAgICAgIFcgIAoyLjYuMzEtcmM1LUJl
YXRyaXggIzEgRGVsbCBEWFAwNTEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSSVA6IDAwMTA6Wzxm
ZmZmZmZmZjgxMjNlYWYyPl0gIFs8ZmZmZmZmZmY4MTIzZWFmMj5dIApidHJmc19yZWxvY2F0ZV9j
aHVuaysweDNiMC8weDU1MSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSU1A6IDAwMTg6ZmZmZjg4MDA1NmRk
OWM1OCAgRUZMQUdTOiAwMDAxMDI4NiAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVs
XSBSQVg6IDAwMDAwMDAwZmZmZmZmZjAgUkJYOiBmZmZmODgwMDdlZWJlMDAwIFJDWDogCmZmZmZm
ZmZmODEyNTY5MGIgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSRFg6IDAw
MDAwMDAwMDAwMDAwMDAgUlNJOiBmZmZmZWEwMDAxNzg1MDY4IFJESTogCmZmZmZmZmZmODE2ZDBi
ODAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSQlA6IGZmZmY4ODAwNGIw
NWZhMjAgUjA4OiAwMDAwMDAwMDAwMDAwMDAwIFIwOTogCjAwMDAwMDAwMDAwMDAwMDAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSMTA6IGZmZmY4ODAwNTFjM2RlNzAgUjEx
OiAwMDAwMDAwMGZmZmZmZmZmIFIxMjogCjAwMDAwMDAwMDAwMDAwMDAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1
ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBSMTM6IDAwMDAwMDAwMDAwMDAxMDAgUjE0OiBmZmZmODgw
MDdlZWU1MDAwIFIxNTogCmZmZmY4ODAwNTZkZDlkNzggICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0
MDoxOSBba2VybmVsXSBGUzogIDAwMDA3ZmEyMDhlOTk3MzAoMDAwMCkgR1M6ZmZmZjg4MDAwMTc5
ZjAwMCgwMDAwKSAKa25sR1M6MDAwMDAwMDAwMDAwMDAwMCAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCkF1ZyAwNiAyMjo0MDoxOSBba2Vy
bmVsXSBDUzogIDAwMTAgRFM6IDAwMDAgRVM6IDAwMDAgQ1IwOiAwMDAwMDAwMDgwMDUwMDNiICAg
ICAgCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSBDUjI6IDAwMDA3ZjdlNWU2ZmMwMDAgQ1IzOiAw
MDAwMDAwMDIxOGI3MDAwIENSNDogCjAwMDAwMDAwMDAwMDA2ZjAKQXVnIDA2IDIyOjQwOjE5IFtr
ZXJuZWxdIERSMDogMDAwMDAwMDAwMDAwMDAwMCBEUjE6IDAwMDAwMDAwMDAwMDAwMDAgRFIyOiAK
MDAwMDAwMDAwMDAwMDAwMApBdWcgMDYgMjI6NDA6MTkgW2tlcm5lbF0gRFIzOiAwMDAwMDAwMDAw
MDAwMDAwIERSNjogMDAwMDAwMDBmZmZmMGZmMCBEUjc6IAowMDAwMDAwMDAwMDAwNDAwCkF1ZyAw
NiAyMjo0MDoxOSBba2VybmVsXSBQcm9jZXNzIGJ0cmZzLXZvbCAocGlkOiA2NzcsIHRocmVhZGlu
Zm8gCmZmZmY4ODAwNTZkZDgwMDAsIHRhc2sgZmZmZjg4MDA0MTk5MWQ2MCkKQXVnIDA2IDIyOjQw
OjE5IFtrZXJuZWxdICBmZmZmODgwMDU2ZGQ5ZDA4IGZmZmZmZmZmODEyMzQ3NjYgZmZmZjg4MDA1
NmRkOWNmMCAKMDAwMDAwMDI4MWMwMDAwMApBdWcgMDYgMjI6NDA6MTkgW2tlcm5lbF0gPDA+IDAw
MDAwMDAwMDAwMDAxMDAgZmZmZjg4MDA1NmRkOWNmOCBmZmZmODgwMDU2ZGQ5ZDAwIApmZmZmODgw
MDdlZWUzODAwCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVsXSA8MD4gMDAwMDAwMDA5NjY0NDllZiAw
MDAwMDAwMDAwMDAwMWRjIGZmZmY4ODAwMWI0ZjU5OTAgCjAwMDAwMDAwMDAwMDAxZWQKQXVnIDA2
IDIyOjQwOjE5IFtrZXJuZWxdICBbPGZmZmZmZmZmODEyMzQ3NjY+XSA/IG1hcF9leHRlbnRfYnVm
ZmVyKzB4OGMvMHhmMApBdWcgMDYgMjI6NDA6MTkgW2tlcm5lbF0gIFs8ZmZmZmZmZmY4MTIyOGNi
OD5dID8gYnRyZnNfaXRlbV9vZmZzZXQrMHhlNC8weGViCkF1ZyAwNiAyMjo0MDoxOSBba2VybmVs
XSAgWzxmZmZmZmZmZjgxMjNmMmMzPl0gPyBidHJmc19iYWxhbmNlKzB4MjA5LzB4Mjg4CkF1ZyAw
NiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMjQzYzc1Pl0gPyBidHJmc19pb2N0bCsw
eDYwNC8weGJkZgpBdWcgMDYgMjI6NDA6MTkgW2tlcm5lbF0gIFs8ZmZmZmZmZmY4MTBkNDkyMj5d
ID8gdmZzX2lvY3RsKzB4MzUvMHhhOApBdWcgMDYgMjI6NDA6MTkgW2tlcm5lbF0gIFs8ZmZmZmZm
ZmY4MTBkNGRhZT5dID8gZG9fdmZzX2lvY3RsKzB4Mzc3LzB4NTQxCkF1ZyAwNiAyMjo0MDoxOSBb
a2VybmVsXSAgWzxmZmZmZmZmZjgxMGQ1MDBkPl0gPyBzeXNfaW9jdGwrMHg5NS8weDllCkF1ZyAw
NiAyMjo0MDoxOSBba2VybmVsXSAgWzxmZmZmZmZmZjgxMDBiZjZiPl0gPyBzeXN0ZW1fY2FsbF9m
YXN0cGF0aCsweDE2LzB4MWIKQXVnIDA2IDIyOjQwOjE5IFtrZXJuZWxdIFJJUCAgWzxmZmZmZmZm
ZjgxMjNlYWYyPl0gCmJ0cmZzX3JlbG9jYXRlX2NodW5rKzB4M2IwLzB4NTUxCkF1ZyAwNiAyMjo0
MDoxOSBba2VybmVsXSAgUlNQIDxmZmZmODgwMDU2ZGQ5YzU4PgpBdWcgMDYgMjI6NDA6MTkgW2tl
cm5lbF0gLS0tWyBlbmQgdHJhY2UgN2MyMjNjOTEzZTJjNWQzYyBdLS0tCgpyZWluc3RhbGxlZCBi
dHJmcy1wcmdzIGZyb20gZ2l0IGFuZCB3aXRoIGRlYnVnIHN5bWJvbHMgKHNpbmNlIGl0IHNlZ2Zh
dWx0ZWQgCndoZW4gaSB3b2tlIHVwKSByZXJ1bm5pbmcgaXQgYXQgdGhlIG1vbWVudCBzbyBtYXli
ZSBpdHMgZG9pbmcgdGhhdCBhZ2FpbiBhbmQgaSAKaGF2ZSBtb3JlIGluZm9tYXRpb24uCnRlbGwg
bWUgaWYgaSBjYW4gZG8gYW55dGhpbmcgbW9yZQoKUm9iZXJ0Cg==

Re: Crash in __btrfs_reserve_extent

[Robert Förster]

Am Donnerstag, 6. August 2009 19:15:35 schrieb Tore Anderson:
> * Josef Bacik
>
> > Alrighty, so btrfs-show says you have used up all of your disk, but
> > for some reason statfs is saying you havent.  Can you cd into / and
> > run
> >
> > du -h -s
> >
> > and then wait a very long time and then tell me what it comes back
> > with :). That will tell us for sure what the hell is going on.  Thank
> > you,
>
> I think this needs to be "du -h -s -x" (otherwise you'll include space
> used by files on other file systems as well).
>
> Best regards,
oops, yea, should probaly had a closer look at its actual output, then i would 
have noticed that.
reposting the rest too since i did a temporary file cleanup in the meantime if 
that even matters, and if i missed anything else:

Beatrix / # du -h -s -x
3,9G    .

Beatrix / # df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 52G  4,3G   47G   9% /
/dev/root              52G  4,3G   47G   9% /
rc-svcdir             1,0M   68K  956K   7% /lib64/rc/init.d
udev                   10M  300K  9,8M   3% /dev
shm                  1004M  5,6M  999M   1% /dev/shm
/dev/sda1             129G   36G   93G  28% /home

Beatrix / # btrfs-show
failed to read /dev/sdc
failed to read /dev/hdb
Label: none  uuid: 0f47144c-4fef-4291-a320-7859a0e7c359
        Total devices 1 FS bytes used 35.46GB
        devid    1 size 128.23GB used 90.04GB path /dev/sda1

Label: none  uuid: 7f4f3bcb-7ba1-49c5-8996-e897b5483c7c
        Total devices 1 FS bytes used 4.23GB
        devid    1 size 51.22GB used 51.22GB path /dev/sdb1

Btrfs Btrfs v0.19

Re: Crash in __btrfs_reserve_extent

[Tore Anderson]

* Josef Bacik

> Alrighty, so btrfs-show says you have used up all of your disk, but
> for some reason statfs is saying you havent.  Can you cd into / and
> run
> 
> du -h -s
> 
> and then wait a very long time and then tell me what it comes back
> with :). That will tell us for sure what the hell is going on.  Thank
> you,

I think this needs to be "du -h -s -x" (otherwise you'll include space
used by files on other file systems as well).

Best regards,
-- 
Tore Anderson
Redpill Linpro AS - http://www.redpill-linpro.com/
Tel: +47 21 54 41 27

Re: Crash in __btrfs_reserve_extent

[Robert Förster]

QW0gRG9ubmVyc3RhZywgNi4gQXVndXN0IDIwMDkgMTc6MjA6NTggc2NocmllYiBKb3NlZiBCYWNp
azoKPiBPbiBUaHUsIEF1ZyAwNiwgMjAwOSBhdCAwNDo1Nzo1MFBNICswMjAwLCBSb2JlcnQgRvZy
c3RlciB3cm90ZToKPiA+IEhlbGxvLAo+ID4KPiA+IGkgdGhvdWdodCBpJ2QganVtcCBpbiBzaW5j
ZSBpIGhhdmUgdGhlIHNhbWUgaXNzdWU6Cj4gPiBidHJmcy1zaG93IG91dHB1dDoKPiA+Cj4gPiBM
YWJlbDogbm9uZSAgdXVpZDogN2Y0ZjNiY2ItN2JhMS00OWM1LTg5OTYtZTg5N2I1NDgzYzdjCj4g
PiAgICAgICAgIFRvdGFsIGRldmljZXMgMSBGUyBieXRlcyB1c2VkIDUuMzJHQgo+ID4gICAgICAg
ICBkZXZpZCAgICAxIHNpemUgNTEuMjJHQiB1c2VkIDUxLjIyR0IgcGF0aCAvZGV2L3NkYjEKPiA+
Cj4gPiBkZiAtaCBvdXRwdXQ6Cj4gPgo+ID4gRmlsZXN5c3RlbSAgICAgICAgICAgIFNpemUgIFVz
ZWQgQXZhaWwgVXNlJSBNb3VudGVkIG9uCj4gPiByb290ZnMgICAgICAgICAgICAgICAgIDUyRyAg
NSw0RyAgIDQ2RyAgMTElIC8KPgo+IEFscmlnaHR5LCBzbyBidHJmcy1zaG93IHNheXMgeW91IGhh
dmUgdXNlZCB1cCBhbGwgb2YgeW91ciBkaXNrLCBidXQgZm9yCj4gc29tZSByZWFzb24gc3RhdGZz
IGlzIHNheWluZyB5b3UgaGF2ZW50LiAgQ2FuIHlvdSBjZCBpbnRvIC8gYW5kIHJ1bgo+Cj4gZHUg
LWggLXMKPgo+IGFuZCB0aGVuIHdhaXQgYSB2ZXJ5IGxvbmcgdGltZSBhbmQgdGhlbiB0ZWxsIG1l
IHdoYXQgaXQgY29tZXMgYmFjayB3aXRoIDopLgo+IFRoYXQgd2lsbCB0ZWxsIHVzIGZvciBzdXJl
IHdoYXQgdGhlIGhlbGwgaXMgZ29pbmcgb24uICBUaGFuayB5b3UsCj4KPiBKb3NlZgoKNDBHICAg
ICAuCg==

Re: Crash in __btrfs_reserve_extent

[Josef Bacik]

On Thu, Aug 06, 2009 at 04:57:50PM +0200, Robert F=F6rster wrote:
> Hello,=20
>=20
> i thought i'd jump in since i have the same issue:
> btrfs-show output:
>=20
> Label: none  uuid: 7f4f3bcb-7ba1-49c5-8996-e897b5483c7c
>         Total devices 1 FS bytes used 5.32GB
>         devid    1 size 51.22GB used 51.22GB path /dev/sdb1
>=20
> df -h output:
>=20
> Filesystem            Size  Used Avail Use% Mounted on
> rootfs                 52G  5,4G   46G  11% /
>=20

Alrighty, so btrfs-show says you have used up all of your disk, but for=
 some
reason statfs is saying you havent.  Can you cd into / and run

du -h -s

and then wait a very long time and then tell me what it comes back with=
 :).
That will tell us for sure what the hell is going on.  Thank you,

Josef
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crash in __btrfs_reserve_extent

[Robert Förster]

Hello, 

i thought i'd jump in since i have the same issue:
btrfs-show output:

Label: none  uuid: 7f4f3bcb-7ba1-49c5-8996-e897b5483c7c
        Total devices 1 FS bytes used 5.32GB
        devid    1 size 51.22GB used 51.22GB path /dev/sdb1

df -h output:

Filesystem            Size  Used Avail Use% Mounted on
rootfs                 52G  5,4G   46G  11% /

Robert