Kernel panic from "btrfs subvolume delete"

Kernel panic from "btrfs subvolume delete"

[Richard Cooper]

Hi All,

I have two machines where I've been testing various btrfs based backup strategies. They are both Cent OS 6 with the standard kernel and btrfs-progs RPMs from the CentOS repos.

- kernel-2.6.32-220.17.1.el6.x86_64
- btrfs-progs-0.19-12.el6.x86_64

Both are currently in a state when trying to delete a subvolume results in the following kernel panic.

----------

[root@backup2 ~]# btrfs subvolume delete /srv/backup_history/2012-06-28-1342
Delete subvolume '/srv/backup_history/2012-06-28-1342'
[root@backup2 ~]# 
Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:------------[ cut here ]------------

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:invalid opcode: 0000 [#1] SMP 

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:last sysfs file: /sys/devices/virtual/block/md1/md/metadata_version

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:Stack:

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:Call Trace:

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:Code: 89 ef e8 84 f5 fe ff 48 89 df 89 45 d8 e8 99 86 fe ff 8b 45 d8 48 8b 5d e0 4c 8b 65 e8 4c 8b 6d f0 4c 8b 75 f8 c9 c3 0f 0b eb fe <0f> 0b eb fe 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 66 66 66 

Message from syslogd@backup2 at Jun 29 08:53:06 ...
 kernel:Kernel panic - not syncing: Fatal exception

----------

Sometimes the kernel:last sysfs file line says "/sys/kernel/kexec_crash_loaded" instead.

My setup is that /srv is a btrfs sat on /dev/md4 which is a 4 drive software RAID5 array. /srv/backups/data is a subvolume containing 65GB worth of test data. I've "btrfs subvolume snapshot"ed that data to a few new subvolumes under  /srv/backup_history/. Now whenever I try to delete any of the snapshots on either machine I get a kernel panic.

btrfsck look like this:

[root@backup2 ~]# btrfsck /dev/md4
found 72254246912 bytes used err is 0
total csum bytes: 66815432
total tree bytes: 3835244544
total fs tree bytes: 3581440000
btree space waste bytes: 1187313778
file data blocks allocated: 68419002368
 referenced 68418383872
Btrfs Btrfs v0.19


What should I do now? Do I need to upgrade to a more recent btrfs? If so, how? Can I provide any more information to help debug and fix the problem?

Regards,

- Rich

Re: Kernel panic from "btrfs subvolume delete"

[Fajar A. Nugraha]

On Fri, Jun 29, 2012 at 5:11 PM, Richard Cooper
<richard@richardcooper.net> wrote:
> Hi All,
>
> I have two machines where I've been testing various btrfs based backup strategies. They are both Cent OS 6 with the standard kernel and btrfs-progs RPMs from the CentOS repos.
>
> - kernel-2.6.32-220.17.1.el6.x86_64
> - btrfs-progs-0.19-12.el6.x86_64

In btrfs terms, 2.6.32 is ... stone age :P

> What should I do now? Do I need to upgrade to a more recent btrfs?

Yep

> If so, how?

https://blogs.oracle.com/linux/entry/oracle_unbreakable_enterprise_kernel_release
http://elrepo.org/tiki/kernel-ml

-- 
Fajar

Re: Kernel panic from "btrfs subvolume delete"

[Duncan]

Fajar A. Nugraha posted on Fri, 29 Jun 2012 17:42:26 +0700 as excerpted:

> On Fri, Jun 29, 2012 at 5:11 PM, Richard Cooper
> <richard@richardcooper.net> wrote:
>> Hi All,
>>
>> I have two machines where I've been testing various btrfs based backup
>> strategies. They are both Cent OS 6 with the standard kernel and
>> btrfs-progs RPMs from the CentOS repos.
>>
>> - kernel-2.6.32-220.17.1.el6.x86_64 - btrfs-progs-0.19-12.el6.x86_64
> 
> In btrfs terms, 2.6.32 is ... stone age :P

Indeed!  As both the kernel option and the btrfs wiki state, btrfs is an 
experimental filesystem under heavy development and fit for testing, not 
operational use.  Oracle and I believe SuSE have paid support now if you 
want it, but to some extent that's by locking down your options, and 
otherwise, it's simply offering to let you pay them for recovery efforts 
if something does go wrong.

Meanwhile, "under heavy development" in practice means that if you're 
using a kernel older than the last upstream release or two (so 3.3 at the 
very oldest!), you're testing extremely outdated code and the value of 
those tests both in reporting problems and in conclusions you yourself 
may draw from them is extremely limited.

Latest upstream release, now 3.4, is really the oldest you should be 
running for btrfs testing, and many people run the development kernel rcs, 
3.5-rc4 currently, or git-kernels, either Linus or btrfs-next (see the 
wiki).

So 2.6.32...  Do you still run kernel 2.2 on your non-btrfs machines, by 
any chance?  Because that's what's comparable, in terms of btrfs 
development vs kernel development.

>> What should I do now? Do I need to upgrade to a more recent btrfs?
> 
> Yep
> 
>> If so, how?
> 
> https://blogs.oracle.com/linux/entry/
oracle_unbreakable_enterprise_kernel_release
> http://elrepo.org/tiki/kernel-ml


Or read up on the wiki and go mainline kernel:

https://btrfs.wiki.kernel.org/

https://btrfs.wiki.kernel.org/index.php/Main_Page#Documentation

https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Kernel panic from "btrfs subvolume delete"

[Richard Cooper]

On 29 Jun 2012, at 11:42, Fajar A. Nugraha wrote:
>> What should I do now? Do I need to upgrade to a more recent btrfs?
> 
> Yep
> 
>> If so, how?
> 
> https://blogs.oracle.com/linux/entry/oracle_unbreakable_enterprise_kernel_release
> http://elrepo.org/tiki/kernel-ml

Perfect, thank you! I was looking for a mainline kernel yum repo but my google-fu was failing me. That looks like just what I need.

I've installed kernel v3.4.4 from http://elrepo.org/tiki/kernel-ml and that seems to have fixed my kernel panic. I'm still using the default Cent OS 6 versions of the btrfs userspace programs (v0.19). Any reason why that might be a bad idea?

Thanks again,

- Rich

Re: Kernel panic from "btrfs subvolume delete"

[Fajar A. Nugraha]

On Fri, Jun 29, 2012 at 9:23 PM, Richard Cooper
<richard@richardcooper.net> wrote:
>>> If so, how?
>>
>> https://blogs.oracle.com/linux/entry/oracle_unbreakable_enterprise_kernel_release
>> http://elrepo.org/tiki/kernel-ml
>
> Perfect, thank you! I was looking for a mainline kernel yum repo but my google-fu was failing me. That looks like just what I need.
>
> I've installed kernel v3.4.4 from http://elrepo.org/tiki/kernel-ml and that seems to have fixed my kernel panic. I'm still using the default Cent OS 6 versions of the btrfs userspace programs (v0.19). Any reason why that might be a bad idea?

At the very least, newer version of btrfsck has --repair, which you
might need later in the future.
There's also features lke forcing a certain compression (e.g. zlib) on
a file as part of "btrfs filesystem defrag" command.

Just grab updated btrfs-progs (or whatever it's called) from Oracle's repo.

-- 
Fajar

Re: Kernel panic from "btrfs subvolume delete"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

On Fri, Jun 29, 2012 at 03:23:13PM +0100, Richard Cooper wrote:
> 
> On 29 Jun 2012, at 11:42, Fajar A. Nugraha wrote:
> >> What should I do now? Do I need to upgrade to a more recent btrfs?
> > 
> > Yep
> > 
> >> If so, how?
> > 
> > https://blogs.oracle.com/linux/entry/oracle_unbreakable_enterprise_kernel_release
> > http://elrepo.org/tiki/kernel-ml
> 
> Perfect, thank you! I was looking for a mainline kernel yum repo but my google-fu was failing me. That looks like just what I need.
> 
> I've installed kernel v3.4.4 from http://elrepo.org/tiki/kernel-ml and that seems to have fixed my kernel panic. I'm still using the default Cent OS 6 versions of the btrfs userspace programs (v0.19). Any reason why that might be a bad idea?

   You miss out on new features (like scrub and btrfsck). Note that
"0.19" could actually be any version from the last 3 years or so. Most
distributions these days are putting a date in their package names --
anything from 20120328 or so is good.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Charting the inexorable advance of Western syphilisation... ---   

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 190 bytes --]