bcache lockup

bcache lockup

[David Rhodes Clymer]

I was just playing around with bcache, and set up an LV as the backing
store for the cache:

make-bcache -C -b512K /dev/sdb && echo /dev/sdb > /sys/fs/bcache/register
make-bcache -B /dev/mapper/zapazoid-bacachetest && echo
/dev/mapper-bacachetest > /sys/fs/bcache/register

echo $CACHE_UUID > /sys/block/dm-11/bcache/attach

mkfs.ext4 /dev/bcache0
mount /dev/bcache0 /mnt
scp hercules.vistashare.net:/mnt/data/backups/2011-12-10-scylla-incr.tar.bz2*
/mnt/

And there it hangs:

[78411.767785] bcache: Caching dm-11, inserted new UUID
4fb485fd-25b0-4e6a-8a57-cb3889f984be
[78601.490591] EXT4-fs (bcache0): mounted filesystem with ordered data
mode. Opts: (null)
[78962.976149] INFO: task jbd2/bcache0-8:22301 blocked for more than
120 seconds.
[78962.976158] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[78962.976165] jbd2/bcache0-8  D 00000000     0 22301      2 0x00000000
[78962.976176]  f2690e50 00000046 00000000 00000000 00000000 00000000
00000000 31b5d000
[78962.976193]  00000000 c14aa500 f2690ffc c14aa500 f2690e50 c14aa500
c14aa500 00000001
[78962.976208]  00000000 00000000 f2cb60c0 f2cb60c0 00000000 f2cb60c0
c10380ee 00000002
[78962.976221] Call Trace:
[78962.976239]  [<c10380ee>] ? load_balance+0x60/0x58a
[78962.976282]  [<f833ca94>] ?
jbd2_journal_commit_transaction+0x17e/0xf36 [jbd2]
[78962.976295]  [<c100b769>] ? __switch_to+0x6f/0xe2
[78962.976305]  [<c12c00df>] ? __schedule+0x58d/0x5b4
[78962.976316]  [<c105115a>] ? wake_up_bit+0x56/0x56
[78962.976324]  [<c102c141>] ? __wake_up_common+0x34/0x5a
[78962.976332]  [<c104562e>] ? lock_timer_base+0x19/0x34
[78962.976341]  [<c1045a5f>] ? try_to_del_timer_sync+0x5c/0x63
[78962.976364]  [<f8340dec>] ? kjournald2+0x9e/0x1c7 [jbd2]
[78962.976373]  [<c105115a>] ? wake_up_bit+0x56/0x56
[78962.976390]  [<f8340d4e>] ? commit_timeout+0x5/0x5 [jbd2]
[78962.976400]  [<c1050e10>] ? kthread+0x63/0x68
[78962.976409]  [<c1050dad>] ? kthread_worker_fn+0x114/0x114
[78962.976418]  [<c12c6ebe>] ? kernel_thread_helper+0x6/0x10

Is there any reason bcache would not work with LVM in particular, or
did I do something else that was silly?

-davidc

Re: bcache lockup

[Kent Overstreet]

On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
<david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
> Is there any reason bcache would not work with LVM in particular, or
> did I do something else that was silly?

Certainly ought to work, but I'm not sure what the last time I tested
it was. I'll see if I can reproduce it.

Re: bcache lockup

[David Rhodes Clymer]

On Fri, Dec 16, 2011 at 1:56 PM, Kent Overstreet
<kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>> Is there any reason bcache would not work with LVM in particular, or
>> did I do something else that was silly?
>
> Certainly ought to work, but I'm not sure what the last time I tested
> it was. I'll see if I can reproduce it.

Well, just for kicks, I setup bcache with a regular disk as the
backing device. Everything seemed to go ok...I was able to create an
ext4 filesystem on it and mount it. I remembered I had forgotten to
attach the cache, so I attached the cache, and tried copying an ISO to
the mounted fs, and it immediately crashed.

 I was doing this remotely, and now I can't get to the box. So I'm not
certain this is the full traceback, but it's what I got before I lost
connection:

 kernel:[50655.044130] ------------[ cut here ]------------
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] invalid opcode: 0000 [#1] SMP
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] Process kworker/5:0 (pid: 5699, ti=c78c2000
task=f27db5f0 task.ti=c78c2000)
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] Stack:
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] Call Trace:
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] Code: 8b 87 88 03 00 00 85 c0 74 17 8b 00 85 c0
74 11 8b 48 28 85 c9 74 0a 89 f2 89 f8 ff d1 85 c0 75 6b 66 83 be 84
00 00 00 00 75 04 <0f> 0b eb fe 8b 9e 90 00 00 00 85 db 75 1d ba 20 00
00 00 89 f8
Message from syslogd@zapazoid at Dec 17 07:39:06 ...
kernel:[50655.048019] EIP: [<f8c759de>] scsi_setup_fs_cmnd+0x3f/0xa4
[scsi_mod] SS:ESP 0068:c78c3ed0


This box has a Phenom II X6 (1035T??). I'm not exactly certain of the
model number.

-davidc

Re: bcache lockup

[Kent Overstreet]

Argh, it's always something.

That looks like you hit the BUG_ON(!req->nr_phys_segments). I thought
I fixed that.

Are you running the latest code? There should be a commit that says
"bcache: Always initialize bi_max_vecs".

On Sat, Dec 17, 2011 at 5:00 AM, David Rhodes Clymer
<david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
> On Fri, Dec 16, 2011 at 1:56 PM, Kent Overstreet
> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>> Is there any reason bcache would not work with LVM in particular, or
>>> did I do something else that was silly?
>>
>> Certainly ought to work, but I'm not sure what the last time I tested
>> it was. I'll see if I can reproduce it.
>
> Well, just for kicks, I setup bcache with a regular disk as the
> backing device. Everything seemed to go ok...I was able to create an
> ext4 filesystem on it and mount it. I remembered I had forgotten to
> attach the cache, so I attached the cache, and tried copying an ISO to
> the mounted fs, and it immediately crashed.
>
>  I was doing this remotely, and now I can't get to the box. So I'm not
> certain this is the full traceback, but it's what I got before I lost
> connection:
>
>  kernel:[50655.044130] ------------[ cut here ]------------
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] invalid opcode: 0000 [#1] SMP
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] Process kworker/5:0 (pid: 5699, ti=c78c2000
> task=f27db5f0 task.ti=c78c2000)
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] Stack:
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] Call Trace:
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] Code: 8b 87 88 03 00 00 85 c0 74 17 8b 00 85 c0
> 74 11 8b 48 28 85 c9 74 0a 89 f2 89 f8 ff d1 85 c0 75 6b 66 83 be 84
> 00 00 00 00 75 04 <0f> 0b eb fe 8b 9e 90 00 00 00 85 db 75 1d ba 20 00
> 00 00 89 f8
> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
> kernel:[50655.048019] EIP: [<f8c759de>] scsi_setup_fs_cmnd+0x3f/0xa4
> [scsi_mod] SS:ESP 0068:c78c3ed0
>
>
> This box has a Phenom II X6 (1035T??). I'm not exactly certain of the
> model number.
>
> -davidc

Re: bcache lockup

[David Rhodes Clymer]

On Sat, Dec 17, 2011 at 11:46 PM, Kent Overstreet
<kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Argh, it's always something.
>
> That looks like you hit the BUG_ON(!req->nr_phys_segments). I thought
> I fixed that.
>
> Are you running the latest code? There should be a commit that says
> "bcache: Always initialize bi_max_vecs".

Yeah, I'm running the latest. The last commit is:

	ffca635f2efb8e091c469ffcc86d4def2d5c381d: Open block device exclusively

The only 2 commit messages referencing bi_max_vecs that I could find
didn't seem to be bcache specific (they were from 2006).

I have no local modifications.

-davidc.


>
> On Sat, Dec 17, 2011 at 5:00 AM, David Rhodes Clymer
> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>> On Fri, Dec 16, 2011 at 1:56 PM, Kent Overstreet
>> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
>>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>>> Is there any reason bcache would not work with LVM in particular, or
>>>> did I do something else that was silly?
>>>
>>> Certainly ought to work, but I'm not sure what the last time I tested
>>> it was. I'll see if I can reproduce it.
>>
>> Well, just for kicks, I setup bcache with a regular disk as the
>> backing device. Everything seemed to go ok...I was able to create an
>> ext4 filesystem on it and mount it. I remembered I had forgotten to
>> attach the cache, so I attached the cache, and tried copying an ISO to
>> the mounted fs, and it immediately crashed.
>>
>>  I was doing this remotely, and now I can't get to the box. So I'm not
>> certain this is the full traceback, but it's what I got before I lost
>> connection:
>>
>>  kernel:[50655.044130] ------------[ cut here ]------------
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] invalid opcode: 0000 [#1] SMP
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] Process kworker/5:0 (pid: 5699, ti=c78c2000
>> task=f27db5f0 task.ti=c78c2000)
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] Stack:
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] Call Trace:
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] Code: 8b 87 88 03 00 00 85 c0 74 17 8b 00 85 c0
>> 74 11 8b 48 28 85 c9 74 0a 89 f2 89 f8 ff d1 85 c0 75 6b 66 83 be 84
>> 00 00 00 00 75 04 <0f> 0b eb fe 8b 9e 90 00 00 00 85 db 75 1d ba 20 00
>> 00 00 89 f8
>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>> kernel:[50655.048019] EIP: [<f8c759de>] scsi_setup_fs_cmnd+0x3f/0xa4
>> [scsi_mod] SS:ESP 0068:c78c3ed0
>>
>>
>> This box has a Phenom II X6 (1035T??). I'm not exactly certain of the
>> model number.
>>
>> -davidc

Re: bcache lockup

[Kent Overstreet]

Appears I didn't push the fix to the public tree...

Give me a day or so, I'll need to merge the latest fixes into the 3.1
tree, test it and make sure I haven't missed anything else...

On Sun, Dec 18, 2011 at 5:22 PM, David Rhodes Clymer
<david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
> On Sat, Dec 17, 2011 at 11:46 PM, Kent Overstreet
> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Argh, it's always something.
>>
>> That looks like you hit the BUG_ON(!req->nr_phys_segments). I thought
>> I fixed that.
>>
>> Are you running the latest code? There should be a commit that says
>> "bcache: Always initialize bi_max_vecs".
>
> Yeah, I'm running the latest. The last commit is:
>
>        ffca635f2efb8e091c469ffcc86d4def2d5c381d: Open block device exclusively
>
> The only 2 commit messages referencing bi_max_vecs that I could find
> didn't seem to be bcache specific (they were from 2006).
>
> I have no local modifications.
>
> -davidc.
>
>
>>
>> On Sat, Dec 17, 2011 at 5:00 AM, David Rhodes Clymer
>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>> On Fri, Dec 16, 2011 at 1:56 PM, Kent Overstreet
>>> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
>>>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>>>> Is there any reason bcache would not work with LVM in particular, or
>>>>> did I do something else that was silly?
>>>>
>>>> Certainly ought to work, but I'm not sure what the last time I tested
>>>> it was. I'll see if I can reproduce it.
>>>
>>> Well, just for kicks, I setup bcache with a regular disk as the
>>> backing device. Everything seemed to go ok...I was able to create an
>>> ext4 filesystem on it and mount it. I remembered I had forgotten to
>>> attach the cache, so I attached the cache, and tried copying an ISO to
>>> the mounted fs, and it immediately crashed.
>>>
>>>  I was doing this remotely, and now I can't get to the box. So I'm not
>>> certain this is the full traceback, but it's what I got before I lost
>>> connection:
>>>
>>>  kernel:[50655.044130] ------------[ cut here ]------------
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] invalid opcode: 0000 [#1] SMP
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] Process kworker/5:0 (pid: 5699, ti=c78c2000
>>> task=f27db5f0 task.ti=c78c2000)
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] Stack:
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] Call Trace:
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] Code: 8b 87 88 03 00 00 85 c0 74 17 8b 00 85 c0
>>> 74 11 8b 48 28 85 c9 74 0a 89 f2 89 f8 ff d1 85 c0 75 6b 66 83 be 84
>>> 00 00 00 00 75 04 <0f> 0b eb fe 8b 9e 90 00 00 00 85 db 75 1d ba 20 00
>>> 00 00 89 f8
>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>> kernel:[50655.048019] EIP: [<f8c759de>] scsi_setup_fs_cmnd+0x3f/0xa4
>>> [scsi_mod] SS:ESP 0068:c78c3ed0
>>>
>>>
>>> This box has a Phenom II X6 (1035T??). I'm not exactly certain of the
>>> model number.
>>>
>>> -davidc

Re: bcache lockup

[David Rhodes Clymer]

On Sun, Dec 18, 2011 at 11:01 PM, Kent Overstreet
<kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Appears I didn't push the fix to the public tree...
>
> Give me a day or so, I'll need to merge the latest fixes into the 3.1
> tree, test it and make sure I haven't missed anything else...


Looks like commit 7c3e597ca0f5c87767548d1e9aced024aa558b2a
resolved this issue for me.

-davidc


>
> On Sun, Dec 18, 2011 at 5:22 PM, David Rhodes Clymer
> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>> On Sat, Dec 17, 2011 at 11:46 PM, Kent Overstreet
>> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> Argh, it's always something.
>>>
>>> That looks like you hit the BUG_ON(!req->nr_phys_segments). I thought
>>> I fixed that.
>>>
>>> Are you running the latest code? There should be a commit that says
>>> "bcache: Always initialize bi_max_vecs".
>>
>> Yeah, I'm running the latest. The last commit is:
>>
>>        ffca635f2efb8e091c469ffcc86d4def2d5c381d: Open block device exclusively
>>
>> The only 2 commit messages referencing bi_max_vecs that I could find
>> didn't seem to be bcache specific (they were from 2006).
>>
>> I have no local modifications.
>>
>> -davidc.
>>
>>
>>>
>>> On Sat, Dec 17, 2011 at 5:00 AM, David Rhodes Clymer
>>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>>> On Fri, Dec 16, 2011 at 1:56 PM, Kent Overstreet
>>>> <kent.overstreet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> On Fri, Dec 16, 2011 at 9:41 AM, David Rhodes Clymer
>>>>> <david-LEGOU7lN1jbY0TyS/+Ba5Q@public.gmane.org> wrote:
>>>>>> Is there any reason bcache would not work with LVM in particular, or
>>>>>> did I do something else that was silly?
>>>>>
>>>>> Certainly ought to work, but I'm not sure what the last time I tested
>>>>> it was. I'll see if I can reproduce it.
>>>>
>>>> Well, just for kicks, I setup bcache with a regular disk as the
>>>> backing device. Everything seemed to go ok...I was able to create an
>>>> ext4 filesystem on it and mount it. I remembered I had forgotten to
>>>> attach the cache, so I attached the cache, and tried copying an ISO to
>>>> the mounted fs, and it immediately crashed.
>>>>
>>>>  I was doing this remotely, and now I can't get to the box. So I'm not
>>>> certain this is the full traceback, but it's what I got before I lost
>>>> connection:
>>>>
>>>>  kernel:[50655.044130] ------------[ cut here ]------------
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] invalid opcode: 0000 [#1] SMP
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] Process kworker/5:0 (pid: 5699, ti=c78c2000
>>>> task=f27db5f0 task.ti=c78c2000)
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] Stack:
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] Call Trace:
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] Code: 8b 87 88 03 00 00 85 c0 74 17 8b 00 85 c0
>>>> 74 11 8b 48 28 85 c9 74 0a 89 f2 89 f8 ff d1 85 c0 75 6b 66 83 be 84
>>>> 00 00 00 00 75 04 <0f> 0b eb fe 8b 9e 90 00 00 00 85 db 75 1d ba 20 00
>>>> 00 00 89 f8
>>>> Message from syslogd@zapazoid at Dec 17 07:39:06 ...
>>>> kernel:[50655.048019] EIP: [<f8c759de>] scsi_setup_fs_cmnd+0x3f/0xa4
>>>> [scsi_mod] SS:ESP 0068:c78c3ed0
>>>>
>>>>
>>>> This box has a Phenom II X6 (1035T??). I'm not exactly certain of the
>>>> model number.
>>>>
>>>> -davidc