* flush-btrfs-1 hangs when building openwrt
@ 2011-01-25 8:51 Daniel Poelzleithner
2011-01-25 15:30 ` Josef Bacik
2011-01-31 10:52 ` Sander
0 siblings, 2 replies; 8+ messages in thread
From: Daniel Poelzleithner @ 2011-01-25 8:51 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 862 bytes --]
Hi,
Since update to 2.6.37 I can't build openwrt on my btrfs buildroot anymore.
I'm not sure if this is related to the other flush-btrfs-1 thread.
plenty of diskspace is free:
/dev/mapper/cruor-build
97G 68G 27G 73% /opt/build
It always hangs when openwrt builds the ext4 image and runs tune2fs on it.
/opt/build/fahrenheit/openwrt/staging_dir/host/bin/tune2fs -O
extents,uninit_bg,dir_index
/opt/build/fahrenheit/openwrt/build_dir/linux-x86_kvm_guest/root.ext4
tune2fs 1.41.13 (13-Dec-2010)
the processes can't be killed.
alt-sysctl-t does not show anything, nor is there a oops.
I put the the openwrt config I'm using at https://gist.github.com/794593
, maybe it is reproduceable.
Linux cruor 2.6.37 #2 SMP Thu Jan 20 02:09:59 CET 2011 x86_64 GNU/Linux
Please CC.
kind regards
Daniel
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-25 8:51 flush-btrfs-1 hangs when building openwrt Daniel Poelzleithner
@ 2011-01-25 15:30 ` Josef Bacik
2011-01-25 21:16 ` Daniel Poelzleithner
2011-01-31 10:52 ` Sander
1 sibling, 1 reply; 8+ messages in thread
From: Josef Bacik @ 2011-01-25 15:30 UTC (permalink / raw)
To: Daniel Poelzleithner; +Cc: linux-btrfs
On Tue, Jan 25, 2011 at 09:51:01AM +0100, Daniel Poelzleithner wrote:
> Hi,
>
> Since update to 2.6.37 I can't build openwrt on my btrfs buildroot anymore.
> I'm not sure if this is related to the other flush-btrfs-1 thread.
>
> plenty of diskspace is free:
>
> /dev/mapper/cruor-build
> 97G 68G 27G 73% /opt/build
>
> It always hangs when openwrt builds the ext4 image and runs tune2fs on it.
>
> /opt/build/fahrenheit/openwrt/staging_dir/host/bin/tune2fs -O
> extents,uninit_bg,dir_index
> /opt/build/fahrenheit/openwrt/build_dir/linux-x86_kvm_guest/root.ext4
> tune2fs 1.41.13 (13-Dec-2010)
>
> the processes can't be killed.
>
> alt-sysctl-t does not show anything, nor is there a oops.
>
> I put the the openwrt config I'm using at https://gist.github.com/794593
> , maybe it is reproduceable.
>
> Linux cruor 2.6.37 #2 SMP Thu Jan 20 02:09:59 CET 2011 x86_64 GNU/Linux
>
How about sysrq+w when it's hanging. Also could you give the exact steps to
reproduce? I went to the openwrt site to try and build, but it seems like
theres alot of moving parts. If you can just tell me what to download and what
you run to reproduce I can try and reproduce locally. Thanks,
Josef
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-25 15:30 ` Josef Bacik
@ 2011-01-25 21:16 ` Daniel Poelzleithner
0 siblings, 0 replies; 8+ messages in thread
From: Daniel Poelzleithner @ 2011-01-25 21:16 UTC (permalink / raw)
To: Josef Bacik; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 913 bytes --]
On 01/25/2011 04:30 PM, Josef Bacik wrote:
> How about sysrq+w when it's hanging.
Shows nothing.
> Also could you give the exact steps to
> reproduce? I went to the openwrt site to try and build, but it seems like
> theres alot of moving parts. If you can just tell me what to download and what
> you run to reproduce I can try and reproduce locally. Thanks,
mkdir /btrfs/with/lots/of/space
cd /btrfs/with/lots/of/space
git clone git://nbd.name/openwrt.git
cd openwrt
wget
https://gist.github.com/raw/794593/b9e7e7b6dce71093a653953d7e39c94a6ffa4528/gistfile1.txt
-O .config
make
# take a nap, will take quite some time on the first run
If you prefere seeing output, which slows things down do
make V=99
instead. I skipped the packages repo here, because I don't think it will
make a difference. It hanges in the final steps of creating the image.
kind regards
Daniel
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-25 8:51 flush-btrfs-1 hangs when building openwrt Daniel Poelzleithner
2011-01-25 15:30 ` Josef Bacik
@ 2011-01-31 10:52 ` Sander
2011-01-31 11:08 ` cwillu
1 sibling, 1 reply; 8+ messages in thread
From: Sander @ 2011-01-31 10:52 UTC (permalink / raw)
To: Daniel Poelzleithner; +Cc: linux-btrfs
Daniel Poelzleithner wrote (ao):
> Since update to 2.6.37 I can't build openwrt on my btrfs buildroot anymore.
> I'm not sure if this is related to the other flush-btrfs-1 thread.
While I thought it was related to a dying disk used for backups, after
your post I think it might not.
Running 2.6.37 on openrd-client (ARM).
It started with hanging jobs on the backup disk. I stopped cron and
could kill most of the jobs. Some are still hanging though.
Since then (uptime 12 days) I see hanging procmail processes, and an
apt-get upgrade last week gave an unkillable dpkg process. All these have
nothing to do with the backup disk. CPU is maxed out:
top - 11:49:54 up 12 days, 1:19, 31 users, load average: 13.54, 13.41, 13.36
Tasks: 201 total, 13 running, 187 sleeping, 0 stopped, 1 zombie
Cpu(s): 41.5%us, 58.5%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 515004k total, 400824k used, 114180k free, 28k buffers
Swap: 4302560k total, 173988k used, 4128572k free, 202948k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1592 ookhoi 20 0 2716 456 348 S 1.9 0.1 25:17.42 showNewMail2
6761 ookhoi 20 0 2736 1000 704 S 1.3 0.2 61:21.93 top
27609 ookhoi 20 0 2736 1264 936 R 1.3 0.2 0:01.06 top
30678 ookhoi 20 0 2736 892 584 S 1.3 0.2 91:37.75 top
6036 ookhoi 39 19 2692 64 52 R 1.0 0.0 869:46.32 procmail
11373 ookhoi 39 19 4800 64 52 R 1.0 0.0 714:25.88 procmail
18871 root 39 19 2540 32 20 R 1.0 0.0 1528:51 lzop
18894 ookhoi 39 19 2692 64 52 R 1.0 0.0 611:16.18 procmail
20305 ookhoi 39 19 2692 68 56 R 1.0 0.0 610:51.97 procmail
20378 ookhoi 39 19 2692 68 56 R 1.0 0.0 610:50.75 procmail
23661 ookhoi 39 19 2692 80 68 R 1.0 0.0 1308:23 procmail
25091 root 20 0 0 0 0 S 1.0 0.0 0:25.63 flush-btrfs-2
26409 root 39 19 2264 32 28 R 1.0 0.0 1526:42 mv
27606 ookhoi 39 19 9084 40 28 R 1.0 0.0 3637:39 procmail
27910 root 39 19 15096 3756 304 R 1.0 0.7 638:46.62 dpkg
11804 ookhoi 39 19 4700 64 52 R 0.6 0.0 714:08.67 procmail
3 root 20 0 0 0 0 R 0.3 0.0 9:39.76 ksoftirqd/0
What can I do to provide more info?
Sander
--
Humilis IT Services and Solutions
http://www.humilis.net
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-31 10:52 ` Sander
@ 2011-01-31 11:08 ` cwillu
2011-01-31 11:18 ` Sander
0 siblings, 1 reply; 8+ messages in thread
From: cwillu @ 2011-01-31 11:08 UTC (permalink / raw)
To: sander; +Cc: Daniel Poelzleithner, linux-btrfs
On Mon, Jan 31, 2011 at 4:52 AM, Sander <sander@humilis.net> wrote:
> Daniel Poelzleithner wrote (ao):
>> Since update to 2.6.37 I can't build openwrt on my btrfs buildroot a=
nymore.
>> I'm not sure if this is related to the other flush-btrfs-1 thread.
>
> While I thought it was related to a dying disk used for backups, afte=
r
> your post I think it might not.
>
> Running 2.6.37 on openrd-client (ARM).
>
> It started with hanging jobs on the backup disk. I stopped cron and
> could kill most of the jobs. Some are still hanging though.
>
> Since then (uptime 12 days) I see hanging procmail processes, and an
> apt-get upgrade last week gave an unkillable dpkg process. All these =
have
> nothing to do with the backup disk. CPU is maxed out:
>
> top - 11:49:54 up 12 days, =A01:19, 31 users, =A0load average: 13.54,=
13.41, 13.36
> Tasks: 201 total, =A013 running, 187 sleeping, =A0 0 stopped, =A0 1 z=
ombie
> Cpu(s): 41.5%us, 58.5%sy, =A00.0%ni, =A00.0%id, =A00.0%wa, =A00.0%hi,=
=A00.0%si, =A00.0%st
> Mem: =A0 =A0515004k total, =A0 400824k used, =A0 114180k free, =A0 =A0=
=A0 28k buffers
> Swap: =A04302560k total, =A0 173988k used, =A04128572k free, =A0 2029=
48k cached
>
> =A0PID USER =A0 =A0 =A0PR =A0NI =A0VIRT =A0RES =A0SHR S %CPU %MEM =A0=
=A0TIME+ =A0COMMAND
> =A01592 ookhoi =A0 =A020 =A0 0 =A02716 =A0456 =A0348 S =A01.9 =A00.1 =
=A025:17.42 showNewMail2
> =A06761 ookhoi =A0 =A020 =A0 0 =A02736 1000 =A0704 S =A01.3 =A00.2 =A0=
61:21.93 top
> 27609 ookhoi =A0 =A020 =A0 0 =A02736 1264 =A0936 R =A01.3 =A00.2 =A0 =
0:01.06 top
> 30678 ookhoi =A0 =A020 =A0 0 =A02736 =A0892 =A0584 S =A01.3 =A00.2 =A0=
91:37.75 top
> =A06036 ookhoi =A0 =A039 =A019 =A02692 =A0 64 =A0 52 R =A01.0 =A00.0 =
869:46.32 procmail
> 11373 ookhoi =A0 =A039 =A019 =A04800 =A0 64 =A0 52 R =A01.0 =A00.0 71=
4:25.88 procmail
> 18871 root =A0 =A0 =A039 =A019 =A02540 =A0 32 =A0 20 R =A01.0 =A00.0 =
=A0 1528:51 lzop
> 18894 ookhoi =A0 =A039 =A019 =A02692 =A0 64 =A0 52 R =A01.0 =A00.0 61=
1:16.18 procmail
> 20305 ookhoi =A0 =A039 =A019 =A02692 =A0 68 =A0 56 R =A01.0 =A00.0 61=
0:51.97 procmail
> 20378 ookhoi =A0 =A039 =A019 =A02692 =A0 68 =A0 56 R =A01.0 =A00.0 61=
0:50.75 procmail
> 23661 ookhoi =A0 =A039 =A019 =A02692 =A0 80 =A0 68 R =A01.0 =A00.0 =A0=
1308:23 procmail
> 25091 root =A0 =A0 =A020 =A0 0 =A0 =A0 0 =A0 =A00 =A0 =A00 S =A01.0 =A0=
0.0 =A0 0:25.63 flush-btrfs-2
> 26409 root =A0 =A0 =A039 =A019 =A02264 =A0 32 =A0 28 R =A01.0 =A00.0 =
=A0 1526:42 mv
> 27606 ookhoi =A0 =A039 =A019 =A09084 =A0 40 =A0 28 R =A01.0 =A00.0 =A0=
3637:39 procmail
> 27910 root =A0 =A0 =A039 =A019 15096 3756 =A0304 R =A01.0 =A00.7 638:=
46.62 dpkg
> 11804 ookhoi =A0 =A039 =A019 =A04700 =A0 64 =A0 52 R =A00.6 =A00.0 71=
4:08.67 procmail
> =A0 =A03 root =A0 =A0 =A020 =A0 0 =A0 =A0 0 =A0 =A00 =A0 =A00 R =A00.=
3 =A00.0 =A0 9:39.76 ksoftirqd/0
>
>
> What can I do to provide more info?
alt-sysrq-w, and then the dmesg output, which will contain then a
backtrace for every blocked process.
(Be careful of typos, there sysrq keystrokes to do other less
diagnostic tasks that you won't want to hit by accident:
http://en.wikipedia.org/wiki/Magic_SysRq_key).
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-31 11:08 ` cwillu
@ 2011-01-31 11:18 ` Sander
2011-01-31 11:26 ` cwillu
0 siblings, 1 reply; 8+ messages in thread
From: Sander @ 2011-01-31 11:18 UTC (permalink / raw)
To: cwillu; +Cc: sander, Daniel Poelzleithner, linux-btrfs
cwillu wrote (ao):
> On Mon, Jan 31, 2011 at 4:52 AM, Sander <sander@humilis.net> wrote:
> > Daniel Poelzleithner wrote (ao):
> >> Since update to 2.6.37 I can't build openwrt on my btrfs buildroot anymore.
> >> I'm not sure if this is related to the other flush-btrfs-1 thread.
> >
> > While I thought it was related to a dying disk used for backups, after
> > your post I think it might not.
> >
> > Running 2.6.37 on openrd-client (ARM).
> >
> > It started with hanging jobs on the backup disk. I stopped cron and
> > could kill most of the jobs. Some are still hanging though.
> >
> > Since then (uptime 12 days) I see hanging procmail processes, and an
> > apt-get upgrade last week gave an unkillable dpkg process. All these have
> > nothing to do with the backup disk. CPU is maxed out:
> >
> > top - 11:49:54 up 12 days, ?1:19, 31 users, ?load average: 13.54, 13.41, 13.36
> > Tasks: 201 total, ?13 running, 187 sleeping, ? 0 stopped, ? 1 zombie
> > Cpu(s): 41.5%us, 58.5%sy, ?0.0%ni, ?0.0%id, ?0.0%wa, ?0.0%hi, ?0.0%si, ?0.0%st
> > Mem: ? ?515004k total, ? 400824k used, ? 114180k free, ? ? ? 28k buffers
> > Swap: ?4302560k total, ? 173988k used, ?4128572k free, ? 202948k cached
> >
> > ?PID USER ? ? ?PR ?NI ?VIRT ?RES ?SHR S %CPU %MEM ? ?TIME+ ?COMMAND
> > ?1592 ookhoi ? ?20 ? 0 ?2716 ?456 ?348 S ?1.9 ?0.1 ?25:17.42 showNewMail2
> > ?6761 ookhoi ? ?20 ? 0 ?2736 1000 ?704 S ?1.3 ?0.2 ?61:21.93 top
> > 27609 ookhoi ? ?20 ? 0 ?2736 1264 ?936 R ?1.3 ?0.2 ? 0:01.06 top
> > 30678 ookhoi ? ?20 ? 0 ?2736 ?892 ?584 S ?1.3 ?0.2 ?91:37.75 top
> > ?6036 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 869:46.32 procmail
> > 11373 ookhoi ? ?39 ?19 ?4800 ? 64 ? 52 R ?1.0 ?0.0 714:25.88 procmail
> > 18871 root ? ? ?39 ?19 ?2540 ? 32 ? 20 R ?1.0 ?0.0 ? 1528:51 lzop
> > 18894 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 611:16.18 procmail
> > 20305 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:51.97 procmail
> > 20378 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:50.75 procmail
> > 23661 ookhoi ? ?39 ?19 ?2692 ? 80 ? 68 R ?1.0 ?0.0 ? 1308:23 procmail
> > 25091 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 S ?1.0 ?0.0 ? 0:25.63 flush-btrfs-2
> > 26409 root ? ? ?39 ?19 ?2264 ? 32 ? 28 R ?1.0 ?0.0 ? 1526:42 mv
> > 27606 ookhoi ? ?39 ?19 ?9084 ? 40 ? 28 R ?1.0 ?0.0 ? 3637:39 procmail
> > 27910 root ? ? ?39 ?19 15096 3756 ?304 R ?1.0 ?0.7 638:46.62 dpkg
> > 11804 ookhoi ? ?39 ?19 ?4700 ? 64 ? 52 R ?0.6 ?0.0 714:08.67 procmail
> > ? ?3 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 R ?0.3 ?0.0 ? 9:39.76 ksoftirqd/0
> >
> >
> > What can I do to provide more info?
>
> alt-sysrq-w, and then the dmesg output, which will contain then a
> backtrace for every blocked process.
Thanks cwillu.
Seems only two processes. And these are related to the backup disk
(which might or might not be broken: can't access it anymore).
Nothing to do with the procmail and dpkg processes.
[1042949.513831] SysRq : Show Blocked State
[1042949.517776] task PC stack pid father
[1042949.523247] cat D c0475dd0 0 30063 1 0x00000001
[1042949.529668] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88)
[1042949.538943] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from [<c01af0e8>] (do_lookup+0x90/0x128)
[1042949.548209] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>] (do_last+0x198/0x5b8)
[1042949.556432] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>] (do_filp_open+0x168/0x49c)
[1042949.565004] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a555c>] (do_sys_open+0x58/0x11c)
[1042949.573838] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee0>] (ret_fast_syscall+0x0/0x2c)
[1042949.582750] cat D c0475dd0 0 4591 1 0x00000001
[1042949.589152] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88)
[1042949.598418] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from [<c01af0e8>] (do_lookup+0x90/0x128)
[1042949.607687] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>] (do_last+0x198/0x5b8)
[1042949.615910] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>] (do_filp_open+0x168/0x49c)
[1042949.624482] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a555c>] (do_sys_open+0x58/0x11c)
[1042949.633315] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee0>] (ret_fast_syscall+0x0/0x2c)
--
Humilis IT Services and Solutions
http://www.humilis.net
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-31 11:18 ` Sander
@ 2011-01-31 11:26 ` cwillu
2011-01-31 11:40 ` Sander
0 siblings, 1 reply; 8+ messages in thread
From: cwillu @ 2011-01-31 11:26 UTC (permalink / raw)
To: sander; +Cc: Daniel Poelzleithner, linux-btrfs
On Mon, Jan 31, 2011 at 5:18 AM, Sander <sander@humilis.net> wrote:
> cwillu wrote (ao):
>> On Mon, Jan 31, 2011 at 4:52 AM, Sander <sander@humilis.net> wrote:
>> > Daniel Poelzleithner wrote (ao):
>> >> Since update to 2.6.37 I can't build openwrt on my btrfs buildroo=
t anymore.
>> >> I'm not sure if this is related to the other flush-btrfs-1 thread=
=2E
>> >
>> > While I thought it was related to a dying disk used for backups, a=
fter
>> > your post I think it might not.
>> >
>> > Running 2.6.37 on openrd-client (ARM).
>> >
>> > It started with hanging jobs on the backup disk. I stopped cron an=
d
>> > could kill most of the jobs. Some are still hanging though.
>> >
>> > Since then (uptime 12 days) I see hanging procmail processes, and =
an
>> > apt-get upgrade last week gave an unkillable dpkg process. All the=
se have
>> > nothing to do with the backup disk. CPU is maxed out:
>> >
>> > top - 11:49:54 up 12 days, ?1:19, 31 users, ?load average: 13.54, =
13.41, 13.36
>> > Tasks: 201 total, ?13 running, 187 sleeping, ? 0 stopped, ? 1 zomb=
ie
>> > Cpu(s): 41.5%us, 58.5%sy, ?0.0%ni, ?0.0%id, ?0.0%wa, ?0.0%hi, ?0.0=
%si, ?0.0%st
>> > Mem: ? ?515004k total, ? 400824k used, ? 114180k free, ? ? ? 28k b=
uffers
>> > Swap: ?4302560k total, ? 173988k used, ?4128572k free, ? 202948k c=
ached
>> >
>> > ?PID USER ? ? ?PR ?NI ?VIRT ?RES ?SHR S %CPU %MEM ? ?TIME+ ?COMMAN=
D
>> > ?1592 ookhoi ? ?20 ? 0 ?2716 ?456 ?348 S ?1.9 ?0.1 ?25:17.42 showN=
ewMail2
>> > ?6761 ookhoi ? ?20 ? 0 ?2736 1000 ?704 S ?1.3 ?0.2 ?61:21.93 top
>> > 27609 ookhoi ? ?20 ? 0 ?2736 1264 ?936 R ?1.3 ?0.2 ? 0:01.06 top
>> > 30678 ookhoi ? ?20 ? 0 ?2736 ?892 ?584 S ?1.3 ?0.2 ?91:37.75 top
>> > ?6036 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 869:46.32 procm=
ail
>> > 11373 ookhoi ? ?39 ?19 ?4800 ? 64 ? 52 R ?1.0 ?0.0 714:25.88 procm=
ail
>> > 18871 root ? ? ?39 ?19 ?2540 ? 32 ? 20 R ?1.0 ?0.0 ? 1528:51 lzop
>> > 18894 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 611:16.18 procm=
ail
>> > 20305 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:51.97 procm=
ail
>> > 20378 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:50.75 procm=
ail
>> > 23661 ookhoi ? ?39 ?19 ?2692 ? 80 ? 68 R ?1.0 ?0.0 ? 1308:23 procm=
ail
>> > 25091 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 S ?1.0 ?0.0 ? 0:25.63 flush=
-btrfs-2
>> > 26409 root ? ? ?39 ?19 ?2264 ? 32 ? 28 R ?1.0 ?0.0 ? 1526:42 mv
>> > 27606 ookhoi ? ?39 ?19 ?9084 ? 40 ? 28 R ?1.0 ?0.0 ? 3637:39 procm=
ail
>> > 27910 root ? ? ?39 ?19 15096 3756 ?304 R ?1.0 ?0.7 638:46.62 dpkg
>> > 11804 ookhoi ? ?39 ?19 ?4700 ? 64 ? 52 R ?0.6 ?0.0 714:08.67 procm=
ail
>> > ? ?3 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 R ?0.3 ?0.0 ? 9:39.76 ksofti=
rqd/0
>> >
>> >
>> > What can I do to provide more info?
>>
>> alt-sysrq-w, and then the dmesg output, which will contain then a
>> backtrace for every blocked process.
>
> Thanks cwillu.
>
> Seems only two processes. And these are related to the backup disk
> (which might or might not be broken: can't access it anymore).
>
> Nothing to do with the procmail and dpkg processes.
>
>
> [1042949.513831] SysRq : Show Blocked State
> [1042949.517776] =A0 task =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0PC stack =A0=
pid father
> [1042949.523247] cat =A0 =A0 =A0 =A0 =A0 D c0475dd0 =A0 =A0 0 30063 =A0=
=A0 =A01 0x00000001
> [1042949.529668] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>=
] (__mutex_lock_slowpath+0x64/0x88)
> [1042949.538943] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from =
[<c01af0e8>] (do_lookup+0x90/0x128)
> [1042949.548209] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>=
] (do_last+0x198/0x5b8)
> [1042949.556432] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>]=
(do_filp_open+0x168/0x49c)
> [1042949.565004] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a5=
55c>] (do_sys_open+0x58/0x11c)
> [1042949.573838] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee=
0>] (ret_fast_syscall+0x0/0x2c)
> [1042949.582750] cat =A0 =A0 =A0 =A0 =A0 D c0475dd0 =A0 =A0 0 =A04591=
=A0 =A0 =A01 0x00000001
> [1042949.589152] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>=
] (__mutex_lock_slowpath+0x64/0x88)
> [1042949.598418] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from =
[<c01af0e8>] (do_lookup+0x90/0x128)
> [1042949.607687] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>=
] (do_last+0x198/0x5b8)
> [1042949.615910] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>]=
(do_filp_open+0x168/0x49c)
> [1042949.624482] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a5=
55c>] (do_sys_open+0x58/0x11c)
> [1042949.633315] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee=
0>] (ret_fast_syscall+0x0/0x2c)
dpkg and procmail were just showing up for you in top because it was
sorting by memory usage, which isn't what we were looking for here.
In your case, the blocking is almost certainly due to your failing
disk.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: flush-btrfs-1 hangs when building openwrt
2011-01-31 11:26 ` cwillu
@ 2011-01-31 11:40 ` Sander
0 siblings, 0 replies; 8+ messages in thread
From: Sander @ 2011-01-31 11:40 UTC (permalink / raw)
To: cwillu; +Cc: sander, Daniel Poelzleithner, linux-btrfs
cwillu wrote (ao):
> On Mon, Jan 31, 2011 at 5:18 AM, Sander <sander@humilis.net> wrote:
> > cwillu wrote (ao):
> >> On Mon, Jan 31, 2011 at 4:52 AM, Sander <sander@humilis.net> wrote:
> >> > It started with hanging jobs on the backup disk. I stopped cron and
> >> > could kill most of the jobs. Some are still hanging though.
> >> >
> >> > Since then (uptime 12 days) I see hanging procmail processes, and an
> >> > apt-get upgrade last week gave an unkillable dpkg process. All these have
> >> > nothing to do with the backup disk. CPU is maxed out:
> >> >
> >> > top - 11:49:54 up 12 days, ?1:19, 31 users, ?load average: 13.54, 13.41, 13.36
> >> > Tasks: 201 total, ?13 running, 187 sleeping, ? 0 stopped, ? 1 zombie
> >> > Cpu(s): 41.5%us, 58.5%sy, ?0.0%ni, ?0.0%id, ?0.0%wa, ?0.0%hi, ?0.0%si, ?0.0%st
> >> > Mem: ? ?515004k total, ? 400824k used, ? 114180k free, ? ? ? 28k buffers
> >> > Swap: ?4302560k total, ? 173988k used, ?4128572k free, ? 202948k cached
> >> >
> >> > ?PID USER ? ? ?PR ?NI ?VIRT ?RES ?SHR S %CPU %MEM ? ?TIME+ ?COMMAND
> >> > ?1592 ookhoi ? ?20 ? 0 ?2716 ?456 ?348 S ?1.9 ?0.1 ?25:17.42 showNewMail2
> >> > ?6761 ookhoi ? ?20 ? 0 ?2736 1000 ?704 S ?1.3 ?0.2 ?61:21.93 top
> >> > 27609 ookhoi ? ?20 ? 0 ?2736 1264 ?936 R ?1.3 ?0.2 ? 0:01.06 top
> >> > 30678 ookhoi ? ?20 ? 0 ?2736 ?892 ?584 S ?1.3 ?0.2 ?91:37.75 top
> >> > ?6036 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 869:46.32 procmail
> >> > 11373 ookhoi ? ?39 ?19 ?4800 ? 64 ? 52 R ?1.0 ?0.0 714:25.88 procmail
> >> > 18871 root ? ? ?39 ?19 ?2540 ? 32 ? 20 R ?1.0 ?0.0 ? 1528:51 lzop
> >> > 18894 ookhoi ? ?39 ?19 ?2692 ? 64 ? 52 R ?1.0 ?0.0 611:16.18 procmail
> >> > 20305 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:51.97 procmail
> >> > 20378 ookhoi ? ?39 ?19 ?2692 ? 68 ? 56 R ?1.0 ?0.0 610:50.75 procmail
> >> > 23661 ookhoi ? ?39 ?19 ?2692 ? 80 ? 68 R ?1.0 ?0.0 ? 1308:23 procmail
> >> > 25091 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 S ?1.0 ?0.0 ? 0:25.63 flush-btrfs-2
> >> > 26409 root ? ? ?39 ?19 ?2264 ? 32 ? 28 R ?1.0 ?0.0 ? 1526:42 mv
> >> > 27606 ookhoi ? ?39 ?19 ?9084 ? 40 ? 28 R ?1.0 ?0.0 ? 3637:39 procmail
> >> > 27910 root ? ? ?39 ?19 15096 3756 ?304 R ?1.0 ?0.7 638:46.62 dpkg
> >> > 11804 ookhoi ? ?39 ?19 ?4700 ? 64 ? 52 R ?0.6 ?0.0 714:08.67 procmail
> >> > ? ?3 root ? ? ?20 ? 0 ? ? 0 ? ?0 ? ?0 R ?0.3 ?0.0 ? 9:39.76 ksoftirqd/0
> >> >
> >> >
> >> > What can I do to provide more info?
> >>
> >> alt-sysrq-w, and then the dmesg output, which will contain then a
> >> backtrace for every blocked process.
> >
> > Thanks cwillu.
> >
> > Seems only two processes. And these are related to the backup disk
> > (which might or might not be broken: can't access it anymore).
> >
> > Nothing to do with the procmail and dpkg processes.
> >
> >
> > [1042949.513831] SysRq : Show Blocked State
> > [1042949.517776] ? task ? ? ? ? ? ? ? ?PC stack ? pid father
> > [1042949.523247] cat ? ? ? ? ? D c0475dd0 ? ? 0 30063 ? ? ?1 0x00000001
> > [1042949.529668] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88)
> > [1042949.538943] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from [<c01af0e8>] (do_lookup+0x90/0x128)
> > [1042949.548209] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>] (do_last+0x198/0x5b8)
> > [1042949.556432] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>] (do_filp_open+0x168/0x49c)
> > [1042949.565004] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a555c>] (do_sys_open+0x58/0x11c)
> > [1042949.573838] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee0>] (ret_fast_syscall+0x0/0x2c)
> > [1042949.582750] cat ? ? ? ? ? D c0475dd0 ? ? 0 ?4591 ? ? ?1 0x00000001
> > [1042949.589152] [<c0475dd0>] (schedule+0x344/0x398) from [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88)
> > [1042949.598418] [<c04764ec>] (__mutex_lock_slowpath+0x64/0x88) from [<c01af0e8>] (do_lookup+0x90/0x128)
> > [1042949.607687] [<c01af0e8>] (do_lookup+0x90/0x128) from [<c01b03f4>] (do_last+0x198/0x5b8)
> > [1042949.615910] [<c01b03f4>] (do_last+0x198/0x5b8) from [<c01b20f8>] (do_filp_open+0x168/0x49c)
> > [1042949.624482] [<c01b20f8>] (do_filp_open+0x168/0x49c) from [<c01a555c>] (do_sys_open+0x58/0x11c)
> > [1042949.633315] [<c01a555c>] (do_sys_open+0x58/0x11c) from [<c0136ee0>] (ret_fast_syscall+0x0/0x2c)
>
> dpkg and procmail were just showing up for you in top because it was
> sorting by memory usage, which isn't what we were looking for here.
It was not. The CPU numbers were low due to a 'find' which consumes a
lot now and then. This one shows better:
top - 12:32:22 up 12 days, 2:01, 32 users, load average: 13.48, 13.37, 13.39
Tasks: 199 total, 12 running, 186 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.0%us, 75.4%sy, 0.0%ni, 24.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 515004k total, 366200k used, 148804k free, 28k buffers
Swap: 4302560k total, 174188k used, 4128372k free, 170124k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11804 ookhoi 39 19 4700 64 52 R 8.8 0.0 717:10.30 procmail
6036 ookhoi 39 19 2692 64 52 R 8.5 0.0 872:47.95 procmail
18871 root 39 19 2540 32 20 R 8.5 0.0 1531:53 lzop
20305 ookhoi 39 19 2692 68 56 R 8.5 0.0 613:53.59 procmail
20378 ookhoi 39 19 2692 68 56 R 8.5 0.0 613:52.37 procmail
23661 ookhoi 39 19 2692 80 68 R 8.5 0.0 1311:24 procmail
27910 root 39 19 15096 3748 304 R 8.5 0.7 641:48.25 dpkg
11373 ookhoi 39 19 4800 64 52 R 8.2 0.0 717:27.50 procmail
18894 ookhoi 39 19 2692 64 52 R 8.2 0.0 614:17.80 procmail
26409 root 39 19 2264 32 28 R 8.2 0.0 1529:44 mv
27606 ookhoi 39 19 9084 40 28 R 8.2 0.0 3640:41 procmail
11120 root 20 0 0 0 0 S 5.6 0.0 0:02.94 flush-btrfs-2
> In your case, the blocking is almost certainly due to your failing
> disk.
Also for procmail and dpkg? Which do not operate on the disk that seems
to fail, and is located under /holding/ ?
Anyway, I'll reboot the machine this afternoon with the suspect disk
removed.
Thanks again for your reply cwillu.
Sander
--
Humilis IT Services and Solutions
http://www.humilis.net
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2011-01-31 11:40 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-01-25 8:51 flush-btrfs-1 hangs when building openwrt Daniel Poelzleithner
2011-01-25 15:30 ` Josef Bacik
2011-01-25 21:16 ` Daniel Poelzleithner
2011-01-31 10:52 ` Sander
2011-01-31 11:08 ` cwillu
2011-01-31 11:18 ` Sander
2011-01-31 11:26 ` cwillu
2011-01-31 11:40 ` Sander
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.