Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

Hi,

from time to time i'm getting a lot of strange errors like these:
[1217894.093015]
[1217894.150529] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1
[1217894.169930] Call Trace:
[1217894.189392]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.209515]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
[1217894.229591]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.249969]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.269840]  [<ffffffff8117f43c>] ? 
xfs_dir2_block_lookup_int+0x45/0x197
[1217894.289755]  [<ffffffff8117f43c>] ? 
xfs_dir2_block_lookup_int+0x45/0x197
[1217894.309385]  [<ffffffff813755ed>] ? tcp_write_xmit+0x883/0x96c
[1217894.328966]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
[1217894.348653]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
[1217894.368696]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
[1217894.388523]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
[1217894.408019]  [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e
[1217894.427418]  [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934
[1217894.446855]  [<ffffffff810c7242>] ? 
poll_select_copy_remaining+0xd0/0xf3
[1217894.466545]  [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b
[1217894.486469]  [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103
[1217894.506529]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
[1217894.526534] ffff8803104df000: 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00  ................
[1217894.546783] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) 
at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff8117bdea
[1217894.546785]
[1217894.606014] Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1
[1217894.626163] Call Trace:
[1217894.645625]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.665465]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
[1217894.685302]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.705135]  [<ffffffff810c7442>] ? poll_freewait+0x3d/0x8a
[1217894.725179]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[1217894.745439]  [<ffffffff8117f43c>] ? 
xfs_dir2_block_lookup_int+0x45/0x197
[1217894.765485]  [<ffffffff8117f43c>] ? 
xfs_dir2_block_lookup_int+0x45/0x197
[1217894.785179]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
[1217894.805010]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
[1217894.824879]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
[1217894.844689]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
[1217894.864661]  [<ffffffff810c1c0f>] ? do_lookup+0xd5/0x1b3
[1217894.884555]  [<ffffffff810c3d04>] ? __link_path_walk+0x8f5/0xda2
[1217894.904367]  [<ffffffff813368e3>] ? sock_aio_write+0xd6/0xe1
[1217894.924229]  [<ffffffff810c43df>] ? path_walk+0x66/0xcb
[1217894.944091]  [<ffffffff810c4512>] ? do_path_lookup+0x20/0x41
[1217894.963992]  [<ffffffff810c4e39>] ? user_path_at+0x48/0x7f
[1217894.984002]  [<ffffffff81053986>] ? autoremove_wake_function+0x0/0x2e
[1217895.004471]  [<ffffffff81059edd>] ? ktime_get_ts+0x68/0xb2
[1217895.024155]  [<ffffffff810bd518>] ? vfs_fstatat+0x2c/0x58
[1217895.042995]  [<ffffffff810bd6a8>] ? sys_newlstat+0x11/0x30
[1217895.061008]  [<ffffffff810baa7e>] ? vfs_write+0x134/0x169
[1217895.079137]  [<ffffffff810bab6f>] ? sys_write+0x45/0x6e
[1217895.096929]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
[1217895.115018] ffff8803104df000: 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 00  ................
[1217895.133667] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) 
at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff8117bdea

I've already repaired the filesystem twice but i'm still getting these 
errors again.

Kernel: Vanilla 2.6.32.22

XFS Progs Debian Lenny: xfs_repair -V
xfs_repair version 2.9.8

Stfean

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 08:25:20 +0200 vous écriviez:

> I've already repaired the filesystem twice but i'm still getting
> these errors again.

I had this problem in the past, but only with much lower kernel
versions, what does "xfs_info /dev/sdc3" says? What are the mount
options?

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

Am 26.10.2010 09:22, schrieb Emmanuel Florac:
> xfs_info /dev/sdc3

Hi here are the details:

root@server361-han:~# xfs_info /dev/sdc3

meta-data=/dev/sdc3              isize=256    agcount=32, 
agsize=40038461 blks
          =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=1281230752, imaxpct=25
          =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
          =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

root@server361-han:~# mount|grep sdc3

/dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota)
/sdc3/serverbackup on /serverbackup type none (rw,bind)
/sdc3/serverbackup_rootserver on /serverbackup_rootserver type none 
(rw,bind)

Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 09:24:57 +0200 vous écriviez:

> /dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota)
> /sdc3/serverbackup on /serverbackup type none (rw,bind)
> /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none 
> (rw,bind)

OK, looks fine to me. Aren't there any messages related to /dev/sdc in
dmesg? Apparently it's a single drive, any IO errors? What's the smart
status like? 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

>> /dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota)
>> /sdc3/serverbackup on /serverbackup type none (rw,bind)
>> /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none
>> (rw,bind)
>
> OK, looks fine to me. Aren't there any messages related to /dev/sdc in
> dmesg? Apparently it's a single drive, any IO errors? What's the smart
> status like?
No it is a Raid 5 (3Ware Controller). First Message in dmesg is 
(attention reverse order):
2010-10-26 02:07:42     [1217871.154330] Pid: 8345, comm: rsync Not 
tainted 2.6.32.22intel #1
2010-10-26 02:07:42     [1217871.178925] Call Trace:
2010-10-26 02:07:42     [1217871.202924] [] ? xfs_da_read_buf+0x24/0x29
2010-10-26 02:07:42     [1217871.226724] [] ? xfs_da_do_buf+0x557/0x620
2010-10-26 02:07:42     [1217871.249821] [] ? xfs_da_read_buf+0x24/0x29
2010-10-26 02:07:42     [1217871.272475] [] ? xfs_da_read_buf+0x24/0x29
2010-10-26 02:07:42     [1217871.294558] [] ? 
xfs_dir2_block_lookup_int+0x45/0x197
2010-10-26 02:07:42     [1217871.316206] [] ? 
xfs_dir2_block_lookup_int+0x45/0x197
2010-10-26 02:07:42     [1217871.337603] [] ? 
xfs_dir2_block_lookup+0x18/0xb0
2010-10-26 02:07:42     [1217871.358986] [] ? xfs_dir_lookup+0xd5/0x147
2010-10-26 02:07:42     [1217871.380585] [] ? xfs_lookup+0x47/0xa3
2010-10-26 02:07:42     [1217871.402313] [] ? xfs_vn_lookup+0x3c/0x7b
2010-10-26 02:07:42     [1217871.424465] [] ? do_lookup+0xd5/0x1b3
2010-10-26 02:07:42     [1217871.445750] [] ? __link_path_walk+0x8f5/0xda2
2010-10-26 02:07:42     [1217871.466323] [] ? sock_aio_write+0xd6/0xe1
2010-10-26 02:07:42     [1217871.487141] [] ? path_walk+0x66/0xcb
2010-10-26 02:07:42     [1217871.507783] [] ? do_path_lookup+0x20/0x41
2010-10-26 02:07:42     [1217871.528673] [] ? user_path_at+0x48/0x7f
2010-10-26 02:07:42     [1217871.549518] [] ? 
autoremove_wake_function+0x0/0x2e
2010-10-26 02:07:42     [1217871.569930] [] ? vfs_fstatat+0x2c/0x58
2010-10-26 02:07:42     [1217871.589714] [] ? _atomic_dec_and_lock+0x33/0x50
2010-10-26 02:07:42     [1217871.609072] [] ? sys_newlstat+0x11/0x30
2010-10-26 02:07:42     [1217871.627679] [] ? mntput_no_expire+0x23/0xdf
2010-10-26 02:07:42     [1217871.645558] [] ? filp_close+0x5b/0x62
2010-10-26 02:07:41     [1217870.507397] ffff8803104df000: 00 00 00 00 
00 00 00 00 00 00 00 00 00 00 00 00 ................
2010-10-26 02:07:41     [1217870.530516] Filesystem "sdc3": XFS internal 
error xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c. 
Caller 0xffffffff8117bdea
2010-10-26 02:07:41     [1217870.530518]
2010-10-26 02:07:41     [1217870.603393] Pid: 8345, comm: rsync Not 
tainted 2.6.32.22intel #1
2010-10-26 02:07:41     [1217870.628925] Call Trace:

Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 09:53:34 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> No it is a Raid 5 (3Ware Controller). 

What model? Which firmware? If it's a 96x0 model, notice that 4.x.x
firmwares added lots of performance enhancements and solved many
problems.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

Hi,

it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.

So you mean i should upgrade to 4.x Firmware? Do i then have to do a 
filesystem repair? Or just wait if the error accours again?

Stefan

Am 26.10.2010 13:01, schrieb Emmanuel Florac:
> Le Tue, 26 Oct 2010 09:53:34 +0200
> Stefan Priebe - Profihost AG<s.priebe@profihost.ag>  écrivait:
>
>> No it is a Raid 5 (3Ware Controller).
>
> What model? Which firmware? If it's a 96x0 model, notice that 4.x.x
> firmwares added lots of performance enhancements and solved many
> problems.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 13:03:07 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.

Augh, this firmware is antique :) The latest is 4.10.xx.xx. Anyway,
don't use firmware before 3.08.xx.xx, there were some nasty bugs.

> So you mean i should upgrade to 4.x Firmware? 

Definitely. It will much improve performances, too. Simply download the
firmware file, extract it, and flash the controller with tw_cli :

tw_cli /cXX update fw=prom0006.img 

(the firmware is always in the prom00xx.img file).

> Do i then have to do a 
> filesystem repair? Or just wait if the error accours again?

No, the filesystem will be fine. However you should start a RAID scrub
with 

tw_cli /cXX/uXX start verify

This will rebuild the parity with the new 4.X format (faster writes)
and help detect any hardware fault.

My other advices : generally don't use cfq scheduler with a RAID
controller, it will defeat the whole purpose of RAID cache and command
reordering abilities. Use noop, generally, and deepen the queue :

echo "noop" > /sys/block/sdc/queue/scheduler
echo 512 > /sys/block/sdc/queue/nr_requests

Don't hesitate to enlarge tremendously the read-ahead cache, too. I
generally use about 512 to 1024 sectors  per drive as a rule of the
thumb, so a 4 drives array will use 2048 to 4096 :

blockdev --setra 4096 /dev/sdc

You should see a 100% write/read speed improvement with these
parameters.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

RE: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Larsen, Tore Høivaag]

You should keep the driver and firmware ins sync wither kernel.  Choose e.g. latest 9.4.3 iso for 2.6.9, and 9.5.3 for RHEL5/SLES11/Fedora13. Do not use 9.5.3 firmware with RHEL4/SLES10 native driver. If I remeber correctly tw_update doesn't even let you mix a downgrade driver with new firmware, and vs.versus.

Iow, If you are on a legacy level, stay with latest legacy 9.4.3 for it.   

http://kb.lsi.com/KnowledgebaseArticle14546.aspx

I've use a lot of 9550SXU and 9650SE and various MegaRaid. I find 3ware much easier to work with and better performing with xfs.


//seisqc1> /c2 show all
/c2 Driver Version = 2.26.08.003-2.6.18RH
/c2 Model = 9550SXU-8LP
/c2 Available Memory = 112MB
/c2 Firmware Version = FE9X 3.08.00.029


Rgds,
--ToreL

________________________________________
From: xfs-bounces@oss.sgi.com [xfs-bounces@oss.sgi.com] On Behalf Of Stefan Priebe - Profihost AG [s.priebe@profihost.ag]
Sent: Tuesday, October 26, 2010 1:03 PM
To: Emmanuel Florac
Cc: xfs@oss.sgi.com
Subject: Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

Hi,

it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.

So you mean i should upgrade to 4.x Firmware? Do i then have to do a
filesystem repair? Or just wait if the error accours again?

Stefan

Am 26.10.2010 13:01, schrieb Emmanuel Florac:
> Le Tue, 26 Oct 2010 09:53:34 +0200
> Stefan Priebe - Profihost AG<s.priebe@profihost.ag>  écrivait:
>
>> No it is a Raid 5 (3Ware Controller).
>
> What model? Which firmware? If it's a 96x0 model, notice that 4.x.x
> firmwares added lots of performance enhancements and solved many
> problems.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 12:30:22 +0100
Larsen, Tore Høivaag <tore.hoivaag.larsen@cggveritas.com> écrivait:

> You should keep the driver and firmware ins sync wither kernel.
> Choose e.g. latest 9.4.3 iso for 2.6.9, and 9.5.3 for
> RHEL5/SLES11/Fedora13. Do not use 9.5.3 firmware with RHEL4/SLES10
> native driver. If I remeber correctly tw_update doesn't even let you
> mix a downgrade driver with new firmware, and vs.versus.

Yes, it will complain in case your driver/api isn't compatible with the
newer firmware. Of course, only actually upgrade if it tells you :
"proceed to update".

However I'm pretty sure 2.6.23 is safe for 4.x firmware (2.6.24
definitely is).

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

Hi,

thanks for your suggestions.

 > My other advices : generally don't use cfq scheduler with a RAID
 > controller, it will defeat the whole purpose of RAID cache and command
 > reordering abilities. Use noop, generally, and deepen the queue :
 >
 > echo "noop">  /sys/block/sdc/queue/scheduler
 > echo 512>  /sys/block/sdc/queue/nr_requests

Is there a way to make this static to this disk?

 > Don't hesitate to enlarge tremendously the read-ahead cache, too. I
 > generally use about 512 to 1024 sectors  per drive as a rule of the
 > thumb, so a 4 drives array will use 2048 to 4096 :
 >
 > blockdev --setra 4096 /dev/sdc
Is there also a way to make this static?

 > You should see a 100% write/read speed improvement with these
 > parameters.
That would be great.


Thanks Stefan

Am 26.10.2010 13:25, schrieb Emmanuel Florac:
> Le Tue, 26 Oct 2010 13:03:07 +0200
> Stefan Priebe - Profihost AG<s.priebe@profihost.ag>  écrivait:
>
>> it is a 9650SE-8LPML with Firmware: FE9X 3.06.00.003.
>
> Augh, this firmware is antique :) The latest is 4.10.xx.xx. Anyway,
> don't use firmware before 3.08.xx.xx, there were some nasty bugs.
>
>> So you mean i should upgrade to 4.x Firmware?
>
> Definitely. It will much improve performances, too. Simply download the
> firmware file, extract it, and flash the controller with tw_cli :
>
> tw_cli /cXX update fw=prom0006.img
>
> (the firmware is always in the prom00xx.img file).
>
>> Do i then have to do a
>> filesystem repair? Or just wait if the error accours again?
>
> No, the filesystem will be fine. However you should start a RAID scrub
> with
>
> tw_cli /cXX/uXX start verify
>
> This will rebuild the parity with the new 4.X format (faster writes)
> and help detect any hardware fault.
>
> My other advices : generally don't use cfq scheduler with a RAID
> controller, it will defeat the whole purpose of RAID cache and command
> reordering abilities. Use noop, generally, and deepen the queue :
>
> echo "noop">  /sys/block/sdc/queue/scheduler
> echo 512>  /sys/block/sdc/queue/nr_requests
>
> Don't hesitate to enlarge tremendously the read-ahead cache, too. I
> generally use about 512 to 1024 sectors  per drive as a rule of the
> thumb, so a 4 drives array will use 2048 to 4096 :
>
> blockdev --setra 4096 /dev/sdc
>
> You should see a 100% write/read speed improvement with these
> parameters.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Tue, 26 Oct 2010 14:09:12 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

>  > echo "noop">  /sys/block/sdc/queue/scheduler
>  > echo 512>  /sys/block/sdc/queue/nr_requests  
> 
> Is there a way to make this static to this disk?
> [snip]
>  > blockdev --setra 4096 /dev/sdc  
> Is there also a way to make this static?

The easiest way is to add these commands to /etc/rc.local.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1253 bytes --]

On Dienstag, 26. Oktober 2010 Emmanuel Florac wrote:
> echo "noop" > /sys/block/sdc/queue/scheduler
> echo 512 > /sys/block/sdc/queue/nr_requests
> blockdev --setra 4096 /dev/sdc

How about this small script in /etc/init.d/boot.local?

for i in /dev/xvd? /dev/sd? ; do
   if test "$i" == "${i%\?}" ; then
      echo i=$i
      blockdev --setra 1024 $i
      j=${i#/dev/}
      echo j=$j
      echo noop >/sys/block/$j/queue/scheduler
      echo 512 >/sys/block/$j/queue/nr_requests
   fi
done

This works for VMs within XenServer (/dev/xvda) and real servers 
(/dev/sda), and sets some values for all drives.

BTW: any recommendations for these tuning knobs on VM-ized servers? What 
should be set in the VMs and what on the host? Has someone got tested 
numbers? I'm currently testing things, but it's so damn hard to get any 
useful, reliable results.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Radiointerview zum Thema Spam ******
http://www.it-podcast.at/archiv.html#podcast-100716

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Dave Chinner]

On Wed, Oct 27, 2010 at 01:23:35AM +0200, Michael Monnerie wrote:
> On Dienstag, 26. Oktober 2010 Emmanuel Florac wrote:
> > echo "noop" > /sys/block/sdc/queue/scheduler
> > echo 512 > /sys/block/sdc/queue/nr_requests
> > blockdev --setra 4096 /dev/sdc
> 
> How about this small script in /etc/init.d/boot.local?
> 
> for i in /dev/xvd? /dev/sd? ; do
>    if test "$i" == "${i%\?}" ; then
>       echo i=$i
>       blockdev --setra 1024 $i
>       j=${i#/dev/}
>       echo j=$j
>       echo noop >/sys/block/$j/queue/scheduler
>       echo 512 >/sys/block/$j/queue/nr_requests
>    fi
> done
> 
> This works for VMs within XenServer (/dev/xvda) and real servers 
> (/dev/sda), and sets some values for all drives.

I'd suggest that people learn how to tweak udev hotplug rules so
that when the device is first created (i.e. during hotplug) the
scheduler, queue depth and readahead are set automatically. That way
you don't have to rely on devices being discovered before your script
runs...

Another benefit of doing it this way is that it is easy to set
default rules for different types of devices based on regex matching
e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to
set up.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Wed, 27 Oct 2010 01:23:35 +0200 vous écriviez:

> BTW: any recommendations for these tuning knobs on VM-ized servers?
> What should be set in the VMs and what on the host?

>From my experience, using iSCSI as the transport to export lvs to the
VMs, the best performance is reach by setting scheduler to noop and
queue depth on the target and the VM; but setting the readahead only on
the VM side.

For different configurations, as usual, YMMV. 

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

So what values do you recommand in general for iSCSI on Serverside and 
which values on iSCSI client side?

Stefan

Am 27.10.2010 08:04, schrieb Emmanuel Florac:
> Le Wed, 27 Oct 2010 01:23:35 +0200 vous écriviez:
>
>> BTW: any recommendations for these tuning knobs on VM-ized servers?
>> What should be set in the VMs and what on the host?
>
>  From my experience, using iSCSI as the transport to export lvs to the
> VMs, the best performance is reach by setting scheduler to noop and
> queue depth on the target and the VM; but setting the readahead only on
> the VM side.
>
> For different configurations, as usual, YMMV.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Wed, 27 Oct 2010 09:00:06 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> So what values do you recommand in general for iSCSI on Serverside
> and which values on iSCSI client side?
> 

Depends upon the hardware. nr_requests should be 512 for 3Ware cards,
256 to 1024 for other controllers (always more than the default 128
anyway). 

Same rule goes for read-ahead; however many RAID controllers (Areca,
Adaptec) "cheat" and do read-ahead optimisation by themselves (which
may harm random access). The default read-ahead value (256) was OK for
obsolete ATA drives with a few KB of cache; nowadays all drives have 32
or 64 MB, and controllers 512 MB to several GB, so more read-ahead can
do no harm but fill the cache up. I often go up to 65536 or 131072
sectors (for 24 or 48 drives arrays). Keep in mind that this is
sequentail-raid optimisation; random access (such as VM work) may give
better result with lower read-ahead values. Benchmarking your
application is king here.

Just for information : I currently run some servers with kernel
2.6.22.18 (march 2008) and 3Ware firmware 4.10.0.7, so upgrading your
2.6.23 system should be perfectly safe from a driver compatibility
point of view.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

I'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.

OK your values are for the iscsi_target right? Should i use the SAME 
values on the iscsi connector side?

Thanks Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Wed, 27 Oct 2010 12:08:57 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

> I'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.
> 
> OK your values are for the iscsi_target right? Should i use the SAME 
> values on the iscsi connector side?

Yes, the initiator values are more important here. Basically, the iscsi
target service should transmit the initiator's requests as they come.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

 > Yes, the initiator values are more important here. Basically, the
 > iscsitarget service should transmit the initiator's requests as they
 > come.
What do you mean by this sentence? Sorry no native english speaker.


Am 27.10.2010 12:12, schrieb Emmanuel Florac:
> Le Wed, 27 Oct 2010 12:08:57 +0200
> Stefan Priebe - Profihost AG<s.priebe@profihost.ag>  écrivait:
>
>> I'm having 2.6.32 Kernel not 2.6.23 :-) Upgrade works fine yesterday.
>>
>> OK your values are for the iscsi_target right? Should i use the SAME
>> values on the iscsi connector side?
>
> Yes, the initiator values are more important here. Basically, the iscsi
> target service should transmit the initiator's requests as they come.
>

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1730 bytes --]

On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
> I'd suggest that people learn how to tweak udev hotplug rules so
> that when the device is first created (i.e. during hotplug) the
> scheduler, queue depth and readahead are set automatically. That way
> you don't have to rely on devices being discovered before your script
> runs...
> 
> Another benefit of doing it this way is that it is easy to set
> default rules for different types of devices based on regex matching
> e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to
> set up.

Sounds very nice. But the script I use will still work when upgrading 
the server from openSUSE 11.2 to 11.3, and is therefore the preferred 
choice for me.

Also, I'd need to find information and learn how to tweak udev hotplug 
rules. I want to implement this on about 30 VMs on 2 different hosts, 
with 3 different release states of servers (openSUSE 11.1, 11.2 and 
11.3). The chance is high that I'd need two or three different udev 
tweaks for the different releases, so I don't see the benefit of udev 
for me. I wrote the script in about the same time I wrote this mail.

That's always the problem between developers ("ah, cool new stuff") and 
admins ("i need this on 500 servers with least possible work for me, and 
it must still work after any updates/upgrades").

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Radiointerview zum Thema Spam ******
http://www.it-podcast.at/archiv.html#podcast-100716

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Wed, 27 Oct 2010 12:22:47 +0200
Stefan Priebe - Profihost AG <s.priebe@profihost.ag> écrivait:

>  > Yes, the initiator values are more important here. Basically, the
>  > iscsitarget service should transmit the initiator's requests as
>  > they come.  
> What do you mean by this sentence? Sorry no native english speaker.

The settings that you apply on the initiator side are sent to the
target ; the target applies them to the underlying device.

This is valid only when exporting a block device (lv, partition), it
wouldn't work the same for fileio.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Dave Chinner]

On Wed, Oct 27, 2010 at 12:58:45PM +0200, Michael Monnerie wrote:
> On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
> > I'd suggest that people learn how to tweak udev hotplug rules so
> > that when the device is first created (i.e. during hotplug) the
> > scheduler, queue depth and readahead are set automatically. That way
> > you don't have to rely on devices being discovered before your script
> > runs...
> > 
> > Another benefit of doing it this way is that it is easy to set
> > default rules for different types of devices based on regex matching
> > e.g. different configs for "sd*" vs "dm*" vs "vd*" are trivial to
> > set up.
> 
> Sounds very nice. But the script I use will still work when upgrading 
> the server from openSUSE 11.2 to 11.3, and is therefore the preferred 
> choice for me.
> 
> Also, I'd need to find information and learn how to tweak udev hotplug 
> rules.

GIYF.

> I want to implement this on about 30 VMs on 2 different hosts, 
> with 3 different release states of servers (openSUSE 11.1, 11.2 and 
> 11.3).

The udev rule format hasn't changed in a long while. The same rule
set should work on all of these.

> The chance is high that I'd need two or three different udev 
> tweaks for the different releases, so I don't see the benefit of udev 
> for me. I wrote the script in about the same time I wrote this mail.

You'd need one regex per device type you want to tweak with
different values.

> That's always the problem between developers ("ah, cool new stuff") and 
> admins ("i need this on 500 servers with least possible work for me, and 
> it must still work after any updates/upgrades").

This is not "cool new stuff" - I've seen it used for exactly this
purpose for _several years_ by distros. e.g. pulling the identifier
string from the device to set hardware device specific tunables,
changing default dm/md readahead, etc. It's not new, and is easy to
configure generically so it works on a wide array of different machines.

And if you hotplug devices, it just works automatically - you don't
need to rerun a script after every hotplug...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Michael Monnerie]

[-- Attachment #1.1: Type: Text/Plain, Size: 1460 bytes --]

On Mittwoch, 27. Oktober 2010 Dave Chinner wrote:
> > Also, I'd need to find information and learn how to tweak udev
> > hotplug  rules.
> 
> GIYF.

Thanks, I know how to search ;-)
*Finding* the information doesn't mean *understanding* how to do it. It 
takes time.

> You'd need one regex per device type you want to tweak with
> different values.
> This is not "cool new stuff" - I've seen it used for exactly this
> purpose for several years by distros. e.g. pulling the identifier
> string from the device to set hardware device specific tunables,
> changing default dm/md readahead, etc. It's not new, and is easy to
> configure generically so it works on a wide array of different
> machines.

I meant: When upgrading the system from 11.1 to 11.2, I'm sure the 
system replaces the correspondig script, or at least messes around with 
them. I know /etc/init.d/boot.local stays the same.
 
> And if you hotplug devices, it just works automatically - you don't
> need to rerun a script after every hotplug...

Now that indeed is a pretty good advantage.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services
http://proteger.at [gesprochen: Prot-e-schee]
Tel: 0660 / 415 65 31

****** Radiointerview zum Thema Spam ******
http://www.it-podcast.at/archiv.html#podcast-100716

// Wir haben im Moment zwei Häuser zu verkaufen:
// http://zmi.at/langegg/
// http://zmi.at/haus2009/

[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 121 bytes --]

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

Hi,

i've done all you advises - updating 3ware, setting other scheduler, 
block size, ...

but today i got again hundrets of these messages:
[318343.901809] Filesystem "sdc3": XFS internal error xfs_da_do_buf(2) 
at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller 0xffffffff8117bdea
[318343.901811]
[318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1
[318343.901815] Call Trace:
[318343.901819]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[318343.901823]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
[318343.901827]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[318343.901831]  [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0
[318343.901834]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
[318343.901839]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
[318343.901843]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
[318343.901846]  [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c
[318343.901850]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
[318343.901854]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
[318343.901857]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
[318343.901861]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
[318343.901865]  [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e
[318343.901869]  [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934
[318343.901873]  [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3
[318343.901877]  [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b
[318343.901880]  [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103
[318343.901884]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b

Stefan

Am 26.10.2010 09:24, schrieb Stefan Priebe - Profihost AG:
> Am 26.10.2010 09:22, schrieb Emmanuel Florac:
>> xfs_info /dev/sdc3
>
> Hi here are the details:
>
> root@server361-han:~# xfs_info /dev/sdc3
>
> meta-data=/dev/sdc3 isize=256 agcount=32, agsize=40038461 blks
> = sectsz=512 attr=0
> data = bsize=4096 blocks=1281230752, imaxpct=25
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096
> log =internal bsize=4096 blocks=32768, version=2
> = sectsz=512 sunit=0 blks, lazy-count=1
> realtime =none extsz=4096 blocks=0, rtextents=0
>
> root@server361-han:~# mount|grep sdc3
>
> /dev/sdc3 on /sdc3 type xfs (rw,noatime,nodiratime,nobarrier,prjquota)
> /sdc3/serverbackup on /serverbackup type none (rw,bind)
> /sdc3/serverbackup_rootserver on /serverbackup_rootserver type none
> (rw,bind)
>
> Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Mon, 01 Nov 2010 09:01:12 +0100 vous écriviez:

> but today i got again hundrets of these messages:
> [318343.901809] Filesystem "sdc3": XFS internal error
> xfs_da_do_buf(2) 

That's weird. What do "uname -a" and "free" tells? I'm wondering if
there isn't some low memory condition at work here.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

>> but today i got again hundrets of these messages:
>> [318343.901809] Filesystem "sdc3": XFS internal error
>> xfs_da_do_buf(2)
>
> That's weird. What do "uname -a" and "free" tells? I'm wondering if
> there isn't some low memory condition at work here.
# uname -a
Linux server 2.6.32.22intel #1 SMP Tue Sep 21 11:05:46 CEST 2010 x86_64 
GNU/Linux

# free
              total       used       free     shared    buffers     cached
Mem:      16470416   15098888    1371528          0         48    6019036
-/+ buffers/cache:    9079804    7390612
Swap:      1469908          0    1469908

Stefan

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Dave Chinner]

On Mon, Nov 01, 2010 at 09:01:12AM +0100, Stefan Priebe - Profihost AG wrote:
> Hi,
> 
> i've done all you advises - updating 3ware, setting other scheduler,
> block size, ...
> 
> but today i got again hundrets of these messages:
> [318343.901809] Filesystem "sdc3": XFS internal error
> xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller
> 0xffffffff8117bdea
> [318343.901811]
> [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1
> [318343.901815] Call Trace:
> [318343.901819]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> [318343.901823]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
> [318343.901827]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> [318343.901831]  [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0
> [318343.901834]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> [318343.901839]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
> [318343.901843]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
> [318343.901846]  [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c
> [318343.901850]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
> [318343.901854]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
> [318343.901857]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
> [318343.901861]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
> [318343.901865]  [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e
> [318343.901869]  [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934
> [318343.901873]  [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3
> [318343.901877]  [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b
> [318343.901880]  [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103
> [318343.901884]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b

Have you run repair to fix all the existing problems after doing all
the firmware upgrades, etc?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

no, as Emmanuel told me i don't have todo that.

Stefan

Am 01.11.2010 13:27, schrieb Dave Chinner:
> On Mon, Nov 01, 2010 at 09:01:12AM +0100, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> i've done all you advises - updating 3ware, setting other scheduler,
>> block size, ...
>>
>> but today i got again hundrets of these messages:
>> [318343.901809] Filesystem "sdc3": XFS internal error
>> xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller
>> 0xffffffff8117bdea
>> [318343.901811]
>> [318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1
>> [318343.901815] Call Trace:
>> [318343.901819]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
>> [318343.901823]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
>> [318343.901827]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
>> [318343.901831]  [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0
>> [318343.901834]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
>> [318343.901839]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
>> [318343.901843]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
>> [318343.901846]  [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c
>> [318343.901850]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
>> [318343.901854]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
>> [318343.901857]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
>> [318343.901861]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
>> [318343.901865]  [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e
>> [318343.901869]  [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934
>> [318343.901873]  [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3
>> [318343.901877]  [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b
>> [318343.901880]  [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103
>> [318343.901884]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
>
> Have you run repair to fix all the existing problems after doing all
> the firmware upgrades, etc?
>
> Cheers,
>
> Dave.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Dave Chinner]

[ please don't top-post - fixed. ]

On Mon, Nov 01, 2010 at 01:28:46PM +0100, Stefan Priebe - Profihost AG wrote:
> Am 01.11.2010 13:27, schrieb Dave Chinner:
> >On Mon, Nov 01, 2010 at 09:01:12AM +0100, Stefan Priebe - Profihost AG wrote:
> >>Hi,
> >>
> >>i've done all you advises - updating 3ware, setting other scheduler,
> >>block size, ...
> >>
> >>but today i got again hundrets of these messages:
> >>[318343.901809] Filesystem "sdc3": XFS internal error
> >>xfs_da_do_buf(2) at line 2112 of file fs/xfs/xfs_da_btree.c.  Caller
> >>0xffffffff8117bdea
> >>[318343.901811]
> >>[318343.901813] Pid: 20299, comm: rsync Not tainted 2.6.32.22intel #1
> >>[318343.901815] Call Trace:
> >>[318343.901819]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> >>[318343.901823]  [<ffffffff8117bcb7>] ? xfs_da_do_buf+0x557/0x620
> >>[318343.901827]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> >>[318343.901831]  [<ffffffff810538a7>] ? bit_waitqueue+0x10/0xa0
> >>[318343.901834]  [<ffffffff8117bdea>] ? xfs_da_read_buf+0x24/0x29
> >>[318343.901839]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
> >>[318343.901843]  [<ffffffff8117f43c>] ? xfs_dir2_block_lookup_int+0x45/0x197
> >>[318343.901846]  [<ffffffff813751d0>] ? tcp_write_xmit+0x466/0x96c
> >>[318343.901850]  [<ffffffff8117f9c2>] ? xfs_dir2_block_lookup+0x18/0xb0
> >>[318343.901854]  [<ffffffff8117e71c>] ? xfs_dir_lookup+0xd5/0x147
> >>[318343.901857]  [<ffffffff811a11bd>] ? xfs_lookup+0x47/0xa3
> >>[318343.901861]  [<ffffffff811a90d1>] ? xfs_vn_lookup+0x3c/0x7b
> >>[318343.901865]  [<ffffffff810c1de7>] ? __lookup_hash+0xfa/0x11e
> >>[318343.901869]  [<ffffffff810c51a8>] ? do_filp_open+0x208/0x934
> >>[318343.901873]  [<ffffffff810c7242>] ? poll_select_copy_remaining+0xd0/0xf3
> >>[318343.901877]  [<ffffffff810cd62a>] ? alloc_fd+0x67/0x10b
> >>[318343.901880]  [<ffffffff810b8a60>] ? do_sys_open+0x55/0x103
> >>[318343.901884]  [<ffffffff8100ae6b>] ? system_call_fastpath+0x16/0x1b
> >
> >Have you run repair to fix all the existing problems after doing all
> >the firmware upgrades, etc?
>
> no, as Emmanuel told me i don't have todo that.

Perhaps you've been told the wrong thing.

You are getting reports of on-disk corruption being encountered. You
need to run repair to find and fix all these existing problems.
before these problems will go away. Otherwise you'll keep getting
these reports every time you read the corrupted directory....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Emmanuel Florac]

Le Mon, 01 Nov 2010 13:28:46 +0100 vous écriviez:

> no, as Emmanuel told me i don't have todo that.

Did you run the "verify" round on the array after the upgrade? That
would rebuild all checksum and insure that the RAID is fully coherent.
If the verify comes out with any message, there is a sloght chance that
the data on disk wasn't entirely coherent or clean.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: Pid: 8345, comm: rsync Not tainted 2.6.32.22intel #1

[Stefan Priebe - Profihost AG]

> Did you run the "verify" round on the array after the upgrade? That
> would rebuild all checksum and insure that the RAID is fully coherent.
> If the verify comes out with any message, there is a sloght chance that
> the data on disk wasn't entirely coherent or clean.

shure i have - was done without any problems.


_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs