* rbd
@ 2011-05-12 10:53 Fyodor Ustinov
2011-05-12 16:08 ` rbd Sage Weil
0 siblings, 1 reply; 7+ messages in thread
From: Fyodor Ustinov @ 2011-05-12 10:53 UTC (permalink / raw)
To: ceph-devel
Hi!
Latest (git pulled) version of 2.6 kernel. Ceph - 0.27.1
Still troubles with rbd. Now with ocfs2 no messages in syslog, but iozone
still return error:
#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 237G 15G 210G 7% /
none 2.0G 164K 2.0G 1% /dev
none 2.0G 0 2.0G 0% /dev/shm
none 2.0G 52K 2.0G 1% /var/run
none 2.0G 0 2.0G 0% /var/lock
/dev/rbd0 1000G 3.2G 997G 1% /mnt
# cd /mnt/
root@stb1:/mnt# ls
lost+found
root@stb1:/mnt# iozone -a -n4g -g20g
Iozone: Performance Test of File I/O
Version $Revision: 3.373 $
Compiled for 64 bit mode.
Build: linux-AMD64
[...]
Run began: Thu May 12 13:42:45 2011
Auto Mode
Using minimum file size of 4194304 kilobytes.
Using maximum file size of 20971520 kilobytes.
Command line used: iozone -a -n4g -g20g
Output is in Kbytes/sec
Time Resolution = 0.000001 seconds.
Processor cache size set to 1024 Kbytes.
Processor cache line size set to 32 bytes.
File stride size set to 17 * record size.
random random
bkwd record stride
KB reclen write rewrite read reread read write
read rewrite read fwrite frewrite fread freread
4194304 64 146906 150779
Error in file: Found ?7f24fb70a210? Expecting ?3a3a3a3a3a3a3a3a? addr
7fe5b720e000
Error in file: Position 2481971200
Record # 37871 Record size 64 kb
where 7fe5b720e000 loop 57344
root@stb1:/mnt#
Can I help to catch this error?
WBR,
Fyodor.
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: rbd 2011-05-12 10:53 rbd Fyodor Ustinov @ 2011-05-12 16:08 ` Sage Weil 2011-05-12 17:01 ` rbd Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2011-05-12 16:08 UTC (permalink / raw) To: Fyodor Ustinov; +Cc: ceph-devel Hi Fyodor, > Hi! > > Latest (git pulled) version of 2.6 kernel. Ceph - 0.27.1 > > Still troubles with rbd. Now with ocfs2 no messages in syslog, but iozone > still return error: > > #df -h > Filesystem Size Used Avail Use% Mounted on > /dev/sda1 237G 15G 210G 7% / > none 2.0G 164K 2.0G 1% /dev > none 2.0G 0 2.0G 0% /dev/shm > none 2.0G 52K 2.0G 1% /var/run > none 2.0G 0 2.0G 0% /var/lock > /dev/rbd0 1000G 3.2G 997G 1% /mnt > > # cd /mnt/ > root@stb1:/mnt# ls > lost+found > root@stb1:/mnt# iozone -a -n4g -g20g > Iozone: Performance Test of File I/O > Version $Revision: 3.373 $ > Compiled for 64 bit mode. > Build: linux-AMD64 > > [...] > > Run began: Thu May 12 13:42:45 2011 > > Auto Mode > Using minimum file size of 4194304 kilobytes. > Using maximum file size of 20971520 kilobytes. > Command line used: iozone -a -n4g -g20g > Output is in Kbytes/sec > Time Resolution = 0.000001 seconds. > Processor cache size set to 1024 Kbytes. > Processor cache line size set to 32 bytes. > File stride size set to 17 * record size. > random random > bkwd record stride > KB reclen write rewrite read reread read write > read rewrite read fwrite frewrite fread freread > 4194304 64 146906 150779 > > Error in file: Found ?7f24fb70a210? Expecting ?3a3a3a3a3a3a3a3a? addr > 7fe5b720e000 > Error in file: Position 2481971200 > Record # 37871 Record size 64 kb > where 7fe5b720e000 loop 57344 > root@stb1:/mnt# > > Can I help to catch this error? This one is going to be a bit difficult because there are several layers (ocfs2, rbd client, osd) involved. It's presumably not ocfs2's fault, but trying the same thing with ext4 or something would remove that from the equation. I'm running iozone on ext3 now and not having any problems. Assuming the problem is in rbd, the question is whether it's the client or server side. That will be harder to diagnose, and will probably involve careful use of fiemap/bmap to map the file offset to a block device offset, which we can then translate into a rados block. That can be pulled directly out of the object store to confirm it has the right content... sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rbd 2011-05-12 16:08 ` rbd Sage Weil @ 2011-05-12 17:01 ` Sage Weil 2011-05-13 15:51 ` rbd Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2011-05-12 17:01 UTC (permalink / raw) To: Fyodor Ustinov; +Cc: ceph-devel On Thu, 12 May 2011, Sage Weil wrote: > equation. I'm running iozone on ext3 now and not having any problems. I take it back.. I just reproduced a similar error on ext2: random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 3570 3040 12640 13725 Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr 40a00000 Error in file: Position 2813329408 Record # 42928 Record size 64 kb where 40a00000 loop 0 I opened a tracker issue http://tracker.newdream.net/issues/1086 sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rbd 2011-05-12 17:01 ` rbd Sage Weil @ 2011-05-13 15:51 ` Sage Weil 2011-05-13 19:11 ` rbd Sage Weil 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2011-05-13 15:51 UTC (permalink / raw) To: Fyodor Ustinov; +Cc: ceph-devel On Thu, 12 May 2011, Sage Weil wrote: > On Thu, 12 May 2011, Sage Weil wrote: > > equation. I'm running iozone on ext3 now and not having any problems. > > I take it back.. I just reproduced a similar error on ext2: > > random random > bkwd record stride > KB reclen write rewrite read reread read write > read rewrite read fwrite frewrite fread freread > 4194304 64 3570 3040 12640 13725 > > Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr > 40a00000 > Error in file: Position 2813329408 > Record # 42928 Record size 64 kb > where 40a00000 loop 0 > > I opened a tracker issue http://tracker.newdream.net/issues/1086 Just an update: we've identified the problem but a fix is still in the works. The bio coming down is spanning an object boundary, and we're having trouble getting the block layer to either not do that or to handle the bio splitting properly. sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rbd 2011-05-13 15:51 ` rbd Sage Weil @ 2011-05-13 19:11 ` Sage Weil 2011-05-14 12:36 ` rbd Fyodor Ustinov 0 siblings, 1 reply; 7+ messages in thread From: Sage Weil @ 2011-05-13 19:11 UTC (permalink / raw) To: Fyodor Ustinov; +Cc: ceph-devel Fyodor, On Fri, 13 May 2011, Sage Weil wrote: > On Thu, 12 May 2011, Sage Weil wrote: > > On Thu, 12 May 2011, Sage Weil wrote: > > > equation. I'm running iozone on ext3 now and not having any problems. > > > > I take it back.. I just reproduced a similar error on ext2: > > > > random random > > bkwd record stride > > KB reclen write rewrite read reread read write > > read rewrite read fwrite frewrite fread freread > > 4194304 64 3570 3040 12640 13725 > > > > Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr > > 40a00000 > > Error in file: Position 2813329408 > > Record # 42928 Record size 64 kb > > where 40a00000 loop 0 > > > > I opened a tracker issue http://tracker.newdream.net/issues/1086 > > Just an update: we've identified the problem but a fix is still in the > works. The bio coming down is spanning an object boundary, and we're > having trouble getting the block layer to either not do that or to handle > the bio splitting properly. Can you try the req_coll branch of git://ceph.newdream.net/git/ceph-client.git and see if that fixes things for you? There's a memory leak fix and a fix for the split bio completion stuff. If it checks out I want to send this to Linus quickly so it'll be included in 2.6.39 (which is imminent). Thanks! sage ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: rbd 2011-05-13 19:11 ` rbd Sage Weil @ 2011-05-14 12:36 ` Fyodor Ustinov 0 siblings, 0 replies; 7+ messages in thread From: Fyodor Ustinov @ 2011-05-14 12:36 UTC (permalink / raw) To: Sage Weil; +Cc: ceph-devel On 05/13/2011 10:11 PM, Sage Weil wrote: > Fyodor, > > On Fri, 13 May 2011, Sage Weil wrote: >> On Thu, 12 May 2011, Sage Weil wrote: >>> On Thu, 12 May 2011, Sage Weil wrote: >>>> equation. I'm running iozone on ext3 now and not having any problems. >>> I take it back.. I just reproduced a similar error on ext2: >>> >>> random random >>> bkwd record stride >>> KB reclen write rewrite read reread read write >>> read rewrite read fwrite frewrite fread freread >>> 4194304 64 3570 3040 12640 13725 >>> >>> Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr >>> 40a00000 >>> Error in file: Position 2813329408 >>> Record # 42928 Record size 64 kb >>> where 40a00000 loop 0 >>> >>> I opened a tracker issue http://tracker.newdream.net/issues/1086 >> Just an update: we've identified the problem but a fix is still in the >> works. The bio coming down is spanning an object boundary, and we're >> having trouble getting the block layer to either not do that or to handle >> the bio splitting properly. > Can you try the req_coll branch of > git://ceph.newdream.net/git/ceph-client.git and see if that fixes things > for you? There's a memory leak fix and a fix for the split bio completion > stuff. If it checks out I want to send this to Linus quickly so it'll be > included in 2.6.39 (which is imminent). root@stb1:~# uname -a Linux stb1 2.6.39-rc7-ufm-ceph #1 SMP Sat May 14 13:50:12 EEST 2011 x86_64 x86_64 x86_64 GNU/Linux root@stb1:~# modprobe rbd root@stb1:~# echo "77.120.112.193 name=admin rbd tmt" > /sys/bus/rbd/add root@stb1:~# mkfs.ocfs2 -M local -T vmstore --fs-features=local,sparse /dev/rbd0 mkfs.ocfs2 1.6.3 Overwriting existing ocfs2 partition. Proceed (y/N): y Filesystem Type of vmstore Label: Features: local sparse backup-super unwritten inline-data strict-journal-super xattr refcount Block size: 4096 (12 bits) Cluster size: 1048576 (20 bits) Volume size: 1073741824000 (1024000 clusters) (262144000 blocks) Cluster groups: 32 (tail covers 24064 clusters, rest cover 32256 clusters) Extent allocator size: 3221225472 (768 groups) Journal size: 134217728 Node slots: 1 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 5 block(s) Formatting Journals: done Growing extent allocator: done Formatting slot map: done Formatting quota files: done Writing lost+found: done mkfs.ocfs2 successful root@stb1:~# mount /dev/rbd0 /mnt root@stb1:~# cd /mnt root@stb1:/mnt# iozone -a -n4g -g20g Iozone: Performance Test of File I/O Version $Revision: 3.373 $ Compiled for 64 bit mode. Build: linux-AMD64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Run began: Sat May 14 14:39:03 2011 Auto Mode Using minimum file size of 4194304 kilobytes. Using maximum file size of 20971520 kilobytes. Command line used: iozone -a -n4g -g20g Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 134566 135254 53718 52836 14839 78946 And now it is 20 minutes in this state.In the logs clean. Incoming network traffic- 3Mbytes/s But, we newer see this test before, likely an extremely low speed test "bkwd read" this is another problem. As you can see - trouble with "Error in file" solved. :) WBR, Fyodor. ^ permalink raw reply [flat|nested] 7+ messages in thread
* rbd @ 2010-08-03 18:49 Yehuda Sadeh Weinraub 0 siblings, 0 replies; 7+ messages in thread From: Yehuda Sadeh Weinraub @ 2010-08-03 18:49 UTC (permalink / raw) To: Andrew Morton, Jens Axboe; +Cc: linux-kernel, linux-fsdevel, Sage Weil Hi Andrew, Jens, We've posted rbd a few times before. Rbd provides a block device that stripes data over ceph's block layer (rados). Other than a few trivial bug fixes it hasn't changed much from our last post, and the latest version can be found on the rbd branch at git://ceph.newdream.net/git/ceph-client.git. First, we'd like to know if it's acceptable to include the rbd block driver as part of the ceph module, or whether we need to restructure ceph so that it'd be possible to add rbd as a separate module. Second, is there any specific person's ACK or review we need in order to send this upstream with the rest of the ceph 2.6.36 queue? Thanks, Yehuda ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2011-05-14 12:36 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-05-12 10:53 rbd Fyodor Ustinov 2011-05-12 16:08 ` rbd Sage Weil 2011-05-12 17:01 ` rbd Sage Weil 2011-05-13 15:51 ` rbd Sage Weil 2011-05-13 19:11 ` rbd Sage Weil 2011-05-14 12:36 ` rbd Fyodor Ustinov -- strict thread matches above, loose matches on Subject: below -- 2010-08-03 18:49 rbd Yehuda Sadeh Weinraub
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.