rbd

rbd

[Fyodor Ustinov]

Hi!

Latest (git pulled) version of 2.6 kernel. Ceph - 0.27.1

Still troubles with rbd. Now with ocfs2 no messages in syslog, but iozone 
still return error:

#df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             237G   15G  210G   7% /
none                  2.0G  164K  2.0G   1% /dev
none                  2.0G     0  2.0G   0% /dev/shm
none                  2.0G   52K  2.0G   1% /var/run
none                  2.0G     0  2.0G   0% /var/lock
/dev/rbd0            1000G  3.2G  997G   1% /mnt

# cd /mnt/
root@stb1:/mnt# ls
lost+found
root@stb1:/mnt# iozone -a -n4g -g20g
        Iozone: Performance Test of File I/O
                Version $Revision: 3.373 $
                Compiled for 64 bit mode.
                Build: linux-AMD64

[...]

        Run began: Thu May 12 13:42:45 2011

        Auto Mode
        Using minimum file size of 4194304 kilobytes.
        Using maximum file size of 20971520 kilobytes.
        Command line used: iozone -a -n4g -g20g
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random  random    
bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    
read  rewrite     read   fwrite frewrite   fread  freread
         4194304      64  146906  150779

Error in file: Found ?7f24fb70a210? Expecting ?3a3a3a3a3a3a3a3a? addr 
7fe5b720e000
Error in file: Position 2481971200
Record # 37871 Record size 64 kb
where 7fe5b720e000 loop 57344
root@stb1:/mnt#           

Can I help to catch this error?

WBR,
    Fyodor.

Re: rbd

[Sage Weil]

Hi Fyodor,

> Hi!
> 
> Latest (git pulled) version of 2.6 kernel. Ceph - 0.27.1
> 
> Still troubles with rbd. Now with ocfs2 no messages in syslog, but iozone 
> still return error:
> 
> #df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda1             237G   15G  210G   7% /
> none                  2.0G  164K  2.0G   1% /dev
> none                  2.0G     0  2.0G   0% /dev/shm
> none                  2.0G   52K  2.0G   1% /var/run
> none                  2.0G     0  2.0G   0% /var/lock
> /dev/rbd0            1000G  3.2G  997G   1% /mnt
> 
> # cd /mnt/
> root@stb1:/mnt# ls
> lost+found
> root@stb1:/mnt# iozone -a -n4g -g20g
>         Iozone: Performance Test of File I/O
>                 Version $Revision: 3.373 $
>                 Compiled for 64 bit mode.
>                 Build: linux-AMD64
> 
> [...]
> 
>         Run began: Thu May 12 13:42:45 2011
> 
>         Auto Mode
>         Using minimum file size of 4194304 kilobytes.
>         Using maximum file size of 20971520 kilobytes.
>         Command line used: iozone -a -n4g -g20g
>         Output is in Kbytes/sec
>         Time Resolution = 0.000001 seconds.
>         Processor cache size set to 1024 Kbytes.
>         Processor cache line size set to 32 bytes.
>         File stride size set to 17 * record size.
>                                                             random  random    
> bkwd   record   stride
>               KB  reclen   write rewrite    read    reread    read   write    
> read  rewrite     read   fwrite frewrite   fread  freread
>          4194304      64  146906  150779
> 
> Error in file: Found ?7f24fb70a210? Expecting ?3a3a3a3a3a3a3a3a? addr 
> 7fe5b720e000
> Error in file: Position 2481971200
> Record # 37871 Record size 64 kb
> where 7fe5b720e000 loop 57344
> root@stb1:/mnt#           
> 
> Can I help to catch this error?

This one is going to be a bit difficult because there are several layers 
(ocfs2, rbd client, osd) involved.  It's presumably not ocfs2's fault, but 
trying the same thing with ext4 or something would remove that from the 
equation.  I'm running iozone on ext3 now and not having any problems.

Assuming the problem is in rbd, the question is whether it's the client or 
server side.  That will be harder to diagnose, and will probably involve 
careful use of fiemap/bmap to map the file offset to a block device 
offset, which we can then translate into a rados block.  That can be 
pulled directly out of the object store to confirm it has the right 
content...

sage

Re: rbd

[Sage Weil]

On Thu, 12 May 2011, Sage Weil wrote:
> equation.  I'm running iozone on ext3 now and not having any problems.

I take it back.. I just reproduced a similar error on ext2:

                                                            random  random    
bkwd   record   stride                                   
              KB  reclen   write rewrite    read    reread    read   write    
read  rewrite     read   fwrite frewrite   fread  freread
         4194304      64    3570    3040    12640    13725

Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr 
40a00000
Error in file: Position 2813329408 
Record # 42928 Record size 64 kb 
where 40a00000 loop 0

I opened a tracker issue http://tracker.newdream.net/issues/1086

sage

Re: rbd

[Sage Weil]

On Thu, 12 May 2011, Sage Weil wrote:
> On Thu, 12 May 2011, Sage Weil wrote:
> > equation.  I'm running iozone on ext3 now and not having any problems.
> 
> I take it back.. I just reproduced a similar error on ext2:
> 
>                                                             random  random    
> bkwd   record   stride                                   
>               KB  reclen   write rewrite    read    reread    read   write    
> read  rewrite     read   fwrite frewrite   fread  freread
>          4194304      64    3570    3040    12640    13725
> 
> Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr 
> 40a00000
> Error in file: Position 2813329408 
> Record # 42928 Record size 64 kb 
> where 40a00000 loop 0
> 
> I opened a tracker issue http://tracker.newdream.net/issues/1086

Just an update: we've identified the problem but a fix is still in the 
works.  The bio coming down is spanning an object boundary, and we're 
having trouble getting the block layer to either not do that or to handle 
the bio splitting properly.

sage

Re: rbd

[Sage Weil]

Fyodor,

On Fri, 13 May 2011, Sage Weil wrote:
> On Thu, 12 May 2011, Sage Weil wrote:
> > On Thu, 12 May 2011, Sage Weil wrote:
> > > equation.  I'm running iozone on ext3 now and not having any problems.
> > 
> > I take it back.. I just reproduced a similar error on ext2:
> > 
> >                                                             random  random    
> > bkwd   record   stride                                   
> >               KB  reclen   write rewrite    read    reread    read   write    
> > read  rewrite     read   fwrite frewrite   fread  freread
> >          4194304      64    3570    3040    12640    13725
> > 
> > Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr 
> > 40a00000
> > Error in file: Position 2813329408 
> > Record # 42928 Record size 64 kb 
> > where 40a00000 loop 0
> > 
> > I opened a tracker issue http://tracker.newdream.net/issues/1086
> 
> Just an update: we've identified the problem but a fix is still in the 
> works.  The bio coming down is spanning an object boundary, and we're 
> having trouble getting the block layer to either not do that or to handle 
> the bio splitting properly.

Can you try the req_coll branch of 
git://ceph.newdream.net/git/ceph-client.git and see if that fixes things 
for you?  There's a memory leak fix and a fix for the split bio completion 
stuff.  If it checks out I want to send this to Linus quickly so it'll be 
included in 2.6.39 (which is imminent).

Thanks!
sage

Re: rbd

[Fyodor Ustinov]

On 05/13/2011 10:11 PM, Sage Weil wrote:
> Fyodor,
>
> On Fri, 13 May 2011, Sage Weil wrote:
>> On Thu, 12 May 2011, Sage Weil wrote:
>>> On Thu, 12 May 2011, Sage Weil wrote:
>>>> equation.  I'm running iozone on ext3 now and not having any problems.
>>> I take it back.. I just reproduced a similar error on ext2:
>>>
>>>                                                              random  random
>>> bkwd   record   stride
>>>                KB  reclen   write rewrite    read    reread    read   write
>>> read  rewrite     read   fwrite frewrite   fread  freread
>>>           4194304      64    3570    3040    12640    13725
>>>
>>> Error in file: Found ?aaaaaaaaaaaaaaaa? Expecting ?3838383838383838? addr
>>> 40a00000
>>> Error in file: Position 2813329408
>>> Record # 42928 Record size 64 kb
>>> where 40a00000 loop 0
>>>
>>> I opened a tracker issue http://tracker.newdream.net/issues/1086
>> Just an update: we've identified the problem but a fix is still in the
>> works.  The bio coming down is spanning an object boundary, and we're
>> having trouble getting the block layer to either not do that or to handle
>> the bio splitting properly.
> Can you try the req_coll branch of
> git://ceph.newdream.net/git/ceph-client.git and see if that fixes things
> for you?  There's a memory leak fix and a fix for the split bio completion
> stuff.  If it checks out I want to send this to Linus quickly so it'll be
> included in 2.6.39 (which is imminent).
root@stb1:~# uname -a
Linux stb1 2.6.39-rc7-ufm-ceph #1 SMP Sat May 14 13:50:12 EEST 2011 
x86_64 x86_64 x86_64 GNU/Linux
root@stb1:~# modprobe rbd
root@stb1:~# echo "77.120.112.193 name=admin rbd tmt" > /sys/bus/rbd/add
root@stb1:~# mkfs.ocfs2 -M local -T vmstore --fs-features=local,sparse 
/dev/rbd0
mkfs.ocfs2 1.6.3
Overwriting existing ocfs2 partition.
Proceed (y/N): y
Filesystem Type of vmstore
Label:
Features: local sparse backup-super unwritten inline-data 
strict-journal-super xattr refcount
Block size: 4096 (12 bits)
Cluster size: 1048576 (20 bits)
Volume size: 1073741824000 (1024000 clusters) (262144000 blocks)
Cluster groups: 32 (tail covers 24064 clusters, rest cover 32256 clusters)
Extent allocator size: 3221225472 (768 groups)
Journal size: 134217728
Node slots: 1
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 5 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Formatting quota files: done
Writing lost+found: done
mkfs.ocfs2 successful

root@stb1:~# mount /dev/rbd0 /mnt
root@stb1:~# cd /mnt
root@stb1:/mnt# iozone -a -n4g -g20g
         Iozone: Performance Test of File I/O
                 Version $Revision: 3.373 $
                 Compiled for 64 bit mode.
                 Build: linux-AMD64

         Contributors:William Norcott, Don Capps, Isom Crawford, Kirby 
Collins
                      Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                      Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                      Randy Dunlap, Mark Montague, Dan Million, Gavin 
Brebner,
                      Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, 
Dave Boone,
                      Erik Habbinga, Kris Strecker, Walter Wong, Joshua 
Root,
                      Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren 
Sawyer.

         Run began: Sat May 14 14:39:03 2011

         Auto Mode
         Using minimum file size of 4194304 kilobytes.
         Using maximum file size of 20971520 kilobytes.
         Command line used: iozone -a -n4g -g20g
         Output is in Kbytes/sec
         Time Resolution = 0.000001 seconds.
         Processor cache size set to 1024 Kbytes.
         Processor cache line size set to 32 bytes.
         File stride size set to 17 * record size.
                                                             random  
random    bkwd   record   stride
               KB  reclen   write rewrite    read    reread    read   
write    read  rewrite     read   fwrite frewrite   fread  freread
          4194304      64  134566  135254    53718    52836   14839   78946

And now it is 20 minutes in this state.In the logs clean. Incoming 
network traffic- 3Mbytes/s
But, we newer see this test before, likely an extremely low speed test 
"bkwd read" this is another problem.

As you can see - trouble with "Error in file" solved. :)

WBR,
     Fyodor.

rbd

[Yehuda Sadeh Weinraub]

Hi Andrew, Jens,

  We've posted rbd a few times before. Rbd provides a block device
that stripes data over ceph's block layer (rados). Other than a few
trivial bug fixes it hasn't changed much from our last post, and the
latest version can be found on the rbd branch at
git://ceph.newdream.net/git/ceph-client.git.

First, we'd like to know if it's acceptable to include the rbd block
driver as part of the ceph module, or whether we need to restructure
ceph so that it'd be possible to add rbd as a separate module.

Second, is there any specific person's ACK or review we need in order
to send this upstream with the rest of the ceph 2.6.36 queue?

Thanks,
Yehuda