Rbd map failure in 3.16.0-55

Rbd map failure in 3.16.0-55

[Varada Kari]

Hi all,

We are working on jewel branch on a test cluster to validate some of the fixes. But landed up in the following error when mapping an image using  krbd on Ubuntu 14.04.2 with 3.16.0-55 kernel version.

$ sudo rbd map -p pool1 rbd1
rbd: sysfs write failed
rbd: map failed: (5) Input/output error


$ uname -a
Linux 3.16.0-55-generic #74~14.04.1-Ubuntu SMP Tue Nov 17 10:15:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$dmesg
....
[11082.199006] libceph: read_partial_message bad hdr  crc 2112154322 != expected 0
[11082.209414] libceph: mon0 x.x.x.x:6789 socket error on read
[11092.238317] libceph: read_partial_message bad hdr  crc 2112154322 != expected 0
[11092.248982] libceph: mon0 x.x.x.x:6789 socket error on read
.....

When I looked at the ceph_msg_header on 3.16 kernel (http://lxr.free-electrons.com/source/include/linux/ceph/msgr.h?v=3.16)

There is one field change from jewel branch 

145 struct ceph_msg_header {
.......
159
160         /* oldest code we think can decode this.  unknown if zero. */
161         __le16 compat_version; <<< New one, which is present from 3.19 onwards
162         __le16 reserved;
163         __le32 crc;       /* header crc32c */
164 } __attribute__ ((packed));

How can map the image in 3.16 kernel apart from upgrading the kernel? Do we have any branch where I can build the modules with latest changes? 

Regards,
Varada

Re: Rbd map failure in 3.16.0-55

[Ilya Dryomov]

On Sat, Dec 12, 2015 at 7:56 AM, Varada Kari <Varada.Kari@sandisk.com> wrote:
> Hi all,
>
> We are working on jewel branch on a test cluster to validate some of the fixes. But landed up in the following error when mapping an image using  krbd on Ubuntu 14.04.2 with 3.16.0-55 kernel version.
>
> $ sudo rbd map -p pool1 rbd1
> rbd: sysfs write failed
> rbd: map failed: (5) Input/output error
>
>
> $ uname -a
> Linux 3.16.0-55-generic #74~14.04.1-Ubuntu SMP Tue Nov 17 10:15:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> $dmesg
> ....
> [11082.199006] libceph: read_partial_message bad hdr  crc 2112154322 != expected 0
> [11082.209414] libceph: mon0 x.x.x.x:6789 socket error on read
> [11092.238317] libceph: read_partial_message bad hdr  crc 2112154322 != expected 0
> [11092.248982] libceph: mon0 x.x.x.x:6789 socket error on read
> .....
>
> When I looked at the ceph_msg_header on 3.16 kernel (http://lxr.free-electrons.com/source/include/linux/ceph/msgr.h?v=3.16)
>
> There is one field change from jewel branch
>
> 145 struct ceph_msg_header {
> .......
> 159
> 160         /* oldest code we think can decode this.  unknown if zero. */
> 161         __le16 compat_version; <<< New one, which is present from 3.19 onwards
> 162         __le16 reserved;
> 163         __le32 crc;       /* header crc32c */
> 164 } __attribute__ ((packed));

That's not a problem - those 16 bits used to be reserved.

>
> How can map the image in 3.16 kernel apart from upgrading the kernel? Do we have any branch where I can build the modules with latest changes?

I'm guessing you have ms_crc_header set to false - the kernel client
always checks header (and middle) checksums, hence the mismatch.

Thanks,

                Ilya

RE: Rbd map failure in 3.16.0-55

[Varada Kari]

Thanks Ilya. Yes, we are setting the value to false. Will check by setting to true and get back. 
We are using this config option set to false in previous release as well, that working fine. Is there any change happened to make it mandatory? 

Varada

> -----Original Message-----
> From: Ilya Dryomov [mailto:idryomov@gmail.com]
> Sent: Saturday, December 12, 2015 4:42 PM
> To: Varada Kari <Varada.Kari@sandisk.com>
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: Rbd map failure in 3.16.0-55
> 
> On Sat, Dec 12, 2015 at 7:56 AM, Varada Kari <Varada.Kari@sandisk.com>
> wrote:
> > Hi all,
> >
> > We are working on jewel branch on a test cluster to validate some of the
> fixes. But landed up in the following error when mapping an image using
> krbd on Ubuntu 14.04.2 with 3.16.0-55 kernel version.
> >
> > $ sudo rbd map -p pool1 rbd1
> > rbd: sysfs write failed
> > rbd: map failed: (5) Input/output error
> >
> >
> > $ uname -a
> > Linux 3.16.0-55-generic #74~14.04.1-Ubuntu SMP Tue Nov 17 10:15:59 UTC
> > 2015 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $dmesg
> > ....
> > [11082.199006] libceph: read_partial_message bad hdr  crc 2112154322
> > != expected 0 [11082.209414] libceph: mon0 x.x.x.x:6789 socket error
> > on read [11092.238317] libceph: read_partial_message bad hdr  crc
> > 2112154322 != expected 0 [11092.248982] libceph: mon0 x.x.x.x:6789
> > socket error on read .....
> >
> > When I looked at the ceph_msg_header on 3.16 kernel
> > (http://lxr.free-electrons.com/source/include/linux/ceph/msgr.h?v=3.16
> > )
> >
> > There is one field change from jewel branch
> >
> > 145 struct ceph_msg_header {
> > .......
> > 159
> > 160         /* oldest code we think can decode this.  unknown if zero. */
> > 161         __le16 compat_version; <<< New one, which is present from 3.19
> onwards
> > 162         __le16 reserved;
> > 163         __le32 crc;       /* header crc32c */
> > 164 } __attribute__ ((packed));
> 
> That's not a problem - those 16 bits used to be reserved.
> 
> >
> > How can map the image in 3.16 kernel apart from upgrading the kernel? Do
> we have any branch where I can build the modules with latest changes?
> 
> I'm guessing you have ms_crc_header set to false - the kernel client always
> checks header (and middle) checksums, hence the mismatch.
> 
> Thanks,
> 
>                 Ilya

RE: Rbd map failure in 3.16.0-55

[Somnath Roy]

Ilya,
If we map with 'nocrc' would that help ?

Thanks & Regards
Somnath

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Ilya Dryomov
Sent: Saturday, December 12, 2015 3:12 AM
To: Varada Kari
Cc: ceph-devel@vger.kernel.org
Subject: Re: Rbd map failure in 3.16.0-55

On Sat, Dec 12, 2015 at 7:56 AM, Varada Kari <Varada.Kari@sandisk.com> wrote:
> Hi all,
>
> We are working on jewel branch on a test cluster to validate some of the fixes. But landed up in the following error when mapping an image using  krbd on Ubuntu 14.04.2 with 3.16.0-55 kernel version.
>
> $ sudo rbd map -p pool1 rbd1
> rbd: sysfs write failed
> rbd: map failed: (5) Input/output error
>
>
> $ uname -a
> Linux 3.16.0-55-generic #74~14.04.1-Ubuntu SMP Tue Nov 17 10:15:59 UTC 
> 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> $dmesg
> ....
> [11082.199006] libceph: read_partial_message bad hdr  crc 2112154322 
> != expected 0 [11082.209414] libceph: mon0 x.x.x.x:6789 socket error 
> on read [11092.238317] libceph: read_partial_message bad hdr  crc 
> 2112154322 != expected 0 [11092.248982] libceph: mon0 x.x.x.x:6789 
> socket error on read .....
>
> When I looked at the ceph_msg_header on 3.16 kernel 
> (http://lxr.free-electrons.com/source/include/linux/ceph/msgr.h?v=3.16
> )
>
> There is one field change from jewel branch
>
> 145 struct ceph_msg_header {
> .......
> 159
> 160         /* oldest code we think can decode this.  unknown if zero. */
> 161         __le16 compat_version; <<< New one, which is present from 3.19 onwards
> 162         __le16 reserved;
> 163         __le32 crc;       /* header crc32c */
> 164 } __attribute__ ((packed));

That's not a problem - those 16 bits used to be reserved.

>
> How can map the image in 3.16 kernel apart from upgrading the kernel? Do we have any branch where I can build the modules with latest changes?

I'm guessing you have ms_crc_header set to false - the kernel client always checks header (and middle) checksums, hence the mismatch.

Thanks,

                Ilya
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Rbd map failure in 3.16.0-55

[Ilya Dryomov]

On Sat, Dec 12, 2015 at 6:42 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
> Ilya,
> If we map with 'nocrc' would that help ?

No, it will disable data crcs, header and middle crcs will still be
checked.  The header/data separation in userspace is fairly new, if
that's something you care about, I can port it for 4.5 - it should be
trivial enough.

Thanks,

                Ilya

Re: Rbd map failure in 3.16.0-55

[Somnath Roy]

Yeah, this happened post Giant..So far our testing was with disabled crc...we will check the impact with crc and let you know..Initially when we started it was bad but porting to new instruction set improved crc could time significantly..we r yet to test that out end-to-end..Also, having that crc enabled is recommended..we will come back for your help if it is really hurting performance..

Thanks 
Somnath

Sent from my iPhone

> On Dec 12, 2015, at 10:56 AM, Ilya Dryomov <idryomov@gmail.com> wrote:
> 
>> On Sat, Dec 12, 2015 at 6:42 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
>> Ilya,
>> If we map with 'nocrc' would that help ?
> 
> No, it will disable data crcs, header and middle crcs will still be
> checked.  The header/data separation in userspace is fairly new, if
> that's something you care about, I can port it for 4.5 - it should be
> trivial enough.
> 
> Thanks,
> 
>                Ilya