[PATCH] rds: Error on offset mismatch if not loopback

[PATCH] rds: Error on offset mismatch if not loopback

[John Jolly]

Attempting an rds connection from the IP address of an IPoIB interface
to itself causes a kernel panic due to a BUG_ON() being triggered. Making
the test less strict allows rds-ping to work without crashing the machine.

A local unprivileged user could use this flaw to crash the sytem.
---
 net/rds/ib_send.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c
index e590949..7920c85 100644
--- a/net/rds/ib_send.c
+++ b/net/rds/ib_send.c
@@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm,
 	int flow_controlled = 0;
 	int nr_sig = 0;
 
-	BUG_ON(off % RDS_FRAG_SIZE);
+	BUG_ON(!conn->c_loopback && off % RDS_FRAG_SIZE);
 	BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));
 
 	/* Do not send cong updates to IB loopback */
-- 
1.7.7

Re: [PATCH] rds: Error on offset mismatch if not loopback

[David Miller]

From: John Jolly <jjolly@suse.com>
Date: Thu, 20 Sep 2012 01:11:34 -0600

> Attempting an rds connection from the IP address of an IPoIB interface
> to itself causes a kernel panic due to a BUG_ON() being triggered. Making
> the test less strict allows rds-ping to work without crashing the machine.
> 
> A local unprivileged user could use this flaw to crash the sytem.

Please read Documentation/SubmittingPatches to learn how to properly
submit a change, in particular your patch submission was missing a
proper signoff.

Thanks.

Re: [PATCH] rds: Error on offset mismatch if not loopback

[John Jolly]

On Fri, Sep 21, 2012 at 01:20:45PM -0400, David Miller wrote:
> From: John Jolly <jjolly@suse.com>
> Date: Thu, 20 Sep 2012 01:11:34 -0600
> 
> > Attempting an rds connection from the IP address of an IPoIB interface
> > to itself causes a kernel panic due to a BUG_ON() being triggered. Making
> > the test less strict allows rds-ping to work without crashing the machine.
> > 
> > A local unprivileged user could use this flaw to crash the sytem.
> 
> Please read Documentation/SubmittingPatches to learn how to properly
> submit a change, in particular your patch submission was missing a
> proper signoff.

Thanks for catching that. Resubmitting with proper signoff.

> 
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

RE: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

> Why are you posting this message a second time?

Reposting just the contents of the second message in case it got missed the previous time.

Looks like the fix pointed to by the previous link is for a panic on a PPC system with a PAGE_SIZE of 64Kbytes.
I think the sequence it was going through before that fix was:
/* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
        }
rds_ib_xmit returns 8240
rds_send_xmit : c_xmit_data_off = 0 + 8240 - 48 (rds header the first time) = 8196
                c_xmit_data_off < 65536 (sg->length)
                calls rds_ib_xmit again
rds_ib_xmit returns 8240
rds_send_xmit: c_xmit_data_off = 8192+8240 = 16432 and calls rds_ib_xmit
rds_ib_xmit : returns 8240
rds_send_xmit: c_xmit_data_off 24672 and calls rds_ib_xmit ...
...
and so on till
rds_send_xmit: c_xmit_data_off 57632 and calls rds_ib_xmit
rds_ib_xmit: returns 8240

On the last iteration it hits the below BUG_ON in rds_send_xmit.
while (ret) {
    tmp = min_t(int, ret, sg->length -
                         conn->c_xmit_data_off);
 [tmp = 7904]
    conn->c_xmit_data_off += tmp;
[c_xmit_data_off = 65536]
    ret -= tmp;
[ret = 8240-7904 = 336]
    if (conn->c_xmit_data_off == sg->length) {
         conn->c_xmit_data_off = 0;
         sg++;
         conn->c_xmit_sg++;
         BUG_ON(ret != 0 &&
             conn->c_xmit_sg == rm->data.op_nents);
    }
}

Since the congestion update over loopback is not actually transmitted as a message,
the multiple iterations we see in the case of ppc is unnecessary.
All that rds_ib_xmit needs to do is return a number of bytes that will tell the caller
that we are done with this message.
  
This might fix the original problem without introducing the current panic:
/* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                scat = &rm->data.op_sg[sg];
                ret = max_t(int, RDS_CONG_MAP_BYTES, scat->length);
                return ret + sizeof(struct rds_header);
        }
It will return 8240 when PAGE_SIZE is 4k and 64k+48 in case of ppc when scat->length is 64k and 
be done with one iteration of rds_send_xmit/rds_ib_xmit loop.

Venkat

Re: [PATCH] rds: Error on offset mismatch if not loopback

[David Miller]

Why are you posting this message a second time?

RE: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

-----Original Message-----
From: Venkat Venkatsubra 
Sent: Tuesday, November 19, 2013 5:33 PM
To: Honggang LI; Josh Hunt
Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org
Subject: RE: [PATCH] rds: Error on offset mismatch if not loopback

We now have lot more information than we did before.
When sending a "congestion update" in rds_ib_xmit() we are now returning an incorrect number as bytes sent:

        BUG_ON(off % RDS_FRAG_SIZE);
        BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));

        /* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                scat = &rm->data.op_sg[sg];
                ret = sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
                ret = min_t(int, ret, scat->length - conn->c_xmit_data_off);
                return ret;
        }

It returns min(8240, 4096-0) i.e. 4096 bytes.
The caller rds_send_xmit() is made to think a partial message (4096 out of 8240) was sent.
It calls rds_ib_xmit() again with a data offset "off" of 4096-48 (rds header) (=4048 bytes). And we hit the BUG_ON.

The reason I didn't hit the panic on my test on Oracle UEK2 which is based on 2.6.39 kernel is it had it like this:
        BUG_ON(off % RDS_FRAG_SIZE);
        BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));

        /* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
        }
(So it wasn't 100% 2.6.39 ;-). )
It returned 8240 bytes. The caller rds_send_xmit decides the full message was sent (48 byte header + 4096 data + 4096 data).
And it worked.

Then I found this info on the change that was done upstream which now causes the panic:
http://marc.info/?l=linux-netdev&m=129908332903057
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6094628bfd94323fc1cea05ec2c6affd98c18f7f 

Will investigate more into which problem the above change addressed.

Venkat
--

Looks like the fix pointed to by the above link is for a panic on a PPC system with a PAGE_SIZE of 64Kbytes.
I think the sequence it was going through before that fix was:
/* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
        }
rds_ib_xmit returns 8240
rds_send_xmit : c_xmit_data_off = 0 + 8240 - 48 (rds header the first time) = 8196
                c_xmit_data_off < 65536 (sg->length)
                calls rds_ib_xmit again
rds_ib_xmit returns 8240
rds_send_xmit: c_xmit_data_off = 8192+8240 = 16432 and calls rds_ib_xmit
rds_ib_xmit : returns 8240
rds_send_xmit: c_xmit_data_off 24672 and calls rds_ib_xmit
...
...
and so on till
rds_send_xmit: c_xmit_data_off 57632 and calls rds_ib_xmit
rds_ib_xmit: returns 8240

On the last iteration it hits the below BUG_ON in rds_send_xmit.
while (ret) {
    tmp = min_t(int, ret, sg->length -
                         conn->c_xmit_data_off);
[tmp = 7904]
    conn->c_xmit_data_off += tmp;
[c_xmit_data_off = 65536]
    ret -= tmp;
[ret = 8240-7904 = 336]
    if (conn->c_xmit_data_off == sg->length) {
         conn->c_xmit_data_off = 0;
         sg++;
         conn->c_xmit_sg++;
         BUG_ON(ret != 0 &&
             conn->c_xmit_sg == rm->data.op_nents);
    }
}

Since the congestion update over loopback is not actually transmitted as a message,
the multiple iterations we see in the case of ppc is unnecessary.
All that rds_ib_xmit needs to do is return a number of bytes that will tell the caller that
we are done with this message.
  
This might fix the original problem without introducing the current panic:
/* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                scat = &rm->data.op_sg[sg];
                ret = max_t(int, RDS_CONG_MAP_BYTES, scat->length);
                return ret + sizeof(struct rds_header);
        }
It will return 8240 when PAGE_SIZE is 4k and 64k+48 in case of ppc when scat->length is 64k and
be done with one iteration of rds_send_xmit/rds_ib_xmit loop.

Venkat

RE: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

We now have lot more information than we did before.
When sending a "congestion update" in rds_ib_xmit() we are now returning an incorrect number as bytes sent:

        BUG_ON(off % RDS_FRAG_SIZE);
        BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));

        /* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                scat = &rm->data.op_sg[sg];
                ret = sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
                ret = min_t(int, ret, scat->length - conn->c_xmit_data_off);
                return ret;
        }

It returns min(8240, 4096-0) i.e. 4096 bytes.
The caller rds_send_xmit() is made to think a partial message (4096 out of 8240) was sent.
It calls rds_ib_xmit() again with a data offset "off" of 4096-48 (rds header) (=4048 bytes). And we hit the BUG_ON.

The reason I didn't hit the panic on my test on Oracle UEK2 which is based on 2.6.39 kernel is it had it like this:
        BUG_ON(off % RDS_FRAG_SIZE);
        BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));

        /* Do not send cong updates to IB loopback */
        if (conn->c_loopback
            && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) {
                rds_cong_map_updated(conn->c_fcong, ~(u64) 0);
                return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES;
        }
(So it wasn't 100% 2.6.39 ;-). )
It returned 8240 bytes. The caller rds_send_xmit decides the full message was sent (48 byte header + 4096 data + 4096 data).
And it worked.

Then I found this info on the change that was done upstream which now causes the panic:
http://marc.info/?l=linux-netdev&m=129908332903057
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6094628bfd94323fc1cea05ec2c6affd98c18f7f 

Will investigate more into which problem the above change addressed.

Venkat

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Honggang LI]

[-- Attachment #1: Type: text/plain, Size: 8977 bytes --]

On 11/14/2013 09:43 PM, Venkat Venkatsubra wrote:
>
> -----Original Message-----
> From: Honggang LI [mailto:honli@redhat.com] 
> Sent: Wednesday, November 13, 2013 6:56 PM
> To: Josh Hunt; Venkat Venkatsubra
> Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org
> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>
> On 11/14/2013 01:40 AM, Josh Hunt wrote:
>> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra 
>> <venkat.x.venkatsubra@oracle.com> wrote:
>>> -----Original Message-----
>>> From: Josh Hunt [mailto:joshhunt00@gmail.com]
>>> Sent: Tuesday, November 12, 2013 10:25 PM
>>> To: David Miller
>>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
>>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>>>
>>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
>>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>>>> From: John Jolly <jjolly@suse.com>
>>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>>>>
>>>>>> Attempting an rds connection from the IP address of an IPoIB 
>>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>>>>>> Making the test less strict allows rds-ping to work without 
>>>>>> crashing the machine.
>>>>>>
>>>>>> A local unprivileged user could use this flaw to crash the system.
>>>>>>
>>>>>> Signed-off-by: John Jolly <jjolly@suse.com>
>>>>> Besides the questions being asked of you by Venkat Venkatsubra, 
>>>>> this patch has another issue.
>>>>>
>>>>> It has been completely corrupted by your email client, it has 
>>>>> turned all TAB characters into spaces, making the patch useless.
>>>>>
>>>>> Please learn how to send a patch unmolested in the body of your 
>>>>> email.  Test it by emailing the patch to yourself, and verifying 
>>>>> that you can in fact apply the patch you receive in that email.
>>>>> Then, and only then, should you consider making a new submission of 
>>>>> this patch.
>>>>>
>>>>> Use Documentation/email-clients.txt for guidance.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org 
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>> I think this issue was lost in the shuffle. It appears that redhat, 
>>>> ubuntu, and oracle are maintaining local patches to resolve this:
>>>>
>>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636
>>>> 85
>>>> 2be130fa15fa8be10d4704e8
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td498
>>>> 53
>>>> 88.html
>>>>
>>>> Given that Oracle has applied it I'll make the assumption that 
>>>> Venkat's question was answered at some point.
>>>>
>>>> David - I can resubmit the patch with the proper signed-off-by and 
>>>> formatting if you are willing to apply it unless John wants to try 
>>>> again. I think it's time this got upstream.
>>>>
>>>> --
>>>> Josh
>>> Ugh.. hopefully resending with all the html crap removed...
>>>
>>> --
>>> Josh
>>>
>>> Hi Josh,
>>>
>>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
>>> Because, rds-ping uses zero byte messages to ping.
>>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
>>> The Oracle's internal code I checked doesn't have that patch applied.
>>>
>>> Venkat
>> No I don't have a test case. I came across this CVE while doing an 
>> audit and noticed it was patched in Ubuntu's kernel and other distros, 
>> but was not in the upstream kernel yet. Quick googling of lkml showed 
>> that there were at least two attempts to get this patch upstream, but 
>> both had issues due to not following the proper submission process:
>>
>> https://lkml.org/lkml/2012/10/22/433
>> https://lkml.org/lkml/2012/9/21/505
>>
>> From my searching it appears the initial bug was found by someone at redhat:
>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>
>> I've added Li Honggang the reporter of this issue from Redhat to the 
>> mail. Hopefully he can share his testcase.
> The test case is very simple:
> Steps to Reproduce:
> 1. yum install -y rds-tools
>
> 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr'
>           inet addr:172.31.0.3  Bcast:172.31.0.255  Mask:255.255.255.0
>
> 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3  <<<< kernel panic (You may need to wait for a few seconds before the kernel panic.)
>> and possibly requires certain hardware as Jay writes in the first link above:
>> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..."
> This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko".
>
> As I know the upstream code of RDS is broken. There are *many* RDS bugs.
>
> Best regards.
> Honggang
>> I was referring to this oracle commit:
>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
>> 2be130fa15fa8be10d4704e8
>>
>> I have no experience with this code. There were a few comments around 
>> the reset and xmit fns about making sure the caller did certain things 
>> if not they were racy, but I have no idea if that's coming into play 
>> here.
>>
> Hi Honggang,
>
> I ran rds-ping over local interface for 30 minutes. I stopped it after that.
> It didn't hit any panic.
>
> # ip addr show dev ib0
> 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 1024
>     link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:21:28:00:01:cf:63:db brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
>     inet 10.196.4.125/30 brd 10.196.4.127 scope global ib0
>     inet6 fe80::221:2800:1cf:63db/64 scope link
>        valid_lft forever preferred_lft forever
> #
>
> # rds-ping  10.196.4.125
>     1: 170 usec
>     2: 171 usec 
>    ....
>    ....
>    ....
>  1860: 173 usec
>  1861: 171 usec
>  1862: 177 usec
>  1863: 168 usec
>  1864: 171 usec
>  1865: 175 usec
> ^C#
>
> I tested with Oracle UEK2 which is based on 2.6.39 kernel. Mellanox IB adaptor.
> 19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
>
> There is something about your setup that must be causing it for you.
> Can I work with you offline if you are available ?
>
> The panic you are hitting is not making sense to me.
>
> Venkat
Hi, Venkat
 It seems we are in different time zone. Please contact me via email if
you need I do something for this bug. Could you please try upstream
kernel 2.6.39. I confirmed that the bug can be reproduced with Mellanox
and QLogic HCA when running  upstream kernel-2.6.39.

[root@rdma01 ~]# ifconfig mlx4_ib1
Ifconfig uses the ioctl access method to get the full address
information, which limits hardware addresses to 8 bytes.
Because Infiniband address has 20 bytes, only the first 8 bytes are
displayed correctly.
Ifconfig is obsolete! For replacement check ip.
mlx4_ib1  Link encap:InfiniBand  HWaddr
80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 
          inet addr:172.31.2.1  Bcast:172.31.2.255  Mask:255.255.255.0
          inet6 addr: fe80::7ae7:d1ff:ff6b:b01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:5 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@rdma01 ~]# rpm -qf /usr/bin/rds-ping
rds-tools-2.0.6-3.el6.x86_64
[root@rdma01 ~]# uname -a
Linux rdma01.rhts.eng.nay.redhat.com 2.6.39 #1 SMP Thu Nov 14 20:25:45
EST 2013 x86_64 x86_64 x86_64 GNU/Linux
[root@rdma01 ~]# ibstat
CA 'mlx4_0'
    CA type: MT26428
    Number of ports: 2
    Firmware version: 2.8.600
    Hardware version: b0
    Node GUID: 0x78e7d1ffff6b0b00
    System image GUID: 0x78e7d1ffff6b0b03
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 40
        Base lid: 1
        LMC: 0
        SM lid: 4
        Capability mask: 0x02510868
        Port GUID: 0x78e7d1ffff6b0b01
        Link layer: InfiniBand
    Port 2:
        State: Down
        Physical state: Polling
        Rate: 70
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02510868
        Port GUID: 0x78e7d1ffff6b0b02
        Link layer: InfiniBand
[root@rdma01 ~]# lspci | grep Mellanox
1f:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
5GT/s - IB QDR / 10GigE] (rev b0)
[root@rdma01 ~]# ssh 172.31.2.2 hostname   (make sure the IPoIB
interface works)
rdma02.rhts.eng.nay.redhat.com
[root@rdma01 ~]# ssh 172.31.2.1 hostname
rdma01.rhts.eng.nay.redhat.com
[root@rdma01 ~]# /usr/bin/rds-ping 172.31.2.1 (kernel panic, please see
the attachment for console log)








[-- Attachment #2: upstream-kernel-2.6.39-rds-ping-panic.log --]
[-- Type: text/x-log, Size: 6544 bytes --]

RDS/IB: connected to 172.31.2.1 version 3.1
RDS/IB: connected to 172.31.2.1 version 3.1
------------[ cut here ]------------
kernel BUG at net/rds/ib_send.c:547!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 6 
Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp scsi_tgt rds_rdma rds_tcp rds ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode cdc_ether usbnet mii serio_raw pcspkr i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg shpchp ioatdma dca i7core_edac edac_core cxgb3 mdio mlx4_ib ib_mad mlx4_en mlx4_core ib_core cxgb4 bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 136, comm: kworker/u:1 Not tainted 2.6.39 #1 IBM System x3650 M3 -[7945O63]-/00D4062
RIP: 0010:[<ffffffffa036d7f9>]  [<ffffffffa036d7f9>] rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
RSP: 0018:ffff880271b51c50  EFLAGS: 00010202
RAX: ffff880266dc2000 RBX: ffff880271639a00 RCX: 0000000000000000
RDX: 0000000000000030 RSI: ffff880471e81000 RDI: ffff880270997cf0
RBP: ffff880271b51d30 R08: 0000000000000fd0 R09: ffff880471e81190
R10: 00000000ffffffff R11: 0000000000000001 R12: ffff880471e81000
R13: ffff880471e81000 R14: 0000000000000000 R15: ffff880271b51d90
FS:  0000000000000000(0000) GS:ffff88047fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b302d2aa080 CR3: 0000000001a03000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:1 (pid: 136, threadinfo ffff880271b50000, task ffff880271ae60c0)
Stack:
 0000000000000400 0000000000000001 0000000000000001 0000000a0000000a
 0000000000000128 ffffffff8114747c 0000000000000002 0000000000000001
 ffff880271ae60c0 ffffffff812402f0 ffff880471e81000 0000003000000002
Call Trace:
 [<ffffffff8114747c>] ? __kmalloc+0x21c/0x230
 [<ffffffff812402f0>] ? sg_init_table+0x30/0x50
 [<ffffffffa0340df2>] ? rds_message_alloc_sgs+0x62/0xa0 [rds]
 [<ffffffffa0341224>] ? rds_message_map_pages+0xa4/0x110 [rds]
 [<ffffffffa0342f5b>] rds_send_xmit+0x38b/0x6e0 [rds]
 [<ffffffffa0344010>] ? rds_recv_worker+0xc0/0xc0 [rds]
 [<ffffffffa0344045>] rds_send_worker+0x35/0xc0 [rds]
 [<ffffffff8107d0f9>] process_one_work+0x129/0x430
 [<ffffffff8107f4ab>] worker_thread+0x17b/0x3c0
 [<ffffffff8107f330>] ? manage_workers+0x120/0x120
 [<ffffffff81084566>] kthread+0x96/0xa0
 [<ffffffff814e4104>] kernel_thread_helper+0x4/0x10
 [<ffffffff810844d0>] ? kthread_worker_fn+0x1a0/0x1a0
 [<ffffffff814e4100>] ? gs_change+0x13/0x13
Code: ff ff e9 b1 fe ff ff 48 8b 0d c4 fe 69 e1 48 89 8d 70 ff ff ff e9 71 ff ff ff 83 bd 7c ff ff ff 00 0f 84 f4 f5 ff ff 0f 0b eb fe <0f> 0b eb fe 44 8b 8d 48 ff ff ff 41 b7 01 e9 51 f6 ff ff 0f 0b 
RIP  [<ffffffffa036d7f9>] rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
 RSP <ffff880271b51c50>
---[ end trace de7f8972e25cd611 ]---
BUG: unable to handle kernel paging request at fffffffffffffff8
IP: [<ffffffff810840c0>] kthread_data+0x10/0x20
PGD 1a05067 PUD 1a06067 PMD 0 
Oops: 0000 [#2] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 6 
Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp scsi_tgt rds_rdma rds_tcp rds ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode cdc_ether usbnet mii serio_raw pcspkr i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg shpchp ioatdma dca i7core_edac edac_core cxgb3 mdio mlx4_ib ib_mad mlx4_en mlx4_core ib_core cxgb4 bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 136, comm: kworker/u:1 Tainted: G      D     2.6.39 #1 IBM System x3650 M3 -[7945O63]-/00D4062
RIP: 0010:[<ffffffff810840c0>]  [<ffffffff810840c0>] kthread_data+0x10/0x20
RSP: 0018:ffff880271b51918  EFLAGS: 00010096
RAX: 0000000000000000 RBX: 0000000000000006 RCX: ffff880271ae60c0
RDX: 0000000000009bf5 RSI: 0000000000000006 RDI: ffff880271ae60c0
RBP: ffff880271b51918 R08: ffff880271ae6570 R09: dead000000200200
R10: 00000000ffffffff R11: 0000000000000007 R12: ffff880271ae6658
R13: 0000000000000006 R14: 0000000000000006 R15: 0000000000000006
FS:  0000000000000000(0000) GS:ffff88047fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: fffffffffffffff8 CR3: 0000000001a03000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:1 (pid: 136, threadinfo ffff880271b50000, task ffff880271ae60c0)
Stack:
 ffff880271b51938 ffffffff8107e385 ffff880272d67b40 ffff88047fc150c0
 ffff880271b519d8 ffffffff814d882e ffff880271b51978 ffff880271ae60c0
 ffff880271ae60c0 00000000000150c0 ffff880271b51fd8 ffff880271b50010
Call Trace:
 [<ffffffff8107e385>] wq_worker_sleeping+0x15/0xa0
 [<ffffffff814d882e>] schedule+0x49e/0x9c0
 [<ffffffff810675d1>] do_exit+0x271/0x430
 [<ffffffff814dc08b>] oops_end+0xab/0xf0
 [<ffffffff8100e7cb>] die+0x5b/0x90
 [<ffffffff814db994>] do_trap+0xc4/0x170
 [<ffffffff8100c695>] do_invalid_op+0x95/0xb0
 [<ffffffffa036d7f9>] ? rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
 [<ffffffff814e3f7b>] invalid_op+0x1b/0x20
 [<ffffffffa036d7f9>] ? rds_ib_xmit+0xa69/0xaf0 [rds_rdma]
 [<ffffffff8114747c>] ? __kmalloc+0x21c/0x230
 [<ffffffff812402f0>] ? sg_init_table+0x30/0x50
 [<ffffffffa0340df2>] ? rds_message_alloc_sgs+0x62/0xa0 [rds]
 [<ffffffffa0341224>] ? rds_message_map_pages+0xa4/0x110 [rds]
 [<ffffffffa0342f5b>] rds_send_xmit+0x38b/0x6e0 [rds]
 [<ffffffffa0344010>] ? rds_recv_worker+0xc0/0xc0 [rds]
 [<ffffffffa0344045>] rds_send_worker+0x35/0xc0 [rds]
 [<ffffffff8107d0f9>] process_one_work+0x129/0x430
 [<ffffffff8107f4ab>] worker_thread+0x17b/0x3c0
 [<ffffffff8107f330>] ? manage_workers+0x120/0x120
 [<ffffffff81084566>] kthread+0x96/0xa0
 [<ffffffff814e4104>] kernel_thread_helper+0x4/0x10
 [<ffffffff810844d0>] ? kthread_worker_fn+0x1a0/0x1a0
 [<ffffffff814e4100>] ? gs_change+0x13/0x13
Code: 1f 44 00 00 65 48 8b 04 25 80 cc 00 00 48 8b 80 40 05 00 00 8b 40 f0 c9 c3 66 90 55 48 89 e5 0f 1f 44 00 00 48 8b 87 40 05 00 00 
 8b 40 f8 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 
RIP  [<ffffffff810840c0>] kthread_data+0x10/0x20
 RSP <ffff880271b51918>
CR2: fffffffffffffff8
---[ end trace de7f8972e25cd612 ]---
Fixing recursive fault but reboot is needed!

RE: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

-----Original Message-----
From: Honggang LI [mailto:honli@redhat.com] 
Sent: Wednesday, November 13, 2013 6:56 PM
To: Josh Hunt; Venkat Venkatsubra
Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org
Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback

On 11/14/2013 01:40 AM, Josh Hunt wrote:
> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra 
> <venkat.x.venkatsubra@oracle.com> wrote:
>>
>> -----Original Message-----
>> From: Josh Hunt [mailto:joshhunt00@gmail.com]
>> Sent: Tuesday, November 12, 2013 10:25 PM
>> To: David Miller
>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>>
>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>>> From: John Jolly <jjolly@suse.com>
>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>>>
>>>>> Attempting an rds connection from the IP address of an IPoIB 
>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>>>>> Making the test less strict allows rds-ping to work without 
>>>>> crashing the machine.
>>>>>
>>>>> A local unprivileged user could use this flaw to crash the system.
>>>>>
>>>>> Signed-off-by: John Jolly <jjolly@suse.com>
>>>> Besides the questions being asked of you by Venkat Venkatsubra, 
>>>> this patch has another issue.
>>>>
>>>> It has been completely corrupted by your email client, it has 
>>>> turned all TAB characters into spaces, making the patch useless.
>>>>
>>>> Please learn how to send a patch unmolested in the body of your 
>>>> email.  Test it by emailing the patch to yourself, and verifying 
>>>> that you can in fact apply the patch you receive in that email.
>>>> Then, and only then, should you consider making a new submission of 
>>>> this patch.
>>>>
>>>> Use Documentation/email-clients.txt for guidance.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe 
>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org 
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>> I think this issue was lost in the shuffle. It appears that redhat, 
>>> ubuntu, and oracle are maintaining local patches to resolve this:
>>>
>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636
>>> 85
>>> 2be130fa15fa8be10d4704e8
>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td498
>>> 53
>>> 88.html
>>>
>>> Given that Oracle has applied it I'll make the assumption that 
>>> Venkat's question was answered at some point.
>>>
>>> David - I can resubmit the patch with the proper signed-off-by and 
>>> formatting if you are willing to apply it unless John wants to try 
>>> again. I think it's time this got upstream.
>>>
>>> --
>>> Josh
>> Ugh.. hopefully resending with all the html crap removed...
>>
>> --
>> Josh
>>
>> Hi Josh,
>>
>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
>> Because, rds-ping uses zero byte messages to ping.
>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
>> The Oracle's internal code I checked doesn't have that patch applied.
>>
>> Venkat
> No I don't have a test case. I came across this CVE while doing an 
> audit and noticed it was patched in Ubuntu's kernel and other distros, 
> but was not in the upstream kernel yet. Quick googling of lkml showed 
> that there were at least two attempts to get this patch upstream, but 
> both had issues due to not following the proper submission process:
>
> https://lkml.org/lkml/2012/10/22/433
> https://lkml.org/lkml/2012/9/21/505
>
> From my searching it appears the initial bug was found by someone at redhat:
> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>
> I've added Li Honggang the reporter of this issue from Redhat to the 
> mail. Hopefully he can share his testcase.
The test case is very simple:
Steps to Reproduce:
1. yum install -y rds-tools

2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr'
          inet addr:172.31.0.3  Bcast:172.31.0.255  Mask:255.255.255.0

3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3  <<<< kernel panic (You may need to wait for a few seconds before the kernel panic.)
>
> and possibly requires certain hardware as Jay writes in the first link above:
> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..."
This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko".

As I know the upstream code of RDS is broken. There are *many* RDS bugs.

Best regards.
Honggang
>
> I was referring to this oracle commit:
> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
> 2be130fa15fa8be10d4704e8
>
> I have no experience with this code. There were a few comments around 
> the reset and xmit fns about making sure the caller did certain things 
> if not they were racy, but I have no idea if that's coming into play 
> here.
>

Hi Honggang,

I ran rds-ping over local interface for 30 minutes. I stopped it after that.
It didn't hit any panic.

# ip addr show dev ib0
6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 1024
    link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:21:28:00:01:cf:63:db brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 10.196.4.125/30 brd 10.196.4.127 scope global ib0
    inet6 fe80::221:2800:1cf:63db/64 scope link
       valid_lft forever preferred_lft forever
#

# rds-ping  10.196.4.125
    1: 170 usec
    2: 171 usec 
   ....
   ....
   ....
 1860: 173 usec
 1861: 171 usec
 1862: 177 usec
 1863: 168 usec
 1864: 171 usec
 1865: 175 usec
^C#

I tested with Oracle UEK2 which is based on 2.6.39 kernel. Mellanox IB adaptor.
19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)

There is something about your setup that must be causing it for you.
Can I work with you offline if you are available ?

The panic you are hitting is not making sense to me.

Venkat

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Josh Hunt]

On Wed, Nov 13, 2013 at 6:55 PM, Honggang LI <honli@redhat.com> wrote:
> On 11/14/2013 01:40 AM, Josh Hunt wrote:
>> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra
>> <venkat.x.venkatsubra@oracle.com> wrote:
>>>
>>> -----Original Message-----
>>> From: Josh Hunt [mailto:joshhunt00@gmail.com]
>>> Sent: Tuesday, November 12, 2013 10:25 PM
>>> To: David Miller
>>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
>>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>>>
>>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
>>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>>>> From: John Jolly <jjolly@suse.com>
>>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>>>>
>>>>>> Attempting an rds connection from the IP address of an IPoIB
>>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>>>>>> Making the test less strict allows rds-ping to work without
>>>>>> crashing the machine.
>>>>>>
>>>>>> A local unprivileged user could use this flaw to crash the system.
>>>>>>
>>>>>> Signed-off-by: John Jolly <jjolly@suse.com>
>>>>> Besides the questions being asked of you by Venkat Venkatsubra, this
>>>>> patch has another issue.
>>>>>
>>>>> It has been completely corrupted by your email client, it has turned
>>>>> all TAB characters into spaces, making the patch useless.
>>>>>
>>>>> Please learn how to send a patch unmolested in the body of your
>>>>> email.  Test it by emailing the patch to yourself, and verifying that
>>>>> you can in fact apply the patch you receive in that email.
>>>>> Then, and only then, should you consider making a new submission of
>>>>> this patch.
>>>>>
>>>>> Use Documentation/email-clients.txt for guidance.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>>
>>>> I think this issue was lost in the shuffle. It appears that redhat,
>>>> ubuntu, and oracle are maintaining local patches to resolve this:
>>>>
>>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
>>>> 2be130fa15fa8be10d4704e8
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853
>>>> 88.html
>>>>
>>>> Given that Oracle has applied it I'll make the assumption that
>>>> Venkat's question was answered at some point.
>>>>
>>>> David - I can resubmit the patch with the proper signed-off-by and
>>>> formatting if you are willing to apply it unless John wants to try
>>>> again. I think it's time this got upstream.
>>>>
>>>> --
>>>> Josh
>>> Ugh.. hopefully resending with all the html crap removed...
>>>
>>> --
>>> Josh
>>>
>>> Hi Josh,
>>>
>>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
>>> Because, rds-ping uses zero byte messages to ping.
>>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
>>> The Oracle's internal code I checked doesn't have that patch applied.
>>>
>>> Venkat
>> No I don't have a test case. I came across this CVE while doing an
>> audit and noticed it was patched in Ubuntu's kernel and other distros,
>> but was not in the upstream kernel yet. Quick googling of lkml showed
>> that there were at least two attempts to get this patch upstream, but
>> both had issues due to not following the proper submission process:
>>
>> https://lkml.org/lkml/2012/10/22/433
>> https://lkml.org/lkml/2012/9/21/505
>>
>> From my searching it appears the initial bug was found by someone at redhat:
>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>
>> I've added Li Honggang the reporter of this issue from Redhat to the
>> mail. Hopefully he can share his testcase.
> The test case is very simple:
> Steps to Reproduce:
> 1. yum install -y rds-tools
>
> 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr'
>           inet addr:172.31.0.3  Bcast:172.31.0.255  Mask:255.255.255.0
>
> 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3  <<<< kernel panic (You
> may need to wait for a few seconds before the kernel panic.)
>>
>> and possibly requires certain hardware as Jay writes in the first link above:
>> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..."
> This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko),
> QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko".
>
> As I know the upstream code of RDS is broken. There are *many* RDS bugs.
>
> Best regards.
> Honggang

Thanks Honggang. I have resubmitted the patch for approval.

-- 
Josh

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Honggang LI]

On 11/14/2013 01:40 AM, Josh Hunt wrote:
> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra
> <venkat.x.venkatsubra@oracle.com> wrote:
>>
>> -----Original Message-----
>> From: Josh Hunt [mailto:joshhunt00@gmail.com]
>> Sent: Tuesday, November 12, 2013 10:25 PM
>> To: David Miller
>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>>
>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>>> From: John Jolly <jjolly@suse.com>
>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>>>
>>>>> Attempting an rds connection from the IP address of an IPoIB
>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>>>>> Making the test less strict allows rds-ping to work without
>>>>> crashing the machine.
>>>>>
>>>>> A local unprivileged user could use this flaw to crash the system.
>>>>>
>>>>> Signed-off-by: John Jolly <jjolly@suse.com>
>>>> Besides the questions being asked of you by Venkat Venkatsubra, this
>>>> patch has another issue.
>>>>
>>>> It has been completely corrupted by your email client, it has turned
>>>> all TAB characters into spaces, making the patch useless.
>>>>
>>>> Please learn how to send a patch unmolested in the body of your
>>>> email.  Test it by emailing the patch to yourself, and verifying that
>>>> you can in fact apply the patch you receive in that email.
>>>> Then, and only then, should you consider making a new submission of
>>>> this patch.
>>>>
>>>> Use Documentation/email-clients.txt for guidance.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>> Please read the FAQ at  http://www.tux.org/lkml/
>>>
>>> I think this issue was lost in the shuffle. It appears that redhat,
>>> ubuntu, and oracle are maintaining local patches to resolve this:
>>>
>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
>>> 2be130fa15fa8be10d4704e8
>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853
>>> 88.html
>>>
>>> Given that Oracle has applied it I'll make the assumption that
>>> Venkat's question was answered at some point.
>>>
>>> David - I can resubmit the patch with the proper signed-off-by and
>>> formatting if you are willing to apply it unless John wants to try
>>> again. I think it's time this got upstream.
>>>
>>> --
>>> Josh
>> Ugh.. hopefully resending with all the html crap removed...
>>
>> --
>> Josh
>>
>> Hi Josh,
>>
>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
>> Because, rds-ping uses zero byte messages to ping.
>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
>> The Oracle's internal code I checked doesn't have that patch applied.
>>
>> Venkat
> No I don't have a test case. I came across this CVE while doing an
> audit and noticed it was patched in Ubuntu's kernel and other distros,
> but was not in the upstream kernel yet. Quick googling of lkml showed
> that there were at least two attempts to get this patch upstream, but
> both had issues due to not following the proper submission process:
>
> https://lkml.org/lkml/2012/10/22/433
> https://lkml.org/lkml/2012/9/21/505
>
> From my searching it appears the initial bug was found by someone at redhat:
> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>
> I've added Li Honggang the reporter of this issue from Redhat to the
> mail. Hopefully he can share his testcase.
The test case is very simple:
Steps to Reproduce:
1. yum install -y rds-tools

2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr'
          inet addr:172.31.0.3  Bcast:172.31.0.255  Mask:255.255.255.0

3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3  <<<< kernel panic (You
may need to wait for a few seconds before the kernel panic.)
>
> and possibly requires certain hardware as Jay writes in the first link above:
> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..."
This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko),
QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko".

As I know the upstream code of RDS is broken. There are *many* RDS bugs.

Best regards.
Honggang
>
> I was referring to this oracle commit:
> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8
>
> I have no experience with this code. There were a few comments around
> the reset and xmit fns about making sure the caller did certain things
> if not they were racy, but I have no idea if that's coming into play
> here.
>

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Josh Hunt]

On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra
<venkat.x.venkatsubra@oracle.com> wrote:
>
>
> -----Original Message-----
> From: Josh Hunt [mailto:joshhunt00@gmail.com]
> Sent: Tuesday, November 12, 2013 10:25 PM
> To: David Miller
> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback
>
> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>>
>>> From: John Jolly <jjolly@suse.com>
>>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>>
>>> > Attempting an rds connection from the IP address of an IPoIB
>>> > interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>>> > Making the test less strict allows rds-ping to work without
>>> > crashing the machine.
>>> >
>>> > A local unprivileged user could use this flaw to crash the system.
>>> >
>>> > Signed-off-by: John Jolly <jjolly@suse.com>
>>>
>>> Besides the questions being asked of you by Venkat Venkatsubra, this
>>> patch has another issue.
>>>
>>> It has been completely corrupted by your email client, it has turned
>>> all TAB characters into spaces, making the patch useless.
>>>
>>> Please learn how to send a patch unmolested in the body of your
>>> email.  Test it by emailing the patch to yourself, and verifying that
>>> you can in fact apply the patch you receive in that email.
>>> Then, and only then, should you consider making a new submission of
>>> this patch.
>>>
>>> Use Documentation/email-clients.txt for guidance.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>> I think this issue was lost in the shuffle. It appears that redhat,
>> ubuntu, and oracle are maintaining local patches to resolve this:
>>
>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
>> 2be130fa15fa8be10d4704e8
>> https://bugzilla.redhat.com/show_bug.cgi?id=822754
>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853
>> 88.html
>>
>> Given that Oracle has applied it I'll make the assumption that
>> Venkat's question was answered at some point.
>>
>> David - I can resubmit the patch with the proper signed-off-by and
>> formatting if you are willing to apply it unless John wants to try
>> again. I think it's time this got upstream.
>>
>> --
>> Josh
>
> Ugh.. hopefully resending with all the html crap removed...
>
> --
> Josh
>
> Hi Josh,
>
> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
> Because, rds-ping uses zero byte messages to ping.
> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
> The Oracle's internal code I checked doesn't have that patch applied.
>
> Venkat

No I don't have a test case. I came across this CVE while doing an
audit and noticed it was patched in Ubuntu's kernel and other distros,
but was not in the upstream kernel yet. Quick googling of lkml showed
that there were at least two attempts to get this patch upstream, but
both had issues due to not following the proper submission process:

https://lkml.org/lkml/2012/10/22/433
https://lkml.org/lkml/2012/9/21/505

>From my searching it appears the initial bug was found by someone at redhat:
https://bugzilla.redhat.com/show_bug.cgi?id=822754

I've added Li Honggang the reporter of this issue from Redhat to the
mail. Hopefully he can share his testcase.

and possibly requires certain hardware as Jay writes in the first link above:
"...some Infiniband HCAs(QLogic, possibly others) the machine will panic..."

I was referring to this oracle commit:
https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8

I have no experience with this code. There were a few comments around
the reset and xmit fns about making sure the caller did certain things
if not they were racy, but I have no idea if that's coming into play
here.

-- 
Josh

RE: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

-----Original Message-----
From: Josh Hunt [mailto:joshhunt00@gmail.com] 
Sent: Tuesday, November 12, 2013 10:25 PM
To: David Miller
Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org
Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback

On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>
>> From: John Jolly <jjolly@suse.com>
>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>
>> > Attempting an rds connection from the IP address of an IPoIB 
>> > interface to itself causes a kernel panic due to a BUG_ON() being triggered.
>> > Making the test less strict allows rds-ping to work without 
>> > crashing the machine.
>> >
>> > A local unprivileged user could use this flaw to crash the system.
>> >
>> > Signed-off-by: John Jolly <jjolly@suse.com>
>>
>> Besides the questions being asked of you by Venkat Venkatsubra, this 
>> patch has another issue.
>>
>> It has been completely corrupted by your email client, it has turned 
>> all TAB characters into spaces, making the patch useless.
>>
>> Please learn how to send a patch unmolested in the body of your 
>> email.  Test it by emailing the patch to yourself, and verifying that 
>> you can in fact apply the patch you receive in that email.
>> Then, and only then, should you consider making a new submission of 
>> this patch.
>>
>> Use Documentation/email-clients.txt for guidance.
>> --
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in the body of a message to majordomo@vger.kernel.org 
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
>
> I think this issue was lost in the shuffle. It appears that redhat, 
> ubuntu, and oracle are maintaining local patches to resolve this:
>
> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685
> 2be130fa15fa8be10d4704e8
> https://bugzilla.redhat.com/show_bug.cgi?id=822754
> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853
> 88.html
>
> Given that Oracle has applied it I'll make the assumption that 
> Venkat's question was answered at some point.
>
> David - I can resubmit the patch with the proper signed-off-by and 
> formatting if you are willing to apply it unless John wants to try 
> again. I think it's time this got upstream.
>
> --
> Josh

Ugh.. hopefully resending with all the html crap removed...

--
Josh

Hi Josh,

No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE).
Because, rds-ping uses zero byte messages to ping.
If you have a test case that reproduces the kernel panic I can try it out and see how that can happen.
The Oracle's internal code I checked doesn't have that patch applied.

Venkat

Re: [PATCH] rds: Error on offset mismatch if not loopback

[David Miller]

From: Josh Hunt <joshhunt00@gmail.com>
Date: Tue, 12 Nov 2013 22:22:11 -0600

> David - I can resubmit the patch with the proper signed-off-by and
> formatting if you are willing to apply it unless John wants to try again. I
> think it's time this got upstream.

Nothing is going to happen until the patch is submitted properly, so
just do, don't ask.

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Josh Hunt]

On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote:
> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote:
>>
>> From: John Jolly <jjolly@suse.com>
>> Date: Fri, 21 Sep 2012 15:32:40 -0600
>>
>> > Attempting an rds connection from the IP address of an IPoIB interface
>> > to itself causes a kernel panic due to a BUG_ON() being triggered.
>> > Making the test less strict allows rds-ping to work without crashing
>> > the machine.
>> >
>> > A local unprivileged user could use this flaw to crash the system.
>> >
>> > Signed-off-by: John Jolly <jjolly@suse.com>
>>
>> Besides the questions being asked of you by Venkat Venkatsubra, this
>> patch has another issue.
>>
>> It has been completely corrupted by your email client, it has
>> turned all TAB characters into spaces, making the patch useless.
>>
>> Please learn how to send a patch unmolested in the body of your
>> email.  Test it by emailing the patch to yourself, and verifying
>> that you can in fact apply the patch you receive in that email.
>> Then, and only then, should you consider making a new submission
>> of this patch.
>>
>> Use Documentation/email-clients.txt for guidance.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
>
> I think this issue was lost in the shuffle. It appears that redhat, ubuntu,
> and oracle are maintaining local patches to resolve this:
>
> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8
> https://bugzilla.redhat.com/show_bug.cgi?id=822754
> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td4985388.html
>
> Given that Oracle has applied it I'll make the assumption that Venkat's
> question was answered at some point.
>
> David - I can resubmit the patch with the proper signed-off-by and
> formatting if you are willing to apply it unless John wants to try again. I
> think it's time this got upstream.
>
> --
> Josh

Ugh.. hopefully resending with all the html crap removed...

-- 
Josh

Re: [PATCH] rds: Error on offset mismatch if not loopback

[David Miller]

From: John Jolly <jjolly@suse.com>
Date: Fri, 21 Sep 2012 15:32:40 -0600

> Attempting an rds connection from the IP address of an IPoIB interface
> to itself causes a kernel panic due to a BUG_ON() being triggered.
> Making the test less strict allows rds-ping to work without crashing
> the machine.
> 
> A local unprivileged user could use this flaw to crash the system.
> 
> Signed-off-by: John Jolly <jjolly@suse.com>

Besides the questions being asked of you by Venkat Venkatsubra, this
patch has another issue.

It has been completely corrupted by your email client, it has
turned all TAB characters into spaces, making the patch useless.

Please learn how to send a patch unmolested in the body of your
email.  Test it by emailing the patch to yourself, and verifying
that you can in fact apply the patch you receive in that email.
Then, and only then, should you consider making a new submission
of this patch.

Use Documentation/email-clients.txt for guidance.

Re: [PATCH] rds: Error on offset mismatch if not loopback

[Venkat Venkatsubra]

On 9/21/2012 4:32 PM, John Jolly wrote:
> Attempting an rds connection from the IP address of an IPoIB interface
> to itself causes a kernel panic due to a BUG_ON() being triggered.
> Making the test less strict allows rds-ping to work without crashing
> the machine.
>
> A local unprivileged user could use this flaw to crash the system.
>
> Signed-off-by: John Jolly<jjolly@suse.com>
> ---
>   net/rds/ib_send.c |    2 +-
>   1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c
> index e590949..7920c85 100644
> --- a/net/rds/ib_send.c
> +++ b/net/rds/ib_send.c
> @@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm,
>          int flow_controlled = 0;
>          int nr_sig = 0;
>
> -       BUG_ON(off % RDS_FRAG_SIZE);
> +       BUG_ON(!conn->c_loopback&&  off % RDS_FRAG_SIZE);
>          BUG_ON(hdr_off != 0&&  hdr_off != sizeof(struct rds_header));
>
>          /* Do not send cong updates to IB loopback */
Hi John,

How do you trigger this BUG_ON ?
With rds-ping I could not hit this condition of non-zero "off % 
RDS_FRAG_SIZE".
rds-ping uses zero byte messages to ping or pong back. How does the 
"off" become non-zero ?

Thanks.

Venkat

[PATCH] rds: Error on offset mismatch if not loopback

[John Jolly]

Attempting an rds connection from the IP address of an IPoIB interface
to itself causes a kernel panic due to a BUG_ON() being triggered.
Making the test less strict allows rds-ping to work without crashing
the machine.

A local unprivileged user could use this flaw to crash the system.

Signed-off-by: John Jolly <jjolly@suse.com>
---
 net/rds/ib_send.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c
index e590949..7920c85 100644
--- a/net/rds/ib_send.c
+++ b/net/rds/ib_send.c
@@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm,
        int flow_controlled = 0;
        int nr_sig = 0;
 
-       BUG_ON(off % RDS_FRAG_SIZE);
+       BUG_ON(!conn->c_loopback && off % RDS_FRAG_SIZE);
        BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header));
 
        /* Do not send cong updates to IB loopback */
-- 
1.7.7