* [PATCH] rds: Error on offset mismatch if not loopback @ 2012-09-21 21:32 John Jolly 2012-09-21 21:38 ` Venkat Venkatsubra 2012-09-22 19:25 ` David Miller 0 siblings, 2 replies; 18+ messages in thread From: John Jolly @ 2012-09-21 21:32 UTC (permalink / raw) To: linux-kernel; +Cc: venkat.x.venkatsubra, netdev Attempting an rds connection from the IP address of an IPoIB interface to itself causes a kernel panic due to a BUG_ON() being triggered. Making the test less strict allows rds-ping to work without crashing the machine. A local unprivileged user could use this flaw to crash the system. Signed-off-by: John Jolly <jjolly@suse.com> --- net/rds/ib_send.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c index e590949..7920c85 100644 --- a/net/rds/ib_send.c +++ b/net/rds/ib_send.c @@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm, int flow_controlled = 0; int nr_sig = 0; - BUG_ON(off % RDS_FRAG_SIZE); + BUG_ON(!conn->c_loopback && off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ -- 1.7.7 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2012-09-21 21:32 [PATCH] rds: Error on offset mismatch if not loopback John Jolly @ 2012-09-21 21:38 ` Venkat Venkatsubra 2012-09-22 19:25 ` David Miller 1 sibling, 0 replies; 18+ messages in thread From: Venkat Venkatsubra @ 2012-09-21 21:38 UTC (permalink / raw) To: John Jolly; +Cc: linux-kernel, netdev On 9/21/2012 4:32 PM, John Jolly wrote: > Attempting an rds connection from the IP address of an IPoIB interface > to itself causes a kernel panic due to a BUG_ON() being triggered. > Making the test less strict allows rds-ping to work without crashing > the machine. > > A local unprivileged user could use this flaw to crash the system. > > Signed-off-by: John Jolly<jjolly@suse.com> > --- > net/rds/ib_send.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c > index e590949..7920c85 100644 > --- a/net/rds/ib_send.c > +++ b/net/rds/ib_send.c > @@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm, > int flow_controlled = 0; > int nr_sig = 0; > > - BUG_ON(off % RDS_FRAG_SIZE); > + BUG_ON(!conn->c_loopback&& off % RDS_FRAG_SIZE); > BUG_ON(hdr_off != 0&& hdr_off != sizeof(struct rds_header)); > > /* Do not send cong updates to IB loopback */ Hi John, How do you trigger this BUG_ON ? With rds-ping I could not hit this condition of non-zero "off % RDS_FRAG_SIZE". rds-ping uses zero byte messages to ping or pong back. How does the "off" become non-zero ? Thanks. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2012-09-21 21:32 [PATCH] rds: Error on offset mismatch if not loopback John Jolly 2012-09-21 21:38 ` Venkat Venkatsubra @ 2012-09-22 19:25 ` David Miller [not found] ` <CAKA=qzac9=UhLF_Z4FnnH+sR7xvkDux4oayC6dPYe=hMLsDxRg@mail.gmail.com> 1 sibling, 1 reply; 18+ messages in thread From: David Miller @ 2012-09-22 19:25 UTC (permalink / raw) To: jjolly; +Cc: linux-kernel, venkat.x.venkatsubra, netdev From: John Jolly <jjolly@suse.com> Date: Fri, 21 Sep 2012 15:32:40 -0600 > Attempting an rds connection from the IP address of an IPoIB interface > to itself causes a kernel panic due to a BUG_ON() being triggered. > Making the test less strict allows rds-ping to work without crashing > the machine. > > A local unprivileged user could use this flaw to crash the system. > > Signed-off-by: John Jolly <jjolly@suse.com> Besides the questions being asked of you by Venkat Venkatsubra, this patch has another issue. It has been completely corrupted by your email client, it has turned all TAB characters into spaces, making the patch useless. Please learn how to send a patch unmolested in the body of your email. Test it by emailing the patch to yourself, and verifying that you can in fact apply the patch you receive in that email. Then, and only then, should you consider making a new submission of this patch. Use Documentation/email-clients.txt for guidance. ^ permalink raw reply [flat|nested] 18+ messages in thread
[parent not found: <CAKA=qzac9=UhLF_Z4FnnH+sR7xvkDux4oayC6dPYe=hMLsDxRg@mail.gmail.com>]
* Re: [PATCH] rds: Error on offset mismatch if not loopback [not found] ` <CAKA=qzac9=UhLF_Z4FnnH+sR7xvkDux4oayC6dPYe=hMLsDxRg@mail.gmail.com> @ 2013-11-13 4:24 ` Josh Hunt 2013-11-13 15:16 ` Venkat Venkatsubra 2013-11-13 6:09 ` David Miller 1 sibling, 1 reply; 18+ messages in thread From: Josh Hunt @ 2013-11-13 4:24 UTC (permalink / raw) To: David Miller; +Cc: jjolly, LKML, venkat.x.venkatsubra, netdev On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: > On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >> >> From: John Jolly <jjolly@suse.com> >> Date: Fri, 21 Sep 2012 15:32:40 -0600 >> >> > Attempting an rds connection from the IP address of an IPoIB interface >> > to itself causes a kernel panic due to a BUG_ON() being triggered. >> > Making the test less strict allows rds-ping to work without crashing >> > the machine. >> > >> > A local unprivileged user could use this flaw to crash the system. >> > >> > Signed-off-by: John Jolly <jjolly@suse.com> >> >> Besides the questions being asked of you by Venkat Venkatsubra, this >> patch has another issue. >> >> It has been completely corrupted by your email client, it has >> turned all TAB characters into spaces, making the patch useless. >> >> Please learn how to send a patch unmolested in the body of your >> email. Test it by emailing the patch to yourself, and verifying >> that you can in fact apply the patch you receive in that email. >> Then, and only then, should you consider making a new submission >> of this patch. >> >> Use Documentation/email-clients.txt for guidance. >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ > > > I think this issue was lost in the shuffle. It appears that redhat, ubuntu, > and oracle are maintaining local patches to resolve this: > > https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8 > https://bugzilla.redhat.com/show_bug.cgi?id=822754 > http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td4985388.html > > Given that Oracle has applied it I'll make the assumption that Venkat's > question was answered at some point. > > David - I can resubmit the patch with the proper signed-off-by and > formatting if you are willing to apply it unless John wants to try again. I > think it's time this got upstream. > > -- > Josh Ugh.. hopefully resending with all the html crap removed... -- Josh ^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-13 4:24 ` Josh Hunt @ 2013-11-13 15:16 ` Venkat Venkatsubra 2013-11-13 17:40 ` Josh Hunt 0 siblings, 1 reply; 18+ messages in thread From: Venkat Venkatsubra @ 2013-11-13 15:16 UTC (permalink / raw) To: Josh Hunt, David Miller; +Cc: jjolly, LKML, netdev -----Original Message----- From: Josh Hunt [mailto:joshhunt00@gmail.com] Sent: Tuesday, November 12, 2013 10:25 PM To: David Miller Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: > On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >> >> From: John Jolly <jjolly@suse.com> >> Date: Fri, 21 Sep 2012 15:32:40 -0600 >> >> > Attempting an rds connection from the IP address of an IPoIB >> > interface to itself causes a kernel panic due to a BUG_ON() being triggered. >> > Making the test less strict allows rds-ping to work without >> > crashing the machine. >> > >> > A local unprivileged user could use this flaw to crash the system. >> > >> > Signed-off-by: John Jolly <jjolly@suse.com> >> >> Besides the questions being asked of you by Venkat Venkatsubra, this >> patch has another issue. >> >> It has been completely corrupted by your email client, it has turned >> all TAB characters into spaces, making the patch useless. >> >> Please learn how to send a patch unmolested in the body of your >> email. Test it by emailing the patch to yourself, and verifying that >> you can in fact apply the patch you receive in that email. >> Then, and only then, should you consider making a new submission of >> this patch. >> >> Use Documentation/email-clients.txt for guidance. >> -- >> To unsubscribe from this list: send the line "unsubscribe >> linux-kernel" in the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> Please read the FAQ at http://www.tux.org/lkml/ > > > I think this issue was lost in the shuffle. It appears that redhat, > ubuntu, and oracle are maintaining local patches to resolve this: > > https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 > 2be130fa15fa8be10d4704e8 > https://bugzilla.redhat.com/show_bug.cgi?id=822754 > http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853 > 88.html > > Given that Oracle has applied it I'll make the assumption that > Venkat's question was answered at some point. > > David - I can resubmit the patch with the proper signed-off-by and > formatting if you are willing to apply it unless John wants to try > again. I think it's time this got upstream. > > -- > Josh Ugh.. hopefully resending with all the html crap removed... -- Josh Hi Josh, No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). Because, rds-ping uses zero byte messages to ping. If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. The Oracle's internal code I checked doesn't have that patch applied. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-13 15:16 ` Venkat Venkatsubra @ 2013-11-13 17:40 ` Josh Hunt 2013-11-14 0:55 ` Honggang LI 0 siblings, 1 reply; 18+ messages in thread From: Josh Hunt @ 2013-11-13 17:40 UTC (permalink / raw) To: Venkat Venkatsubra, honli; +Cc: David Miller, jjolly, LKML, netdev On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra <venkat.x.venkatsubra@oracle.com> wrote: > > > -----Original Message----- > From: Josh Hunt [mailto:joshhunt00@gmail.com] > Sent: Tuesday, November 12, 2013 10:25 PM > To: David Miller > Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org > Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback > > On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: >> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >>> >>> From: John Jolly <jjolly@suse.com> >>> Date: Fri, 21 Sep 2012 15:32:40 -0600 >>> >>> > Attempting an rds connection from the IP address of an IPoIB >>> > interface to itself causes a kernel panic due to a BUG_ON() being triggered. >>> > Making the test less strict allows rds-ping to work without >>> > crashing the machine. >>> > >>> > A local unprivileged user could use this flaw to crash the system. >>> > >>> > Signed-off-by: John Jolly <jjolly@suse.com> >>> >>> Besides the questions being asked of you by Venkat Venkatsubra, this >>> patch has another issue. >>> >>> It has been completely corrupted by your email client, it has turned >>> all TAB characters into spaces, making the patch useless. >>> >>> Please learn how to send a patch unmolested in the body of your >>> email. Test it by emailing the patch to yourself, and verifying that >>> you can in fact apply the patch you receive in that email. >>> Then, and only then, should you consider making a new submission of >>> this patch. >>> >>> Use Documentation/email-clients.txt for guidance. >>> -- >>> To unsubscribe from this list: send the line "unsubscribe >>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>> Please read the FAQ at http://www.tux.org/lkml/ >> >> >> I think this issue was lost in the shuffle. It appears that redhat, >> ubuntu, and oracle are maintaining local patches to resolve this: >> >> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 >> 2be130fa15fa8be10d4704e8 >> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853 >> 88.html >> >> Given that Oracle has applied it I'll make the assumption that >> Venkat's question was answered at some point. >> >> David - I can resubmit the patch with the proper signed-off-by and >> formatting if you are willing to apply it unless John wants to try >> again. I think it's time this got upstream. >> >> -- >> Josh > > Ugh.. hopefully resending with all the html crap removed... > > -- > Josh > > Hi Josh, > > No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). > Because, rds-ping uses zero byte messages to ping. > If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. > The Oracle's internal code I checked doesn't have that patch applied. > > Venkat No I don't have a test case. I came across this CVE while doing an audit and noticed it was patched in Ubuntu's kernel and other distros, but was not in the upstream kernel yet. Quick googling of lkml showed that there were at least two attempts to get this patch upstream, but both had issues due to not following the proper submission process: https://lkml.org/lkml/2012/10/22/433 https://lkml.org/lkml/2012/9/21/505 >From my searching it appears the initial bug was found by someone at redhat: https://bugzilla.redhat.com/show_bug.cgi?id=822754 I've added Li Honggang the reporter of this issue from Redhat to the mail. Hopefully he can share his testcase. and possibly requires certain hardware as Jay writes in the first link above: "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..." I was referring to this oracle commit: https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8 I have no experience with this code. There were a few comments around the reset and xmit fns about making sure the caller did certain things if not they were racy, but I have no idea if that's coming into play here. -- Josh ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-13 17:40 ` Josh Hunt @ 2013-11-14 0:55 ` Honggang LI 2013-11-14 1:27 ` Josh Hunt 2013-11-14 13:43 ` Venkat Venkatsubra 0 siblings, 2 replies; 18+ messages in thread From: Honggang LI @ 2013-11-14 0:55 UTC (permalink / raw) To: Josh Hunt, Venkat Venkatsubra; +Cc: David Miller, jjolly, LKML, netdev On 11/14/2013 01:40 AM, Josh Hunt wrote: > On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra > <venkat.x.venkatsubra@oracle.com> wrote: >> >> -----Original Message----- >> From: Josh Hunt [mailto:joshhunt00@gmail.com] >> Sent: Tuesday, November 12, 2013 10:25 PM >> To: David Miller >> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org >> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback >> >> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: >>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >>>> From: John Jolly <jjolly@suse.com> >>>> Date: Fri, 21 Sep 2012 15:32:40 -0600 >>>> >>>>> Attempting an rds connection from the IP address of an IPoIB >>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered. >>>>> Making the test less strict allows rds-ping to work without >>>>> crashing the machine. >>>>> >>>>> A local unprivileged user could use this flaw to crash the system. >>>>> >>>>> Signed-off-by: John Jolly <jjolly@suse.com> >>>> Besides the questions being asked of you by Venkat Venkatsubra, this >>>> patch has another issue. >>>> >>>> It has been completely corrupted by your email client, it has turned >>>> all TAB characters into spaces, making the patch useless. >>>> >>>> Please learn how to send a patch unmolested in the body of your >>>> email. Test it by emailing the patch to yourself, and verifying that >>>> you can in fact apply the patch you receive in that email. >>>> Then, and only then, should you consider making a new submission of >>>> this patch. >>>> >>>> Use Documentation/email-clients.txt for guidance. >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> Please read the FAQ at http://www.tux.org/lkml/ >>> >>> I think this issue was lost in the shuffle. It appears that redhat, >>> ubuntu, and oracle are maintaining local patches to resolve this: >>> >>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 >>> 2be130fa15fa8be10d4704e8 >>> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853 >>> 88.html >>> >>> Given that Oracle has applied it I'll make the assumption that >>> Venkat's question was answered at some point. >>> >>> David - I can resubmit the patch with the proper signed-off-by and >>> formatting if you are willing to apply it unless John wants to try >>> again. I think it's time this got upstream. >>> >>> -- >>> Josh >> Ugh.. hopefully resending with all the html crap removed... >> >> -- >> Josh >> >> Hi Josh, >> >> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). >> Because, rds-ping uses zero byte messages to ping. >> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. >> The Oracle's internal code I checked doesn't have that patch applied. >> >> Venkat > No I don't have a test case. I came across this CVE while doing an > audit and noticed it was patched in Ubuntu's kernel and other distros, > but was not in the upstream kernel yet. Quick googling of lkml showed > that there were at least two attempts to get this patch upstream, but > both had issues due to not following the proper submission process: > > https://lkml.org/lkml/2012/10/22/433 > https://lkml.org/lkml/2012/9/21/505 > > From my searching it appears the initial bug was found by someone at redhat: > https://bugzilla.redhat.com/show_bug.cgi?id=822754 > > I've added Li Honggang the reporter of this issue from Redhat to the > mail. Hopefully he can share his testcase. The test case is very simple: Steps to Reproduce: 1. yum install -y rds-tools 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr' inet addr:172.31.0.3 Bcast:172.31.0.255 Mask:255.255.255.0 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3 <<<< kernel panic (You may need to wait for a few seconds before the kernel panic.) > > and possibly requires certain hardware as Jay writes in the first link above: > "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..." This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko". As I know the upstream code of RDS is broken. There are *many* RDS bugs. Best regards. Honggang > > I was referring to this oracle commit: > https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636852be130fa15fa8be10d4704e8 > > I have no experience with this code. There were a few comments around > the reset and xmit fns about making sure the caller did certain things > if not they were racy, but I have no idea if that's coming into play > here. > ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-14 0:55 ` Honggang LI @ 2013-11-14 1:27 ` Josh Hunt 2013-11-14 13:43 ` Venkat Venkatsubra 1 sibling, 0 replies; 18+ messages in thread From: Josh Hunt @ 2013-11-14 1:27 UTC (permalink / raw) To: Honggang LI; +Cc: Venkat Venkatsubra, David Miller, jjolly, LKML, netdev On Wed, Nov 13, 2013 at 6:55 PM, Honggang LI <honli@redhat.com> wrote: > On 11/14/2013 01:40 AM, Josh Hunt wrote: >> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra >> <venkat.x.venkatsubra@oracle.com> wrote: >>> >>> -----Original Message----- >>> From: Josh Hunt [mailto:joshhunt00@gmail.com] >>> Sent: Tuesday, November 12, 2013 10:25 PM >>> To: David Miller >>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org >>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback >>> >>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: >>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >>>>> From: John Jolly <jjolly@suse.com> >>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600 >>>>> >>>>>> Attempting an rds connection from the IP address of an IPoIB >>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered. >>>>>> Making the test less strict allows rds-ping to work without >>>>>> crashing the machine. >>>>>> >>>>>> A local unprivileged user could use this flaw to crash the system. >>>>>> >>>>>> Signed-off-by: John Jolly <jjolly@suse.com> >>>>> Besides the questions being asked of you by Venkat Venkatsubra, this >>>>> patch has another issue. >>>>> >>>>> It has been completely corrupted by your email client, it has turned >>>>> all TAB characters into spaces, making the patch useless. >>>>> >>>>> Please learn how to send a patch unmolested in the body of your >>>>> email. Test it by emailing the patch to yourself, and verifying that >>>>> you can in fact apply the patch you receive in that email. >>>>> Then, and only then, should you consider making a new submission of >>>>> this patch. >>>>> >>>>> Use Documentation/email-clients.txt for guidance. >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> Please read the FAQ at http://www.tux.org/lkml/ >>>> >>>> I think this issue was lost in the shuffle. It appears that redhat, >>>> ubuntu, and oracle are maintaining local patches to resolve this: >>>> >>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 >>>> 2be130fa15fa8be10d4704e8 >>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td49853 >>>> 88.html >>>> >>>> Given that Oracle has applied it I'll make the assumption that >>>> Venkat's question was answered at some point. >>>> >>>> David - I can resubmit the patch with the proper signed-off-by and >>>> formatting if you are willing to apply it unless John wants to try >>>> again. I think it's time this got upstream. >>>> >>>> -- >>>> Josh >>> Ugh.. hopefully resending with all the html crap removed... >>> >>> -- >>> Josh >>> >>> Hi Josh, >>> >>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). >>> Because, rds-ping uses zero byte messages to ping. >>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. >>> The Oracle's internal code I checked doesn't have that patch applied. >>> >>> Venkat >> No I don't have a test case. I came across this CVE while doing an >> audit and noticed it was patched in Ubuntu's kernel and other distros, >> but was not in the upstream kernel yet. Quick googling of lkml showed >> that there were at least two attempts to get this patch upstream, but >> both had issues due to not following the proper submission process: >> >> https://lkml.org/lkml/2012/10/22/433 >> https://lkml.org/lkml/2012/9/21/505 >> >> From my searching it appears the initial bug was found by someone at redhat: >> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >> >> I've added Li Honggang the reporter of this issue from Redhat to the >> mail. Hopefully he can share his testcase. > The test case is very simple: > Steps to Reproduce: > 1. yum install -y rds-tools > > 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr' > inet addr:172.31.0.3 Bcast:172.31.0.255 Mask:255.255.255.0 > > 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3 <<<< kernel panic (You > may need to wait for a few seconds before the kernel panic.) >> >> and possibly requires certain hardware as Jay writes in the first link above: >> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..." > This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), > QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko". > > As I know the upstream code of RDS is broken. There are *many* RDS bugs. > > Best regards. > Honggang Thanks Honggang. I have resubmitted the patch for approval. -- Josh ^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-14 0:55 ` Honggang LI 2013-11-14 1:27 ` Josh Hunt @ 2013-11-14 13:43 ` Venkat Venkatsubra 2013-11-15 2:32 ` Honggang LI 1 sibling, 1 reply; 18+ messages in thread From: Venkat Venkatsubra @ 2013-11-14 13:43 UTC (permalink / raw) To: Honggang LI, Josh Hunt; +Cc: David Miller, jjolly, LKML, netdev -----Original Message----- From: Honggang LI [mailto:honli@redhat.com] Sent: Wednesday, November 13, 2013 6:56 PM To: Josh Hunt; Venkat Venkatsubra Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback On 11/14/2013 01:40 AM, Josh Hunt wrote: > On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra > <venkat.x.venkatsubra@oracle.com> wrote: >> >> -----Original Message----- >> From: Josh Hunt [mailto:joshhunt00@gmail.com] >> Sent: Tuesday, November 12, 2013 10:25 PM >> To: David Miller >> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org >> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback >> >> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: >>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >>>> From: John Jolly <jjolly@suse.com> >>>> Date: Fri, 21 Sep 2012 15:32:40 -0600 >>>> >>>>> Attempting an rds connection from the IP address of an IPoIB >>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered. >>>>> Making the test less strict allows rds-ping to work without >>>>> crashing the machine. >>>>> >>>>> A local unprivileged user could use this flaw to crash the system. >>>>> >>>>> Signed-off-by: John Jolly <jjolly@suse.com> >>>> Besides the questions being asked of you by Venkat Venkatsubra, >>>> this patch has another issue. >>>> >>>> It has been completely corrupted by your email client, it has >>>> turned all TAB characters into spaces, making the patch useless. >>>> >>>> Please learn how to send a patch unmolested in the body of your >>>> email. Test it by emailing the patch to yourself, and verifying >>>> that you can in fact apply the patch you receive in that email. >>>> Then, and only then, should you consider making a new submission of >>>> this patch. >>>> >>>> Use Documentation/email-clients.txt for guidance. >>>> -- >>>> To unsubscribe from this list: send the line "unsubscribe >>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>> Please read the FAQ at http://www.tux.org/lkml/ >>> >>> I think this issue was lost in the shuffle. It appears that redhat, >>> ubuntu, and oracle are maintaining local patches to resolve this: >>> >>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636 >>> 85 >>> 2be130fa15fa8be10d4704e8 >>> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td498 >>> 53 >>> 88.html >>> >>> Given that Oracle has applied it I'll make the assumption that >>> Venkat's question was answered at some point. >>> >>> David - I can resubmit the patch with the proper signed-off-by and >>> formatting if you are willing to apply it unless John wants to try >>> again. I think it's time this got upstream. >>> >>> -- >>> Josh >> Ugh.. hopefully resending with all the html crap removed... >> >> -- >> Josh >> >> Hi Josh, >> >> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). >> Because, rds-ping uses zero byte messages to ping. >> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. >> The Oracle's internal code I checked doesn't have that patch applied. >> >> Venkat > No I don't have a test case. I came across this CVE while doing an > audit and noticed it was patched in Ubuntu's kernel and other distros, > but was not in the upstream kernel yet. Quick googling of lkml showed > that there were at least two attempts to get this patch upstream, but > both had issues due to not following the proper submission process: > > https://lkml.org/lkml/2012/10/22/433 > https://lkml.org/lkml/2012/9/21/505 > > From my searching it appears the initial bug was found by someone at redhat: > https://bugzilla.redhat.com/show_bug.cgi?id=822754 > > I've added Li Honggang the reporter of this issue from Redhat to the > mail. Hopefully he can share his testcase. The test case is very simple: Steps to Reproduce: 1. yum install -y rds-tools 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr' inet addr:172.31.0.3 Bcast:172.31.0.255 Mask:255.255.255.0 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3 <<<< kernel panic (You may need to wait for a few seconds before the kernel panic.) > > and possibly requires certain hardware as Jay writes in the first link above: > "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..." This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko". As I know the upstream code of RDS is broken. There are *many* RDS bugs. Best regards. Honggang > > I was referring to this oracle commit: > https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 > 2be130fa15fa8be10d4704e8 > > I have no experience with this code. There were a few comments around > the reset and xmit fns about making sure the caller did certain things > if not they were racy, but I have no idea if that's coming into play > here. > Hi Honggang, I ran rds-ping over local interface for 30 minutes. I stopped it after that. It didn't hit any panic. # ip addr show dev ib0 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 1024 link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:21:28:00:01:cf:63:db brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff inet 10.196.4.125/30 brd 10.196.4.127 scope global ib0 inet6 fe80::221:2800:1cf:63db/64 scope link valid_lft forever preferred_lft forever # # rds-ping 10.196.4.125 1: 170 usec 2: 171 usec .... .... .... 1860: 173 usec 1861: 171 usec 1862: 177 usec 1863: 168 usec 1864: 171 usec 1865: 175 usec ^C# I tested with Oracle UEK2 which is based on 2.6.39 kernel. Mellanox IB adaptor. 19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) There is something about your setup that must be causing it for you. Can I work with you offline if you are available ? The panic you are hitting is not making sense to me. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-14 13:43 ` Venkat Venkatsubra @ 2013-11-15 2:32 ` Honggang LI 2013-11-19 23:33 ` Venkat Venkatsubra 0 siblings, 1 reply; 18+ messages in thread From: Honggang LI @ 2013-11-15 2:32 UTC (permalink / raw) To: Venkat Venkatsubra, Josh Hunt; +Cc: David Miller, jjolly, LKML, netdev [-- Attachment #1: Type: text/plain, Size: 8977 bytes --] On 11/14/2013 09:43 PM, Venkat Venkatsubra wrote: > > -----Original Message----- > From: Honggang LI [mailto:honli@redhat.com] > Sent: Wednesday, November 13, 2013 6:56 PM > To: Josh Hunt; Venkat Venkatsubra > Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org > Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback > > On 11/14/2013 01:40 AM, Josh Hunt wrote: >> On Wed, Nov 13, 2013 at 9:16 AM, Venkat Venkatsubra >> <venkat.x.venkatsubra@oracle.com> wrote: >>> -----Original Message----- >>> From: Josh Hunt [mailto:joshhunt00@gmail.com] >>> Sent: Tuesday, November 12, 2013 10:25 PM >>> To: David Miller >>> Cc: jjolly@suse.com; LKML; Venkat Venkatsubra; netdev@vger.kernel.org >>> Subject: Re: [PATCH] rds: Error on offset mismatch if not loopback >>> >>> On Tue, Nov 12, 2013 at 10:22 PM, Josh Hunt <joshhunt00@gmail.com> wrote: >>>> On Sat, Sep 22, 2012 at 2:25 PM, David Miller <davem@davemloft.net> wrote: >>>>> From: John Jolly <jjolly@suse.com> >>>>> Date: Fri, 21 Sep 2012 15:32:40 -0600 >>>>> >>>>>> Attempting an rds connection from the IP address of an IPoIB >>>>>> interface to itself causes a kernel panic due to a BUG_ON() being triggered. >>>>>> Making the test less strict allows rds-ping to work without >>>>>> crashing the machine. >>>>>> >>>>>> A local unprivileged user could use this flaw to crash the system. >>>>>> >>>>>> Signed-off-by: John Jolly <jjolly@suse.com> >>>>> Besides the questions being asked of you by Venkat Venkatsubra, >>>>> this patch has another issue. >>>>> >>>>> It has been completely corrupted by your email client, it has >>>>> turned all TAB characters into spaces, making the patch useless. >>>>> >>>>> Please learn how to send a patch unmolested in the body of your >>>>> email. Test it by emailing the patch to yourself, and verifying >>>>> that you can in fact apply the patch you receive in that email. >>>>> Then, and only then, should you consider making a new submission of >>>>> this patch. >>>>> >>>>> Use Documentation/email-clients.txt for guidance. >>>>> -- >>>>> To unsubscribe from this list: send the line "unsubscribe >>>>> linux-kernel" in the body of a message to majordomo@vger.kernel.org >>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>> Please read the FAQ at http://www.tux.org/lkml/ >>>> I think this issue was lost in the shuffle. It appears that redhat, >>>> ubuntu, and oracle are maintaining local patches to resolve this: >>>> >>>> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d636 >>>> 85 >>>> 2be130fa15fa8be10d4704e8 >>>> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >>>> http://ubuntu.5.x6.nabble.com/CVE-2012-2372-RDS-local-ping-DOS-td498 >>>> 53 >>>> 88.html >>>> >>>> Given that Oracle has applied it I'll make the assumption that >>>> Venkat's question was answered at some point. >>>> >>>> David - I can resubmit the patch with the proper signed-off-by and >>>> formatting if you are willing to apply it unless John wants to try >>>> again. I think it's time this got upstream. >>>> >>>> -- >>>> Josh >>> Ugh.. hopefully resending with all the html crap removed... >>> >>> -- >>> Josh >>> >>> Hi Josh, >>> >>> No, I still didn't get an answer for how "off" could be non-zero in case of rds-ping to hit BUG_ON(off % RDS_FRAG_SIZE). >>> Because, rds-ping uses zero byte messages to ping. >>> If you have a test case that reproduces the kernel panic I can try it out and see how that can happen. >>> The Oracle's internal code I checked doesn't have that patch applied. >>> >>> Venkat >> No I don't have a test case. I came across this CVE while doing an >> audit and noticed it was patched in Ubuntu's kernel and other distros, >> but was not in the upstream kernel yet. Quick googling of lkml showed >> that there were at least two attempts to get this patch upstream, but >> both had issues due to not following the proper submission process: >> >> https://lkml.org/lkml/2012/10/22/433 >> https://lkml.org/lkml/2012/9/21/505 >> >> From my searching it appears the initial bug was found by someone at redhat: >> https://bugzilla.redhat.com/show_bug.cgi?id=822754 >> >> I've added Li Honggang the reporter of this issue from Redhat to the >> mail. Hopefully he can share his testcase. > The test case is very simple: > Steps to Reproduce: > 1. yum install -y rds-tools > > 2. [root@rdma3 ~]# ifconfig ib0 | grep 'inet addr' > inet addr:172.31.0.3 Bcast:172.31.0.255 Mask:255.255.255.0 > > 3. [root@rdma3 ~]# /usr/bin/rds-ping 172.31.0.3 <<<< kernel panic (You may need to wait for a few seconds before the kernel panic.) >> and possibly requires certain hardware as Jay writes in the first link above: >> "...some Infiniband HCAs(QLogic, possibly others) the machine will panic..." > This bug can be reproduced with Mellanox HCAs (mlx4_ib.ko and mthca.ko), QLogic HCA (ib_qib.ko). I did not test the QLogic HCA running "ib_ipath.ko". > > As I know the upstream code of RDS is broken. There are *many* RDS bugs. > > Best regards. > Honggang >> I was referring to this oracle commit: >> https://oss.oracle.com/git/?p=redpatch.git;a=commit;h=c7b6a0a1d8d63685 >> 2be130fa15fa8be10d4704e8 >> >> I have no experience with this code. There were a few comments around >> the reset and xmit fns about making sure the caller did certain things >> if not they were racy, but I have no idea if that's coming into play >> here. >> > Hi Honggang, > > I ran rds-ping over local interface for 30 minutes. I stopped it after that. > It didn't hit any panic. > > # ip addr show dev ib0 > 6: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast qlen 1024 > link/infiniband 80:00:00:48:fe:80:00:00:00:00:00:00:00:21:28:00:01:cf:63:db brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff > inet 10.196.4.125/30 brd 10.196.4.127 scope global ib0 > inet6 fe80::221:2800:1cf:63db/64 scope link > valid_lft forever preferred_lft forever > # > > # rds-ping 10.196.4.125 > 1: 170 usec > 2: 171 usec > .... > .... > .... > 1860: 173 usec > 1861: 171 usec > 1862: 177 usec > 1863: 168 usec > 1864: 171 usec > 1865: 175 usec > ^C# > > I tested with Oracle UEK2 which is based on 2.6.39 kernel. Mellanox IB adaptor. > 19:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) > > There is something about your setup that must be causing it for you. > Can I work with you offline if you are available ? > > The panic you are hitting is not making sense to me. > > Venkat Hi, Venkat It seems we are in different time zone. Please contact me via email if you need I do something for this bug. Could you please try upstream kernel 2.6.39. I confirmed that the bug can be reproduced with Mellanox and QLogic HCA when running upstream kernel-2.6.39. [root@rdma01 ~]# ifconfig mlx4_ib1 Ifconfig uses the ioctl access method to get the full address information, which limits hardware addresses to 8 bytes. Because Infiniband address has 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is obsolete! For replacement check ip. mlx4_ib1 Link encap:InfiniBand HWaddr 80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet addr:172.31.2.1 Bcast:172.31.2.255 Mask:255.255.255.0 inet6 addr: fe80::7ae7:d1ff:ff6b:b01/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:5 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [root@rdma01 ~]# rpm -qf /usr/bin/rds-ping rds-tools-2.0.6-3.el6.x86_64 [root@rdma01 ~]# uname -a Linux rdma01.rhts.eng.nay.redhat.com 2.6.39 #1 SMP Thu Nov 14 20:25:45 EST 2013 x86_64 x86_64 x86_64 GNU/Linux [root@rdma01 ~]# ibstat CA 'mlx4_0' CA type: MT26428 Number of ports: 2 Firmware version: 2.8.600 Hardware version: b0 Node GUID: 0x78e7d1ffff6b0b00 System image GUID: 0x78e7d1ffff6b0b03 Port 1: State: Active Physical state: LinkUp Rate: 40 Base lid: 1 LMC: 0 SM lid: 4 Capability mask: 0x02510868 Port GUID: 0x78e7d1ffff6b0b01 Link layer: InfiniBand Port 2: State: Down Physical state: Polling Rate: 70 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510868 Port GUID: 0x78e7d1ffff6b0b02 Link layer: InfiniBand [root@rdma01 ~]# lspci | grep Mellanox 1f:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0) [root@rdma01 ~]# ssh 172.31.2.2 hostname (make sure the IPoIB interface works) rdma02.rhts.eng.nay.redhat.com [root@rdma01 ~]# ssh 172.31.2.1 hostname rdma01.rhts.eng.nay.redhat.com [root@rdma01 ~]# /usr/bin/rds-ping 172.31.2.1 (kernel panic, please see the attachment for console log) [-- Attachment #2: upstream-kernel-2.6.39-rds-ping-panic.log --] [-- Type: text/x-log, Size: 6544 bytes --] RDS/IB: connected to 172.31.2.1 version 3.1 RDS/IB: connected to 172.31.2.1 version 3.1 ------------[ cut here ]------------ kernel BUG at net/rds/ib_send.c:547! invalid opcode: 0000 [#1] SMP last sysfs file: /sys/devices/system/cpu/online CPU 6 Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp scsi_tgt rds_rdma rds_tcp rds ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode cdc_ether usbnet mii serio_raw pcspkr i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg shpchp ioatdma dca i7core_edac edac_core cxgb3 mdio mlx4_ib ib_mad mlx4_en mlx4_core ib_core cxgb4 bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Pid: 136, comm: kworker/u:1 Not tainted 2.6.39 #1 IBM System x3650 M3 -[7945O63]-/00D4062 RIP: 0010:[<ffffffffa036d7f9>] [<ffffffffa036d7f9>] rds_ib_xmit+0xa69/0xaf0 [rds_rdma] RSP: 0018:ffff880271b51c50 EFLAGS: 00010202 RAX: ffff880266dc2000 RBX: ffff880271639a00 RCX: 0000000000000000 RDX: 0000000000000030 RSI: ffff880471e81000 RDI: ffff880270997cf0 RBP: ffff880271b51d30 R08: 0000000000000fd0 R09: ffff880471e81190 R10: 00000000ffffffff R11: 0000000000000001 R12: ffff880471e81000 R13: ffff880471e81000 R14: 0000000000000000 R15: ffff880271b51d90 FS: 0000000000000000(0000) GS:ffff88047fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00002b302d2aa080 CR3: 0000000001a03000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/u:1 (pid: 136, threadinfo ffff880271b50000, task ffff880271ae60c0) Stack: 0000000000000400 0000000000000001 0000000000000001 0000000a0000000a 0000000000000128 ffffffff8114747c 0000000000000002 0000000000000001 ffff880271ae60c0 ffffffff812402f0 ffff880471e81000 0000003000000002 Call Trace: [<ffffffff8114747c>] ? __kmalloc+0x21c/0x230 [<ffffffff812402f0>] ? sg_init_table+0x30/0x50 [<ffffffffa0340df2>] ? rds_message_alloc_sgs+0x62/0xa0 [rds] [<ffffffffa0341224>] ? rds_message_map_pages+0xa4/0x110 [rds] [<ffffffffa0342f5b>] rds_send_xmit+0x38b/0x6e0 [rds] [<ffffffffa0344010>] ? rds_recv_worker+0xc0/0xc0 [rds] [<ffffffffa0344045>] rds_send_worker+0x35/0xc0 [rds] [<ffffffff8107d0f9>] process_one_work+0x129/0x430 [<ffffffff8107f4ab>] worker_thread+0x17b/0x3c0 [<ffffffff8107f330>] ? manage_workers+0x120/0x120 [<ffffffff81084566>] kthread+0x96/0xa0 [<ffffffff814e4104>] kernel_thread_helper+0x4/0x10 [<ffffffff810844d0>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff814e4100>] ? gs_change+0x13/0x13 Code: ff ff e9 b1 fe ff ff 48 8b 0d c4 fe 69 e1 48 89 8d 70 ff ff ff e9 71 ff ff ff 83 bd 7c ff ff ff 00 0f 84 f4 f5 ff ff 0f 0b eb fe <0f> 0b eb fe 44 8b 8d 48 ff ff ff 41 b7 01 e9 51 f6 ff ff 0f 0b RIP [<ffffffffa036d7f9>] rds_ib_xmit+0xa69/0xaf0 [rds_rdma] RSP <ffff880271b51c50> ---[ end trace de7f8972e25cd611 ]--- BUG: unable to handle kernel paging request at fffffffffffffff8 IP: [<ffffffff810840c0>] kthread_data+0x10/0x20 PGD 1a05067 PUD 1a06067 PMD 0 Oops: 0000 [#2] SMP last sysfs file: /sys/devices/system/cpu/online CPU 6 Modules linked in: ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp scsi_tgt rds_rdma rds_tcp rds ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_addr ipv6 ib_sa microcode cdc_ether usbnet mii serio_raw pcspkr i2c_i801 i2c_core iTCO_wdt iTCO_vendor_support sg shpchp ioatdma dca i7core_edac edac_core cxgb3 mdio mlx4_ib ib_mad mlx4_en mlx4_core ib_core cxgb4 bnx2 ext4 mbcache jbd2 sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Pid: 136, comm: kworker/u:1 Tainted: G D 2.6.39 #1 IBM System x3650 M3 -[7945O63]-/00D4062 RIP: 0010:[<ffffffff810840c0>] [<ffffffff810840c0>] kthread_data+0x10/0x20 RSP: 0018:ffff880271b51918 EFLAGS: 00010096 RAX: 0000000000000000 RBX: 0000000000000006 RCX: ffff880271ae60c0 RDX: 0000000000009bf5 RSI: 0000000000000006 RDI: ffff880271ae60c0 RBP: ffff880271b51918 R08: ffff880271ae6570 R09: dead000000200200 R10: 00000000ffffffff R11: 0000000000000007 R12: ffff880271ae6658 R13: 0000000000000006 R14: 0000000000000006 R15: 0000000000000006 FS: 0000000000000000(0000) GS:ffff88047fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: fffffffffffffff8 CR3: 0000000001a03000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/u:1 (pid: 136, threadinfo ffff880271b50000, task ffff880271ae60c0) Stack: ffff880271b51938 ffffffff8107e385 ffff880272d67b40 ffff88047fc150c0 ffff880271b519d8 ffffffff814d882e ffff880271b51978 ffff880271ae60c0 ffff880271ae60c0 00000000000150c0 ffff880271b51fd8 ffff880271b50010 Call Trace: [<ffffffff8107e385>] wq_worker_sleeping+0x15/0xa0 [<ffffffff814d882e>] schedule+0x49e/0x9c0 [<ffffffff810675d1>] do_exit+0x271/0x430 [<ffffffff814dc08b>] oops_end+0xab/0xf0 [<ffffffff8100e7cb>] die+0x5b/0x90 [<ffffffff814db994>] do_trap+0xc4/0x170 [<ffffffff8100c695>] do_invalid_op+0x95/0xb0 [<ffffffffa036d7f9>] ? rds_ib_xmit+0xa69/0xaf0 [rds_rdma] [<ffffffff814e3f7b>] invalid_op+0x1b/0x20 [<ffffffffa036d7f9>] ? rds_ib_xmit+0xa69/0xaf0 [rds_rdma] [<ffffffff8114747c>] ? __kmalloc+0x21c/0x230 [<ffffffff812402f0>] ? sg_init_table+0x30/0x50 [<ffffffffa0340df2>] ? rds_message_alloc_sgs+0x62/0xa0 [rds] [<ffffffffa0341224>] ? rds_message_map_pages+0xa4/0x110 [rds] [<ffffffffa0342f5b>] rds_send_xmit+0x38b/0x6e0 [rds] [<ffffffffa0344010>] ? rds_recv_worker+0xc0/0xc0 [rds] [<ffffffffa0344045>] rds_send_worker+0x35/0xc0 [rds] [<ffffffff8107d0f9>] process_one_work+0x129/0x430 [<ffffffff8107f4ab>] worker_thread+0x17b/0x3c0 [<ffffffff8107f330>] ? manage_workers+0x120/0x120 [<ffffffff81084566>] kthread+0x96/0xa0 [<ffffffff814e4104>] kernel_thread_helper+0x4/0x10 [<ffffffff810844d0>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff814e4100>] ? gs_change+0x13/0x13 Code: 1f 44 00 00 65 48 8b 04 25 80 cc 00 00 48 8b 80 40 05 00 00 8b 40 f0 c9 c3 66 90 55 48 89 e5 0f 1f 44 00 00 48 8b 87 40 05 00 00 8b 40 f8 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f RIP [<ffffffff810840c0>] kthread_data+0x10/0x20 RSP <ffff880271b51918> CR2: fffffffffffffff8 ---[ end trace de7f8972e25cd612 ]--- Fixing recursive fault but reboot is needed! ^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-15 2:32 ` Honggang LI @ 2013-11-19 23:33 ` Venkat Venkatsubra 2013-11-20 18:09 ` Venkat Venkatsubra 0 siblings, 1 reply; 18+ messages in thread From: Venkat Venkatsubra @ 2013-11-19 23:33 UTC (permalink / raw) To: Honggang LI, Josh Hunt; +Cc: David Miller, jjolly, LKML, netdev We now have lot more information than we did before. When sending a "congestion update" in rds_ib_xmit() we are now returning an incorrect number as bytes sent: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); scat = &rm->data.op_sg[sg]; ret = sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; ret = min_t(int, ret, scat->length - conn->c_xmit_data_off); return ret; } It returns min(8240, 4096-0) i.e. 4096 bytes. The caller rds_send_xmit() is made to think a partial message (4096 out of 8240) was sent. It calls rds_ib_xmit() again with a data offset "off" of 4096-48 (rds header) (=4048 bytes). And we hit the BUG_ON. The reason I didn't hit the panic on my test on Oracle UEK2 which is based on 2.6.39 kernel is it had it like this: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; } (So it wasn't 100% 2.6.39 ;-). ) It returned 8240 bytes. The caller rds_send_xmit decides the full message was sent (48 byte header + 4096 data + 4096 data). And it worked. Then I found this info on the change that was done upstream which now causes the panic: http://marc.info/?l=linux-netdev&m=129908332903057 http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6094628bfd94323fc1cea05ec2c6affd98c18f7f Will investigate more into which problem the above change addressed. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-19 23:33 ` Venkat Venkatsubra @ 2013-11-20 18:09 ` Venkat Venkatsubra 2013-11-20 18:54 ` David Miller 0 siblings, 1 reply; 18+ messages in thread From: Venkat Venkatsubra @ 2013-11-20 18:09 UTC (permalink / raw) To: Honggang LI, Josh Hunt; +Cc: David Miller, jjolly, LKML, netdev -----Original Message----- From: Venkat Venkatsubra Sent: Tuesday, November 19, 2013 5:33 PM To: Honggang LI; Josh Hunt Cc: David Miller; jjolly@suse.com; LKML; netdev@vger.kernel.org Subject: RE: [PATCH] rds: Error on offset mismatch if not loopback We now have lot more information than we did before. When sending a "congestion update" in rds_ib_xmit() we are now returning an incorrect number as bytes sent: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); scat = &rm->data.op_sg[sg]; ret = sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; ret = min_t(int, ret, scat->length - conn->c_xmit_data_off); return ret; } It returns min(8240, 4096-0) i.e. 4096 bytes. The caller rds_send_xmit() is made to think a partial message (4096 out of 8240) was sent. It calls rds_ib_xmit() again with a data offset "off" of 4096-48 (rds header) (=4048 bytes). And we hit the BUG_ON. The reason I didn't hit the panic on my test on Oracle UEK2 which is based on 2.6.39 kernel is it had it like this: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; } (So it wasn't 100% 2.6.39 ;-). ) It returned 8240 bytes. The caller rds_send_xmit decides the full message was sent (48 byte header + 4096 data + 4096 data). And it worked. Then I found this info on the change that was done upstream which now causes the panic: http://marc.info/?l=linux-netdev&m=129908332903057 http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6094628bfd94323fc1cea05ec2c6affd98c18f7f Will investigate more into which problem the above change addressed. Venkat -- Looks like the fix pointed to by the above link is for a panic on a PPC system with a PAGE_SIZE of 64Kbytes. I think the sequence it was going through before that fix was: /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; } rds_ib_xmit returns 8240 rds_send_xmit : c_xmit_data_off = 0 + 8240 - 48 (rds header the first time) = 8196 c_xmit_data_off < 65536 (sg->length) calls rds_ib_xmit again rds_ib_xmit returns 8240 rds_send_xmit: c_xmit_data_off = 8192+8240 = 16432 and calls rds_ib_xmit rds_ib_xmit : returns 8240 rds_send_xmit: c_xmit_data_off 24672 and calls rds_ib_xmit ... ... and so on till rds_send_xmit: c_xmit_data_off 57632 and calls rds_ib_xmit rds_ib_xmit: returns 8240 On the last iteration it hits the below BUG_ON in rds_send_xmit. while (ret) { tmp = min_t(int, ret, sg->length - conn->c_xmit_data_off); [tmp = 7904] conn->c_xmit_data_off += tmp; [c_xmit_data_off = 65536] ret -= tmp; [ret = 8240-7904 = 336] if (conn->c_xmit_data_off == sg->length) { conn->c_xmit_data_off = 0; sg++; conn->c_xmit_sg++; BUG_ON(ret != 0 && conn->c_xmit_sg == rm->data.op_nents); } } Since the congestion update over loopback is not actually transmitted as a message, the multiple iterations we see in the case of ppc is unnecessary. All that rds_ib_xmit needs to do is return a number of bytes that will tell the caller that we are done with this message. This might fix the original problem without introducing the current panic: /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); scat = &rm->data.op_sg[sg]; ret = max_t(int, RDS_CONG_MAP_BYTES, scat->length); return ret + sizeof(struct rds_header); } It will return 8240 when PAGE_SIZE is 4k and 64k+48 in case of ppc when scat->length is 64k and be done with one iteration of rds_send_xmit/rds_ib_xmit loop. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-20 18:09 ` Venkat Venkatsubra @ 2013-11-20 18:54 ` David Miller 2013-11-20 21:28 ` Venkat Venkatsubra 0 siblings, 1 reply; 18+ messages in thread From: David Miller @ 2013-11-20 18:54 UTC (permalink / raw) To: venkat.x.venkatsubra; +Cc: honli, joshhunt00, jjolly, linux-kernel, netdev Why are you posting this message a second time? ^ permalink raw reply [flat|nested] 18+ messages in thread
* RE: [PATCH] rds: Error on offset mismatch if not loopback 2013-11-20 18:54 ` David Miller @ 2013-11-20 21:28 ` Venkat Venkatsubra 0 siblings, 0 replies; 18+ messages in thread From: Venkat Venkatsubra @ 2013-11-20 21:28 UTC (permalink / raw) To: David Miller; +Cc: honli, joshhunt00, jjolly, linux-kernel, netdev > Why are you posting this message a second time? Reposting just the contents of the second message in case it got missed the previous time. Looks like the fix pointed to by the previous link is for a panic on a PPC system with a PAGE_SIZE of 64Kbytes. I think the sequence it was going through before that fix was: /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; } rds_ib_xmit returns 8240 rds_send_xmit : c_xmit_data_off = 0 + 8240 - 48 (rds header the first time) = 8196 c_xmit_data_off < 65536 (sg->length) calls rds_ib_xmit again rds_ib_xmit returns 8240 rds_send_xmit: c_xmit_data_off = 8192+8240 = 16432 and calls rds_ib_xmit rds_ib_xmit : returns 8240 rds_send_xmit: c_xmit_data_off 24672 and calls rds_ib_xmit ... ... and so on till rds_send_xmit: c_xmit_data_off 57632 and calls rds_ib_xmit rds_ib_xmit: returns 8240 On the last iteration it hits the below BUG_ON in rds_send_xmit. while (ret) { tmp = min_t(int, ret, sg->length - conn->c_xmit_data_off); [tmp = 7904] conn->c_xmit_data_off += tmp; [c_xmit_data_off = 65536] ret -= tmp; [ret = 8240-7904 = 336] if (conn->c_xmit_data_off == sg->length) { conn->c_xmit_data_off = 0; sg++; conn->c_xmit_sg++; BUG_ON(ret != 0 && conn->c_xmit_sg == rm->data.op_nents); } } Since the congestion update over loopback is not actually transmitted as a message, the multiple iterations we see in the case of ppc is unnecessary. All that rds_ib_xmit needs to do is return a number of bytes that will tell the caller that we are done with this message. This might fix the original problem without introducing the current panic: /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); scat = &rm->data.op_sg[sg]; ret = max_t(int, RDS_CONG_MAP_BYTES, scat->length); return ret + sizeof(struct rds_header); } It will return 8240 when PAGE_SIZE is 4k and 64k+48 in case of ppc when scat->length is 64k and be done with one iteration of rds_send_xmit/rds_ib_xmit loop. Venkat ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback [not found] ` <CAKA=qzac9=UhLF_Z4FnnH+sR7xvkDux4oayC6dPYe=hMLsDxRg@mail.gmail.com> 2013-11-13 4:24 ` Josh Hunt @ 2013-11-13 6:09 ` David Miller 1 sibling, 0 replies; 18+ messages in thread From: David Miller @ 2013-11-13 6:09 UTC (permalink / raw) To: joshhunt00; +Cc: jjolly, linux-kernel, venkat.x.venkatsubra, netdev From: Josh Hunt <joshhunt00@gmail.com> Date: Tue, 12 Nov 2013 22:22:11 -0600 > David - I can resubmit the patch with the proper signed-off-by and > formatting if you are willing to apply it unless John wants to try again. I > think it's time this got upstream. Nothing is going to happen until the patch is submitted properly, so just do, don't ask. ^ permalink raw reply [flat|nested] 18+ messages in thread
* [PATCH] rds: Error on offset mismatch if not loopback @ 2012-09-20 7:11 John Jolly 2012-09-21 17:20 ` David Miller 0 siblings, 1 reply; 18+ messages in thread From: John Jolly @ 2012-09-20 7:11 UTC (permalink / raw) To: linux-kernel; +Cc: Venkat Venkatsubra, netdev Attempting an rds connection from the IP address of an IPoIB interface to itself causes a kernel panic due to a BUG_ON() being triggered. Making the test less strict allows rds-ping to work without crashing the machine. A local unprivileged user could use this flaw to crash the sytem. --- net/rds/ib_send.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/net/rds/ib_send.c b/net/rds/ib_send.c index e590949..7920c85 100644 --- a/net/rds/ib_send.c +++ b/net/rds/ib_send.c @@ -544,7 +544,7 @@ int rds_ib_xmit(struct rds_connection *conn, struct rds_message *rm, int flow_controlled = 0; int nr_sig = 0; - BUG_ON(off % RDS_FRAG_SIZE); + BUG_ON(!conn->c_loopback && off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ -- 1.7.7 ^ permalink raw reply related [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2012-09-20 7:11 John Jolly @ 2012-09-21 17:20 ` David Miller 2012-09-21 21:28 ` John Jolly 0 siblings, 1 reply; 18+ messages in thread From: David Miller @ 2012-09-21 17:20 UTC (permalink / raw) To: jjolly; +Cc: linux-kernel, venkat.x.venkatsubra, netdev From: John Jolly <jjolly@suse.com> Date: Thu, 20 Sep 2012 01:11:34 -0600 > Attempting an rds connection from the IP address of an IPoIB interface > to itself causes a kernel panic due to a BUG_ON() being triggered. Making > the test less strict allows rds-ping to work without crashing the machine. > > A local unprivileged user could use this flaw to crash the sytem. Please read Documentation/SubmittingPatches to learn how to properly submit a change, in particular your patch submission was missing a proper signoff. Thanks. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: [PATCH] rds: Error on offset mismatch if not loopback 2012-09-21 17:20 ` David Miller @ 2012-09-21 21:28 ` John Jolly 0 siblings, 0 replies; 18+ messages in thread From: John Jolly @ 2012-09-21 21:28 UTC (permalink / raw) To: David Miller; +Cc: linux-kernel, venkat.x.venkatsubra, netdev On Fri, Sep 21, 2012 at 01:20:45PM -0400, David Miller wrote: > From: John Jolly <jjolly@suse.com> > Date: Thu, 20 Sep 2012 01:11:34 -0600 > > > Attempting an rds connection from the IP address of an IPoIB interface > > to itself causes a kernel panic due to a BUG_ON() being triggered. Making > > the test less strict allows rds-ping to work without crashing the machine. > > > > A local unprivileged user could use this flaw to crash the sytem. > > Please read Documentation/SubmittingPatches to learn how to properly > submit a change, in particular your patch submission was missing a > proper signoff. Thanks for catching that. Resubmitting with proper signoff. > > Thanks. > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2013-11-20 21:28 UTC | newest]
Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-09-21 21:32 [PATCH] rds: Error on offset mismatch if not loopback John Jolly
2012-09-21 21:38 ` Venkat Venkatsubra
2012-09-22 19:25 ` David Miller
[not found] ` <CAKA=qzac9=UhLF_Z4FnnH+sR7xvkDux4oayC6dPYe=hMLsDxRg@mail.gmail.com>
2013-11-13 4:24 ` Josh Hunt
2013-11-13 15:16 ` Venkat Venkatsubra
2013-11-13 17:40 ` Josh Hunt
2013-11-14 0:55 ` Honggang LI
2013-11-14 1:27 ` Josh Hunt
2013-11-14 13:43 ` Venkat Venkatsubra
2013-11-15 2:32 ` Honggang LI
2013-11-19 23:33 ` Venkat Venkatsubra
2013-11-20 18:09 ` Venkat Venkatsubra
2013-11-20 18:54 ` David Miller
2013-11-20 21:28 ` Venkat Venkatsubra
2013-11-13 6:09 ` David Miller
-- strict thread matches above, loose matches on Subject: below --
2012-09-20 7:11 John Jolly
2012-09-21 17:20 ` David Miller
2012-09-21 21:28 ` John Jolly
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).