From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754525AbcAHKDR (ORCPT ); Fri, 8 Jan 2016 05:03:17 -0500 Received: from userp1040.oracle.com ([156.151.31.81]:35520 "EHLO userp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754533AbcAHKDM (ORCPT ); Fri, 8 Jan 2016 05:03:12 -0500 Subject: Re: [PATCH] net: add per device sg_max_frags for skb To: Eric Dumazet References: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> Cc: "David S. Miller" , Alexey Kuznetsov , James Morris , Hideaki YOSHIFUJI , Patrick McHardy , Alexei Starovoitov , Jiri Pirko , Eric Dumazet , Daniel Borkmann , Nicolas Dichtel , "Eric W. Biederman " , Salam Noureddine , Jarod Wilson , Toshiaki Makita , Julian Anastasov , Ying Xue , Craig Gallek , Mel Gorman , "\" hannes\"@stressinduktion.org " , Edward Jee , Julia Lawall , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Haakon Bugge , Knut Omang , Wei Lin Guay , Santosh Shilimkar , Yuval Shaia From: Hans Westgaard Ry Message-ID: <568F890D.70004@oracle.com> Date: Fri, 8 Jan 2016 11:01:49 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-Source-IP: aserv0022.oracle.com [141.146.126.234] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 01/06/2016 03:05 PM, Eric Dumazet wrote: > On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote: >> Devices may have limits on the number of fragments in an skb they >> support. Current codebase uses a constant as maximum for number of >> fragments (MAX_SKB_FRAGS) one skb can hold and use. >> >> When enabling scatter/gather and running traffic with many small >> messages the codebase uses the maximum number of fragments and thereby >> violates the max for certain devices. >> >> An example of such a violation is when running IPoIB on a HCA >> supporting 16 SGE on an architecture with 4K pagesize. The >> MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another >> segment we end up with send_requests with 18 SGE resulting in >> kernel-panic. >> >> The patch allows the device to limit the maximum number fragments used >> in one skb. >> >> The functionality corresponds to gso_max_size/gso_max_segs for gso. > Unfortunately this is not the right place to fix this issue. > > Think about forwarding workloads, where the SKB is cooked by GRO engine. > > Anyway, local TCP stack uses 32KB page fragments, so typical skb has no > more than 3 frags. > > Look at ndo_features_check(), where the problematic device driver can > add its logic. > > > I've had a look at ndo_features_check and understand that I could supply my own version of the routine, but I wasn't able to figure out how that would solve my problem. As far as I can see the routine is not called in the part of code handling scatter/gather. Could you help out with more info? Hans From mboxrd@z Thu Jan 1 00:00:00 1970 From: Hans Westgaard Ry Subject: Re: [PATCH] net: add per device sg_max_frags for skb Date: Fri, 8 Jan 2016 11:01:49 +0100 Message-ID: <568F890D.70004@oracle.com> References: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "David S. Miller" , Alexey Kuznetsov , James Morris , Hideaki YOSHIFUJI , Patrick McHardy , Alexei Starovoitov , Jiri Pirko , Eric Dumazet , Daniel Borkmann , Nicolas Dichtel , "Eric W. Biederman " , Salam Noureddine , Jarod Wilson , Toshiaki Makita , Julian Anastasov , Ying Xue , Craig Gallek , Mel Gorman , "\" hannes\"@stressinduktion.org " , Edward Jee , Julia Lawall , netdev@vger.kernel.org, linux-kernel@vger.ke To: Eric Dumazet Return-path: In-Reply-To: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> Sender: linux-kernel-owner@vger.kernel.org List-Id: netdev.vger.kernel.org On 01/06/2016 03:05 PM, Eric Dumazet wrote: > On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote: >> Devices may have limits on the number of fragments in an skb they >> support. Current codebase uses a constant as maximum for number of >> fragments (MAX_SKB_FRAGS) one skb can hold and use. >> >> When enabling scatter/gather and running traffic with many small >> messages the codebase uses the maximum number of fragments and thereby >> violates the max for certain devices. >> >> An example of such a violation is when running IPoIB on a HCA >> supporting 16 SGE on an architecture with 4K pagesize. The >> MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another >> segment we end up with send_requests with 18 SGE resulting in >> kernel-panic. >> >> The patch allows the device to limit the maximum number fragments used >> in one skb. >> >> The functionality corresponds to gso_max_size/gso_max_segs for gso. > Unfortunately this is not the right place to fix this issue. > > Think about forwarding workloads, where the SKB is cooked by GRO engine. > > Anyway, local TCP stack uses 32KB page fragments, so typical skb has no > more than 3 frags. > > Look at ndo_features_check(), where the problematic device driver can > add its logic. > > > I've had a look at ndo_features_check and understand that I could supply my own version of the routine, but I wasn't able to figure out how that would solve my problem. As far as I can see the routine is not called in the part of code handling scatter/gather. Could you help out with more info? Hans