From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752746AbcAFOFU (ORCPT ); Wed, 6 Jan 2016 09:05:20 -0500 Received: from mail-pf0-f180.google.com ([209.85.192.180]:34903 "EHLO mail-pf0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752056AbcAFOFR (ORCPT ); Wed, 6 Jan 2016 09:05:17 -0500 Message-ID: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> Subject: Re: [PATCH] net: add per device sg_max_frags for skb From: Eric Dumazet To: Hans Westgaard Ry Cc: "David S. Miller" , Alexey Kuznetsov , James Morris , Hideaki YOSHIFUJI , Patrick McHardy , Alexei Starovoitov , Jiri Pirko , Eric Dumazet , Daniel Borkmann , Nicolas Dichtel , "\"\"Eric W. Biederman\"\"" , Salam Noureddine , Jarod Wilson , Toshiaki Makita , Julian Anastasov , Ying Xue , Craig Gallek , Mel Gorman , "\"\"hannes@stressinduktion.org\"\"" , Edward Jee , Julia Lawall , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Haakon Bugge , Knut Omang , Wei Lin Guay , Santosh Shilimkar , Yuval Shaia Date: Wed, 06 Jan 2016 06:05:15 -0800 In-Reply-To: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> References: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.10.4-0ubuntu2 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote: > Devices may have limits on the number of fragments in an skb they > support. Current codebase uses a constant as maximum for number of > fragments (MAX_SKB_FRAGS) one skb can hold and use. > > When enabling scatter/gather and running traffic with many small > messages the codebase uses the maximum number of fragments and thereby > violates the max for certain devices. > > An example of such a violation is when running IPoIB on a HCA > supporting 16 SGE on an architecture with 4K pagesize. The > MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another > segment we end up with send_requests with 18 SGE resulting in > kernel-panic. > > The patch allows the device to limit the maximum number fragments used > in one skb. > > The functionality corresponds to gso_max_size/gso_max_segs for gso. Unfortunately this is not the right place to fix this issue. Think about forwarding workloads, where the SKB is cooked by GRO engine. Anyway, local TCP stack uses 32KB page fragments, so typical skb has no more than 3 frags. Look at ndo_features_check(), where the problematic device driver can add its logic. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Eric Dumazet Subject: Re: [PATCH] net: add per device sg_max_frags for skb Date: Wed, 06 Jan 2016 06:05:15 -0800 Message-ID: <1452089115.8255.171.camel@edumazet-glaptop2.roam.corp.google.com> References: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: "David S. Miller" , Alexey Kuznetsov , James Morris , Hideaki YOSHIFUJI , Patrick McHardy , Alexei Starovoitov , Jiri Pirko , Eric Dumazet , Daniel Borkmann , Nicolas Dichtel , "\"\"Eric W. Biederman\"\"" , Salam Noureddine , Jarod Wilson , Toshiaki Makita , Julian Anastasov , Ying Xue , Craig Gallek , Mel Gorman , "\"\"hannes@stressinduktion.org\"\"" , Edward Jee , Julia Lawall , netdev@vger.kernel.org, linux-kerne To: Hans Westgaard Ry Return-path: Received: from mail-pf0-f180.google.com ([209.85.192.180]:34903 "EHLO mail-pf0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752056AbcAFOFR (ORCPT ); Wed, 6 Jan 2016 09:05:17 -0500 In-Reply-To: <1452086182-26748-1-git-send-email-hans.westgaard.ry@oracle.com> Sender: netdev-owner@vger.kernel.org List-ID: On Wed, 2016-01-06 at 14:16 +0100, Hans Westgaard Ry wrote: > Devices may have limits on the number of fragments in an skb they > support. Current codebase uses a constant as maximum for number of > fragments (MAX_SKB_FRAGS) one skb can hold and use. > > When enabling scatter/gather and running traffic with many small > messages the codebase uses the maximum number of fragments and thereby > violates the max for certain devices. > > An example of such a violation is when running IPoIB on a HCA > supporting 16 SGE on an architecture with 4K pagesize. The > MAX_SKB_FRAGS will be 17 (64K/4K+1) and because IPoIB adds yet another > segment we end up with send_requests with 18 SGE resulting in > kernel-panic. > > The patch allows the device to limit the maximum number fragments used > in one skb. > > The functionality corresponds to gso_max_size/gso_max_segs for gso. Unfortunately this is not the right place to fix this issue. Think about forwarding workloads, where the SKB is cooked by GRO engine. Anyway, local TCP stack uses 32KB page fragments, so typical skb has no more than 3 frags. Look at ndo_features_check(), where the problematic device driver can add its logic.