From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Anders K. Pedersen | Cohaesio" Subject: Re: Linux 4.12+ memory leak on router with i40e NICs Date: Thu, 5 Oct 2017 05:19:13 +0000 Message-ID: <1507180753.20182.8.camel@cohaesio.com> References: <1507121766.30720.4.camel@cohaesio.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Cc: "netdev@vger.kernel.org" , "intel-wired-lan@lists.osuosl.org" , "alexander.h.duyck@intel.com" To: "alexander.duyck@gmail.com" Return-path: Received: from bender.cohaesio.com ([212.97.128.130]:39974 "EHLO mail.cohaesio.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750851AbdJEFTP (ORCPT ); Thu, 5 Oct 2017 01:19:15 -0400 In-Reply-To: Content-Language: en-US Content-ID: <28E24AA3FB41A14EACDF2DEAA58CB034@surftown.com> Sender: netdev-owner@vger.kernel.org List-ID: T24gb25zLCAyMDE3LTEwLTA0IGF0IDA4OjMyIC0wNzAwLCBBbGV4YW5kZXIgRHV5Y2sgd3JvdGU6 DQo+IE9uIFdlZCwgT2N0IDQsIDIwMTcgYXQgNTo1NiBBTSwgQW5kZXJzIEsuIFBlZGVyc2VuIHwg Q29oYWVzaW8NCj4gPGFrcEBjb2hhZXNpby5jb20+IHdyb3RlOg0KPiA+IEhlbGxvLA0KPiA+IA0K PiA+IEFmdGVyIHVwZGF0aW5nIG9uZSBvZiBvdXIgTGludXggYmFzZWQgcm91dGVycyB0byBrZXJu ZWwgNC4xMyBpdA0KPiA+IGJlZ2FuDQo+ID4gbGVha2luZyBtZW1vcnkgcXVpdGUgZmFzdCAoYWJv dXQgMSBHQiBldmVyeSBoYWxmIGhvdXIpLiBUbyBuYXJyb3cNCj4gPiB3ZQ0KPiA+IHRyaWVkIHZh cmlvdXMga2VybmVsIHZlcnNpb25zIGFuZCBmb3VuZCB0aGF0IDQuMTEuMTIgaXMgb2theSwgd2hp bGUNCj4gPiA0LjEyIGFsc28gbGVha3MsIHNvIHdlIGRpZCBhIGJpc2VjdGlvbiBiZXR3ZWVuIDQu MTEgYW5kIDQuMTIuDQo+ID4gDQo+ID4gVGhlIGZpcnN0IGJpc2VjdGlvbiBlbmRlZCBhdA0KPiA+ ICJbNjk2NGU1M2Y1NTgzN2IwYzQ5ZWQ2MGQzNjY1NmQyZTBlZTRmYzI3Yl0gaTQwZTogZml4IGhh bmRsaW5nIG9mDQo+ID4gSFcNCj4gPiBBVFIgZXZpY3Rpb24iLCB3aGljaCBmaXhlcyBzb21lIGZs YWcgaGFuZGxpbmcgdGhhdCB3YXMgYnJva2VuIGJ5DQo+ID4gNDc5OTRjMTE5YTM2ICJpNDBlOiBy ZW1vdmUgaHdfZGlzYWJsZWRfZmxhZ3MgaW4gZmF2b3Igb2YgdXNpbmcNCj4gPiBzZXBhcmF0ZQ0K PiA+IGZsYWcgYml0cyIsIHNvIEkgZGlkIGEgc2Vjb25kIGJpc2VjdGlvbiwgd2hlcmUgSSBhZGRl ZCA2OTY0ZTUzZjU1ODMNCj4gPiAiaTQwZTogZml4IGhhbmRsaW5nIG9mIEhXIEFUUiBldmljdGlv biIgdG8gdGhlIHN0ZXBzIHRoYXQgaGFkDQo+ID4gNDc5OTRjMTE5YTM2ICJpNDBlOiByZW1vdmUg aHdfZGlzYWJsZWRfZmxhZ3MgaW4gZmF2b3Igb2YgdXNpbmcNCj4gPiBzZXBhcmF0ZQ0KPiA+IGZs YWcgYml0cyIgaW4gdGhlbS4NCj4gPiANCj4gPiBUaGUgc2Vjb25kIGJpc2VjdGlvbiBlbmRlZCBh dA0KPiA+ICJbMGU2MjZmZjdjY2JmYzQzYzZjYzRhZWVhNjExYzQwYjg5OTY4MjM4Ml0gaTQwZTog Rml4IHN1cHBvcnQgZm9yDQo+ID4gZmxvdw0KPiA+IGRpcmVjdG9yIHByb2dyYW1taW5nIHN0YXR1 cyIsIHdoZXJlIEkgZG9uJ3Qgc2VlIGFueSBvYnZpb3VzDQo+ID4gcHJvYmxlbXMsDQo+ID4gc28g SSdtIGhvcGluZyBmb3Igc29tZSBhc3Npc3RhbmNlLg0KPiA+IA0KPiA+IFRoZSByb3V0ZXIgaXMg YSBQb3dlckVkZ2UgUjczMCBzZXJ2ZXIgKEhhc3dlbGwgYmFzZWQpIHdpdGggdGhyZWUNCj4gPiBJ bnRlbA0KPiA+IE5JQ3MgKGFsbCB1c2luZyB0aGUgaTQwZSBkcml2ZXIpOg0KPiA+IA0KPiA+IFg3 MTAgcXVhZCBwb3J0IDEwIEdiRSBTRlArOiBldGgwIGV0aDEgZXRoMiBldGgzDQo+ID4gWDcxMCBx dWFkIHBvcnQgMTAgR2JFIFNGUCs6IGV0aDQgZXRoNSBldGg2IGV0aDcNCj4gPiBYTDcxMCBkdWFs IHBvcnQgNDAgR2JFIFFTRlArOiBldGg4IGV0aDkNCj4gPiANCj4gPiBUaGUgTklDcyBhcmUgYWdn cmVnYXRlZCB3aXRoIExBQ1Agd2l0aCB0aGUgdGVhbSBkcml2ZXI6DQo+ID4gDQo+ID4gdGVhbTA6 IGV0aDkgKDQwIEdiRSBzZWxlY3RlZCBwcmltYXJ5KSwgYW5kIGV0aDMsIGV0aDcgKDEwIEdiRSBu b24tDQo+ID4gc2VsZWN0ZWQgYmFja3VwcykNCj4gPiB0ZWFtMTogZXRoMCwgZXRoMSwgZXRoNCwg ZXRoNSAoYWxsIDEwIEdiRSBzZWxlY3RlZCkNCj4gPiANCj4gPiB0ZWFtMCBpcyB1c2VkIGZvciBp bnRlcm5hbCBuZXR3b3JrcyBhbmQgaGFzIG9uZSB1bnRhZ2dlZCBhbmQgZm91cg0KPiA+IHRhZ2dl ZCBWTEFOIGludGVyZmFjZXMsIHdoaWxlIHRlYW0xIGhhcyBhbiBleHRlcm5hbCB1cGxpbmsNCj4g PiBjb25uZWN0aW9uDQo+ID4gd2l0aG91dCBhbnkgVkxBTnMuDQo+ID4gDQo+ID4gVGhlIHJvdXRl ciBydW5zIGFuIGVCR1Agc2Vzc2lvbiBvbiB0ZWFtMSB0byBvbmUgb2Ygb3VyIHVwbGlua3MsIGFu ZA0KPiA+IGlCR1AgdmlhIHRlYW0wIHRvIG91ciBvdGhlciBib3JkZXIgcm91dGVycy4gSXQgYWxz byBydW5zIE9TUEYgb24NCj4gPiB0aGUNCj4gPiBpbnRlcm5hbCBWTEFOcyBvbiB0ZWFtMC4gT25l IHRoaW5nIEkndmUgbm90aWNlZCBpcyB0aGF0IHdoZW4gT1NQRg0KPiA+IGlzDQo+ID4gbm90IGFu bm91bmNpbmcgYSBkZWZhdWx0IGdhdGV3YXkgdG8gdGhlIGludGVybmFsIG5ldHdvcmtzLCBzbyB0 aGVyZQ0KPiA+IGlzDQo+ID4gYWxtb3N0IG5vIHRyYWZmaWMgY29taW5nIGluIG9uIHRlYW0wIGFu ZCBvdXQgb24gdGVhbTEsIGJ1dCBzdGlsbA0KPiA+IHBsZW50eQ0KPiA+IG9mIHRyYWZmaWMgY29t aW5nIGluIG9uIHRlYW0xIGFuZCBvdXQgdmlhIHRlYW0wLCB0aGVyZSdzIG5vIG1lbW9yeQ0KPiA+ IGxlYWsNCj4gPiAob3IgYXQgbGVhc3QgaXQgaXMgc28gc21hbGwgdGhhdCB3ZSBoYXZlbid0IGRl dGVjdGVkIGl0KS4gQnV0IGFzDQo+ID4gc29vbg0KPiA+IGFzIHdlIGNvbmZpZ3VyZSBPU1BGIHRv IGFubm91bmNlIGEgZGVmYXVsdCBnYXRld2F5IHRvIHRoZSBpbnRlcm5hbA0KPiA+IFZMQU5zLCBz byB3ZSBnZXQgdHJhZmZpYyBmcm9tIHRlYW0wIHRvIHRlYW0xIHRoZSBsZWFraW5nIGJlZ2lucy4N Cj4gPiBTdG9wcGluZyB0aGUgT1NQRiBkZWZhdWx0IGdhdGV3YXkgYW5ub3VuY2VtZW50IGFnYWlu IGFsc28gc3RvcHMgdGhlDQo+ID4gbGVha2luZywgYnV0IGRvZXMgbm90IHJlbGVhc2UgYWxyZWFk eSBsZWFrZWQgbWVtb3J5Lg0KPiA+IA0KPiA+IFNvIHRoaXMgbGVhZHMgdG8gbWUgc3VzcGVjdCB0 aGF0IHRoZSBsZWFraW5nIGlzIHJlbGF0ZWQgdG8gUlggb24NCj4gPiB0ZWFtMA0KPiA+ICh3aGVy ZSBYTDcxMCBldGg5IGlzIG5vcm1hbGx5IHRoZSBvbmx5IGFjdGl2ZSBpbnRlcmZhY2UpIG9yIFRY IG9uDQo+ID4gdGVhbTENCj4gPiAoWDcxMCBldGgwLCBldGgxLCBldGg0LCBldGg1KS4gVGhlIGZp cnN0IGJhZCBjb21taXQgaXMgcmVsYXRlZCB0bw0KPiA+IFJYDQo+ID4gY2xlYW5pbmcsIHdoaWNo IHN1Z2dlc3RzIFJYIG9uIHRlYW0wLiBTaW5jZSB3ZSdyZSBvbmx5IHNlZWluZyB0aGUNCj4gPiBs ZWFrDQo+ID4gZm9yIG91ciBvdXRib3VuZCB0cmFmZmljLCBJIHN1c3BlY3QgZWl0aGVyIGEgZGlm ZmVyZW5jZSBiZXR3ZWVuIHRoZQ0KPiA+IFg3MTAgdnMuIFhMNzEwIE5JQ3MsIG9yIHRoYXQgdGhl IGluYm91bmQgdHJhZmZpYyBpcyBmb3IgcmVsYXRpdmVseQ0KPiA+IGZldw0KPiA+IGRlc3RpbmF0 aW9uIGFkZHJlc3NlcyAob25seSBvdXIgb3duIHN5c3RlbXMpIHdoaWxlIHRoZSBvdXRib3VuZA0K PiA+IHRyYWZmaWMNCj4gPiBpcyBmb3IgbWFueSBkaWZmZXJlbnQgYWRkcmVzc2VzIG9uIHRoZSBp bnRlcm5ldC4gQnV0IEknbSBqdXN0DQo+ID4gZ3Vlc3NpbmcNCj4gPiBoZXJlLg0KPiA+IA0KPiA+ IEkndmUgdHJpZWQga21lbWxlYWssIGJ1dCBpdCBvbmx5IGZvdW5kIGEgZmV3IGtCIG9mIHN1c3Bl Y3RlZCBtZW1vcnkNCj4gPiBsZWFrcyAoc2V2ZXJhbCBvZiB3aGljaCBkaXNhcHBlYXJlZCBhZ2Fp biBhZnRlciBhIHdoaWxlKS4NCj4gPiANCj4gPiBCZWxvdyBJJ3ZlIGluY2x1ZGVkIG1vcmUgZGV0 YWlscyAtIGdpdCBiaXNlY3QgbG9ncywgZXRodG9vbCAtaSwNCj4gPiBkbWVzZywNCj4gPiBLZXJu ZWwgLmNvbmZpZywgYW5kIHZhcmlvdXMgbWVtb3J5IHJlbGF0ZWQgL3Byb2MgZmlsZXMuIEFueSBo ZWxwIG9yDQo+ID4gc3VnZ2VzdGlvbnMgd291bGQgYmUgbXVjaCBhcHByZWNpYXRlZCwgYW5kIHBs ZWFzZSBsZXQgbWUga25vdyBpZg0KPiA+IG1vcmUNCj4gPiBpbmZvcm1hdGlvbiBpcyBuZWVkZWQg b3IgdGhlcmUncyBzb21ldGhpbmcgSSBzaG91bGQgdHJ5Lg0KPiA+IA0KPiA+IFJlZ2FyZHMsDQo+ ID4gQW5kZXJzIEsuIFBlZGVyc2VuDQo+ID4gDQo+IA0KPiBIaSBBbmRlcnMsDQo+IA0KPiBJIHRo aW5rIEkgc2VlIHRoZSBwcm9ibGVtIGFuZCBzaG91bGQgaGF2ZSBhIHBhdGNoIHN1Ym1pdHRlZCBz aG9ydGx5DQo+IHRvDQo+IGFkZHJlc3MgaXQuIEZyb20gd2hhdCBJIGNhbiB0ZWxsIGl0IGxvb2tz IGxpa2UgdGhlIGlzc3VlIGlzIHRoYXQgd2UNCj4gd2VyZW4ndCBwcm9wZXJseSByZWN5Y2xpbmcg dGhlIHBhZ2VzIGFzc29jaWF0ZWQgd2l0aCBkZXNjcmlwdG9ycyB0aGF0DQo+IGNvbnRhaW5lZCBh biBSeCBwcm9ncmFtbWluZyBzdGF0dXMuIEZvciBub3cgdGhlIHdvcmthcm91bmQgd291bGQgYmUN Cj4gdG8NCj4gdHJ5IGRpc2FibGluZyBBVFIgdmlhIHRoZSAiZXRodG9vbCAtLXNldC1wcml2LWZs YWdzIiBjb21tYW5kLiBJDQo+IHNob3VsZA0KPiBoYXZlIGEgcGF0Y2ggb3V0IGluIHRoZSBuZXh0 IGhvdXIgb3Igc28gdGhhdCB5b3UgY2FuIHRyeSB0ZXN0aW5nIHRvDQo+IHZlcmlmeSBpZiBpdCBh ZGRyZXNzZXMgdGhlIGlzc3VlLg0KPiANCj4gVGhhbmtzLg0KPiANCj4gLSBBbGV4DQoNClRoYW5r cyBBbGV4LA0KDQpJIHdpbGwgdGVzdCB0aGUgcGF0Y2ggaW4gb3VyIG5leHQgc2VydmljZSB3aW5k b3cgb24gVHVlc2RheSBtb3JuaW5nLg0KDQpSZWdhcmRzLA0KQW5kZXJz From mboxrd@z Thu Jan 1 00:00:00 1970 From: Anders K. Pedersen | Cohaesio Date: Thu, 5 Oct 2017 05:19:13 +0000 Subject: [Intel-wired-lan] Linux 4.12+ memory leak on router with i40e NICs In-Reply-To: References: <1507121766.30720.4.camel@cohaesio.com> Message-ID: <1507180753.20182.8.camel@cohaesio.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On ons, 2017-10-04 at 08:32 -0700, Alexander Duyck wrote: > On Wed, Oct 4, 2017 at 5:56 AM, Anders K. Pedersen | Cohaesio > wrote: > > Hello, > > > > After updating one of our Linux based routers to kernel 4.13 it > > began > > leaking memory quite fast (about 1 GB every half hour). To narrow > > we > > tried various kernel versions and found that 4.11.12 is okay, while > > 4.12 also leaks, so we did a bisection between 4.11 and 4.12. > > > > The first bisection ended at > > "[6964e53f55837b0c49ed60d36656d2e0ee4fc27b] i40e: fix handling of > > HW > > ATR eviction", which fixes some flag handling that was broken by > > 47994c119a36 "i40e: remove hw_disabled_flags in favor of using > > separate > > flag bits", so I did a second bisection, where I added 6964e53f5583 > > "i40e: fix handling of HW ATR eviction" to the steps that had > > 47994c119a36 "i40e: remove hw_disabled_flags in favor of using > > separate > > flag bits" in them. > > > > The second bisection ended at > > "[0e626ff7ccbfc43c6cc4aeea611c40b899682382] i40e: Fix support for > > flow > > director programming status", where I don't see any obvious > > problems, > > so I'm hoping for some assistance. > > > > The router is a PowerEdge R730 server (Haswell based) with three > > Intel > > NICs (all using the i40e driver): > > > > X710 quad port 10 GbE SFP+: eth0 eth1 eth2 eth3 > > X710 quad port 10 GbE SFP+: eth4 eth5 eth6 eth7 > > XL710 dual port 40 GbE QSFP+: eth8 eth9 > > > > The NICs are aggregated with LACP with the team driver: > > > > team0: eth9 (40 GbE selected primary), and eth3, eth7 (10 GbE non- > > selected backups) > > team1: eth0, eth1, eth4, eth5 (all 10 GbE selected) > > > > team0 is used for internal networks and has one untagged and four > > tagged VLAN interfaces, while team1 has an external uplink > > connection > > without any VLANs. > > > > The router runs an eBGP session on team1 to one of our uplinks, and > > iBGP via team0 to our other border routers. It also runs OSPF on > > the > > internal VLANs on team0. One thing I've noticed is that when OSPF > > is > > not announcing a default gateway to the internal networks, so there > > is > > almost no traffic coming in on team0 and out on team1, but still > > plenty > > of traffic coming in on team1 and out via team0, there's no memory > > leak > > (or at least it is so small that we haven't detected it). But as > > soon > > as we configure OSPF to announce a default gateway to the internal > > VLANs, so we get traffic from team0 to team1 the leaking begins. > > Stopping the OSPF default gateway announcement again also stops the > > leaking, but does not release already leaked memory. > > > > So this leads to me suspect that the leaking is related to RX on > > team0 > > (where XL710 eth9 is normally the only active interface) or TX on > > team1 > > (X710 eth0, eth1, eth4, eth5). The first bad commit is related to > > RX > > cleaning, which suggests RX on team0. Since we're only seeing the > > leak > > for our outbound traffic, I suspect either a difference between the > > X710 vs. XL710 NICs, or that the inbound traffic is for relatively > > few > > destination addresses (only our own systems) while the outbound > > traffic > > is for many different addresses on the internet. But I'm just > > guessing > > here. > > > > I've tried kmemleak, but it only found a few kB of suspected memory > > leaks (several of which disappeared again after a while). > > > > Below I've included more details - git bisect logs, ethtool -i, > > dmesg, > > Kernel .config, and various memory related /proc files. Any help or > > suggestions would be much appreciated, and please let me know if > > more > > information is needed or there's something I should try. > > > > Regards, > > Anders K. Pedersen > > > > Hi Anders, > > I think I see the problem and should have a patch submitted shortly > to > address it. From what I can tell it looks like the issue is that we > weren't properly recycling the pages associated with descriptors that > contained an Rx programming status. For now the workaround would be > to > try disabling ATR via the "ethtool --set-priv-flags" command. I > should > have a patch out in the next hour or so that you can try testing to > verify if it addresses the issue. > > Thanks. > > - Alex Thanks Alex, I will test the patch in our next service window on Tuesday morning. Regards, Anders