From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751251AbcGMMT3 (ORCPT ); Wed, 13 Jul 2016 08:19:29 -0400 Received: from mail-wm0-f46.google.com ([74.125.82.46]:37191 "EHLO mail-wm0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750978AbcGMMTU (ORCPT ); Wed, 13 Jul 2016 08:19:20 -0400 Date: Wed, 13 Jul 2016 14:18:28 +0200 From: Michal Hocko To: Matthias Dahl Cc: linux-raid@vger.kernel.org, linux-mm@kvack.org, dm-devel@redhat.com, linux-kernel@vger.kernel.org Subject: Re: Page Allocation Failures/OOM with dm-crypt on software RAID10 (Intel Rapid Storage) Message-ID: <20160713121828.GI28723@dhcp22.suse.cz> References: <02580b0a303da26b669b4a9892624b13@mail.ud19.udmedia.de> <20160712095013.GA14591@dhcp22.suse.cz> <20160712114920.GF14586@dhcp22.suse.cz> <20160712140715.GL14586@dhcp22.suse.cz> <459d501038de4d25db6d140ac5ea5f8d@mail.ud19.udmedia.de> <20160713112126.GH28723@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20160713112126.GH28723@dhcp22.suse.cz> User-Agent: Mutt/1.6.0 (2016-04-01) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed 13-07-16 13:21:26, Michal Hocko wrote: > On Tue 12-07-16 16:56:32, Matthias Dahl wrote: [...] > > If that support is baked into the Fedora provided kernel that is. If > > you could give me a few hints or pointers, how to properly do a allocator > > trace point and get some decent data out of it, that would be nice. > > You need to have a kernel with CONFIG_TRACEPOINTS and then enable them > via debugfs. You are interested in kmalloc tracepoint and specify a size > as a filter to only see those that are really interesting. I haven't > checked your slabinfo yet - hope to get to it later today. The largest contributors seem to be $ zcat slabinfo.txt.gz | awk '{printf "%s %d\n" , $1, $6*$15}' | head_and_tail.sh 133 | paste-with-diff.sh | sort -n -k3 initial diff [#pages] radix_tree_node 3444 2592 debug_objects_cache 388 46159 file_lock_ctx 114 138570 buffer_head 5616 238704 kmalloc-256 328 573164 bio-3 24 1118984 So it seems we are accumulating bios and 256B objects. Buffer heads as well but so much. Having over 4G worth of bios sounds really suspicious. Note that they pin pages to be written so this might be consuming the rest of the unaccounted memory! So the main question is why those bios do not get dispatched or finished. -- Michal Hocko SUSE Labs