From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1761579AbcLWCFv convert rfc822-to-8bit (ORCPT ); Thu, 22 Dec 2016 21:05:51 -0500 Received: from mail1.bemta8.messagelabs.com ([216.82.243.200]:38491 "EHLO mail1.bemta8.messagelabs.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752456AbcLWCFu (ORCPT ); Thu, 22 Dec 2016 21:05:50 -0500 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprCKsWRWlGSWpSXmKPExsWS8eIhk25VW0y EweJHmhZPP/WxWFzeNYfN4t6a/6wWr78tY7Y43nuAyYHVY8GmUo/NK7Q8Nq3qZPPY9GkSu8fn TXIBrFGsmXlJ+RUJrBlvD19lK+gwqvjRWdnAOEG9i5GTQ0jgCaPEpB92XYxcQPZCRon+JSdYu xg5ONgE1CV+n+ADqRER8JL4uusMG4jNLDCBUeL1UlEQW1ggTOLNmidsEDXhEgsW3meHsN0k5v YtBbNZBFQl3nw+xQpi8wLNaercxgax6y6jxOIHa5lAEpwCdhIfDlwHsxkFZCWmPbrPBLFMXGL utFlgzRICAhJL9pxnhrBFJV4+/gd2p4SAvMSWWYIQ5ToSC3Z/grpTW2LZwtfMEHsFJU7OfMIy gVFkFpKps5C0zELSMgtJywJGllWM6sWpRWWpRboWeklFmekZJbmJmTm6hgYWermpxcWJ6ak5i UnFesn5uZsYgbHFAAQ7GA83Ox9ilORgUhLlveASEyHEl5SfUpmRWJwRX1Sak1p8iFGGg0NJgr ewFSgnWJSanlqRlpkDjHKYtAQHj5II74sWoDRvcUFibnFmOkTqFKOilDivPkifAEgiozQPrg2 WWC4xykoJ8zICHSLEU5BalJtZgir/ilGcg1FJmDcVZApPZl4J3PRXQIuZgBbbNkeDLC5JREhJ NTB2bt+1as7ldY6P8nKuKC4sDvn44ZrVj7YvP7Y8l036f+pB5+pYN34eMSvLVoabfrfWZ+yYm fO5kanuw6RdLEc5d8je17/RF2+z3mNSw5X4jk7P0yErpBl2Sk5yeBElkXLhUekfyd9+/NLZn9 8WPMi4+vvus9S+M3lNP2tkYzmici7MseXz0uhUYinOSDTUYi4qTgQAXVTGOycDAAA= X-Env-Sender: caods1@lenovo.com X-Msg-Ref: server-5.tower-144.messagelabs.com!1482458746!3483927!1 X-Originating-IP: [104.232.225.2] X-StarScan-Received: X-StarScan-Version: 9.1.1; banners=-,-,- X-VirusChecked: Checked From: Dashi DS1 Cao To: Hugh Dickins , Peter Zijlstra CC: Michal Hocko , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" Subject: RE: A small window for a race condition in mm/rmap.c:page_lock_anon_vma_read Thread-Topic: A small window for a race condition in mm/rmap.c:page_lock_anon_vma_read Thread-Index: AdJaoiBI0AeNZbJZSQCgF3f04k/2ogAs3HmAADB0LAAAEgMEAAAXeGgQ Date: Fri, 23 Dec 2016 02:02:14 +0000 Message-ID: <23B7B563BA4E9446B962B142C86EF24ADBF309@CNMAILEX03.lenovo.com> References: <23B7B563BA4E9446B962B142C86EF24ADBD62C@CNMAILEX03.lenovo.com> <20161221144343.GD593@dhcp22.suse.cz> <20161222135106.GY3124@twins.programming.kicks-ass.net> In-Reply-To: Accept-Language: zh-CN, en-US Content-Language: zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.96.19.89] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org The kernel version is "RELEASE: 3.10.0-327.36.3.el7.x86_64". It was the latest kernel release of CentOS 7.2 at that time, or maybe still now. I've tried to print the value of anon_vma from other three dumps, but the content is not available in the dump. "gdb: page excluded: kernel virtual address: ffff882b47ddadc0" I guess it is not copied out because it has already been put into some unused list. Dashi Cao -----Original Message----- From: Hugh Dickins [mailto:hughd@google.com] Sent: Friday, December 23, 2016 6:27 AM To: Peter Zijlstra Cc: Michal Hocko ; Dashi DS1 Cao ; linux-mm@kvack.org; linux-kernel@vger.kernel.org; Hugh Dickins Subject: Re: A small window for a race condition in mm/rmap.c:page_lock_anon_vma_read On Thu, 22 Dec 2016, Peter Zijlstra wrote: > On Wed, Dec 21, 2016 at 03:43:43PM +0100, Michal Hocko wrote: > > anon_vma locking is clever^Wsubtle as hell. CC Peter... > > > > On Tue 20-12-16 09:32:27, Dashi DS1 Cao wrote: > > > I've collected four crash dumps with similar backtrace. > > > > > > PID: 247 TASK: ffff881fcfad8000 CPU: 14 COMMAND: "kswapd1" > > > #0 [ffff881fcfad7978] machine_kexec at ffffffff81051e9b > > > #1 [ffff881fcfad79d8] crash_kexec at ffffffff810f27e2 > > > #2 [ffff881fcfad7aa8] oops_end at ffffffff8163f448 > > > #3 [ffff881fcfad7ad0] die at ffffffff8101859b > > > #4 [ffff881fcfad7b00] do_general_protection at ffffffff8163ed3e > > > #5 [ffff881fcfad7b30] general_protection at ffffffff8163e5e8 > > > [exception RIP: down_read_trylock+9] > > > RIP: ffffffff810aa9f9 RSP: ffff881fcfad7be0 RFLAGS: 00010286 > > > RAX: 0000000000000000 RBX: ffff882b47ddadc0 RCX: 0000000000000000 > > > RDX: 0000000000000000 RSI: 0000000000000000 RDI: > > > 91550b2b32f5a3e8 > > > > rdi is obviously a mess - smells like a string. So either sombody > > has overwritten root_anon_vma or this is really a use after free... > > e8 - ??? > a3 - ??? > f5 - ??? > 32 - 2 > 2b - + > b - > > 55 - U > 91 - ??? > > Not a string.. > > > > RBP: ffff881fcfad7be0 R8: ffffea00ecc28860 R9: ffff883fcffeae28 > > > R10: ffffffff81a691a0 R11: 0000000000000001 R12: ffff882b47ddadc1 > > > R13: ffffea00ecc28840 R14: 91550b2b32f5a3e8 R15: ffffea00ecc28840 > > > ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0000 > > > #6 [ffff881fcfad7be8] page_lock_anon_vma_read at ffffffff811a3365 > > > #7 [ffff881fcfad7c18] page_referenced at ffffffff811a35e7 > > > #8 [ffff881fcfad7c90] shrink_active_list at ffffffff8117e8cc > > > #9 [ffff881fcfad7d48] balance_pgdat at ffffffff81180288 > > > #10 [ffff881fcfad7e20] kswapd at ffffffff81180813 > > > #11 [ffff881fcfad7ec8] kthread at ffffffff810a5b8f > > > #12 [ffff881fcfad7f50] ret_from_fork at ffffffff81646a98 > > > > > > I suspect my customer hits into a small window of a race condition in mm/rmap.c: page_lock_anon_vma_read. > > > struct anon_vma *page_lock_anon_vma_read(struct page *page) { > > > struct anon_vma *anon_vma = NULL; > > > struct anon_vma *root_anon_vma; > > > unsigned long anon_mapping; > > > > > > rcu_read_lock(); > > > anon_mapping = (unsigned long)READ_ONCE(page->mapping); > > > if ((anon_mapping & PAGE_MAPPING_FLAGS) != PAGE_MAPPING_ANON) > > > goto out; > > > if (!page_mapped(page)) > > > goto out; > > > > > > anon_vma = (struct anon_vma *) (anon_mapping - PAGE_MAPPING_ANON); > > > root_anon_vma = READ_ONCE(anon_vma->root); > > > > Could you dump the anon_vma and struct page as well? > > > > > if (down_read_trylock(&root_anon_vma->rwsem)) { > > > /* > > > * If the page is still mapped, then this anon_vma is still > > > * its anon_vma, and holding the mutex ensures that it will > > > * not go away, see anon_vma_free(). > > > */ > > > if (!page_mapped(page)) { > > > up_read(&root_anon_vma->rwsem); > > > anon_vma = NULL; > > > } > > > goto out; > > > } > > > ... > > > } > > > > > > Between the time the two "page_mapped(page)" are checked, the > > > address (anon_mapping - PAGE_MAPPING_ANON) is unmapped! However it > > > seems that anon_vma->root could still be read in but the value is > > > wild. So the kernel crashes in down_read_trylock. But it's weird > > > that all the "struct page" has its member "_mapcount" still with > > > value 0, not -1, in the four crashes. > > So the point is that while we hold rcu_read_lock() the actual memory > backing the anon_vmas cannot be freed. It can be reused, but only for > another anon_vma. > > Now, anon_vma_alloc() sets ->root to self, while anon_vma_free() > leaves > ->root set to whatever. And any other ->root assignment is to a valid > anon_vma. > > Therefore, the same rules that ensure anon_vma stays valid, should > also ensure anon_vma->root stays valid. > > Now, one thing that might go wobbly is that ->root assignments are not > done using WRITE_ONCE(), this means a naughty compiler can miscompile > those stores and introduce store-tearing, if our READ_ONCE() would > observe such a tear, we'd be up some creek without a paddle. We would indeed. And this being the season of goodwill, I'm biting my tongue not to say what I think of the prospect of store tearing. But that zeroed anon_vma implies tearing not the problem here anyway. > > Now, its been a long time since I looked at any of this code, and I > see that Hugh has fixed at least two wobblies in my original code. Nothing much, and this (admittedly subtle) technique has been working well for years, so I'm sceptical about "a small window for a race condition". But Dashi's right to point out that the struct page has _mapcount 0 (not -1 for logical 0) in these cases: it looks as if something is freeing (or corrupting) the anon_vma despite it still having pages mapped, or something is misaccounting (or corrupting) the _mapcount. But I've no idea what, and we have not heard such reports elsewhere. We don't even know what kernel this is - something special, perhaps? Hugh