From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751664AbcBLEEt (ORCPT ); Thu, 11 Feb 2016 23:04:49 -0500 Received: from e38.co.us.ibm.com ([32.97.110.159]:52635 "EHLO e38.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751519AbcBLEEr (ORCPT ); Thu, 11 Feb 2016 23:04:47 -0500 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: aneesh.kumar@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org;linux-s390@vger.kernel.org From: "Aneesh Kumar K.V" To: Gerald Schaefer , "Kirill A. Shutemov" Cc: "Kirill A. Shutemov" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton , Linus Torvalds , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Martin Schwidefsky , Heiko Carstens , linux-s390@vger.kernel.org, Sebastian Ott Subject: Re: [BUG] random kernel crashes after THP rework on s390 (maybe also on PowerPC and ARM) In-Reply-To: <20160211205702.24f0d17a@thinkpad> References: <20160211192223.4b517057@thinkpad> <20160211190942.GA10244@node.shutemov.name> <20160211205702.24f0d17a@thinkpad> User-Agent: Notmuch/0.20.2 (http://notmuchmail.org) Emacs/24.5.1 (x86_64-pc-linux-gnu) Date: Fri, 12 Feb 2016 09:34:33 +0530 Message-ID: <87a8n6shf2.fsf@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16021204-0029-0000-0000-0000107153B1 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Gerald Schaefer writes: > On Thu, 11 Feb 2016 21:09:42 +0200 > "Kirill A. Shutemov" wrote: > >> On Thu, Feb 11, 2016 at 07:22:23PM +0100, Gerald Schaefer wrote: >> > Hi, >> > >> > Sebastian Ott reported random kernel crashes beginning with v4.5-rc1 and >> > he also bisected this to commit 61f5d698 "mm: re-enable THP". Further >> > review of the THP rework patches, which cannot be bisected, revealed >> > commit fecffad "s390, thp: remove infrastructure for handling splitting PMDs" >> > (and also similar commits for other archs). >> > >> > This commit removes the THP splitting bit and also the architecture >> > implementation of pmdp_splitting_flush(), which took care of the IPI for >> > fast_gup serialization. The commit message says >> > >> > pmdp_splitting_flush() is not needed too: on splitting PMD we will do >> > pmdp_clear_flush() + set_pte_at(). pmdp_clear_flush() will do IPI as >> > needed for fast_gup >> > >> > The assumption that a TLB flush will also produce an IPI is wrong on s390, >> > and maybe also on other architectures, and I thought that this was actually >> > the main reason for having an arch-specific pmdp_splitting_flush(). >> > >> > At least PowerPC and ARM also had an individual implementation of >> > pmdp_splitting_flush() that used kick_all_cpus_sync() instead of a TLB >> > flush to send the IPI, and those were also removed. Putting the arch >> > maintainers and mailing lists on cc to verify. >> > >> > On s390 this will break the IPI serialization against fast_gup, which >> > would certainly explain the random kernel crashes, please revert or fix >> > the pmdp_splitting_flush() removal. >> >> Sorry for that. >> >> I believe, the problem was already addressed for PowerPC: >> >> http://lkml.kernel.org/g/454980831-16631-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com >> >> I think kick_all_cpus_sync() in arch-specific pmdp_invalidate() would do >> the trick, right? > > Hmm, not sure about that. After pmdp_invalidate(), a pmd_none() check in > fast_gup will still return false, because the pmd is not empty (at least > on s390). Why can't we do this ? I did this for ppc64. void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, 0); + pmd_hugepage_update(vma->vm_mm, address, pmdp, ~0UL, 0); >So I don't see spontaneously how it will help fast_gup to break > out to the slow path in case of THP splitting. > >> >> If yes, I'll prepare patch tomorrow (some sleep required). >> > > We'll check if adding kick_all_cpus_sync() to pmdp_invalidate() helps. > It would also be good if Martin has a look at this, he'll return on > Monday. -aneesh From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-ob0-f173.google.com (mail-ob0-f173.google.com [209.85.214.173]) by kanga.kvack.org (Postfix) with ESMTP id 88D046B0005 for ; Thu, 11 Feb 2016 23:04:46 -0500 (EST) Received: by mail-ob0-f173.google.com with SMTP id wb13so105818786obb.1 for ; Thu, 11 Feb 2016 20:04:46 -0800 (PST) Received: from e38.co.us.ibm.com (e38.co.us.ibm.com. [32.97.110.159]) by mx.google.com with ESMTPS id z127si954360oif.103.2016.02.11.20.04.45 for (version=TLS1_2 cipher=AES128-SHA bits=128/128); Thu, 11 Feb 2016 20:04:45 -0800 (PST) Received: from localhost by e38.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 11 Feb 2016 21:04:45 -0700 Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id D3EA33E40044 for ; Thu, 11 Feb 2016 21:04:42 -0700 (MST) Received: from d03av01.boulder.ibm.com (d03av01.boulder.ibm.com [9.17.195.167]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id u1C44gKk27328588 for ; Thu, 11 Feb 2016 21:04:42 -0700 Received: from d03av01.boulder.ibm.com (localhost [127.0.0.1]) by d03av01.boulder.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id u1C44fsh012199 for ; Thu, 11 Feb 2016 21:04:42 -0700 From: "Aneesh Kumar K.V" Subject: Re: [BUG] random kernel crashes after THP rework on s390 (maybe also on PowerPC and ARM) In-Reply-To: <20160211205702.24f0d17a@thinkpad> References: <20160211192223.4b517057@thinkpad> <20160211190942.GA10244@node.shutemov.name> <20160211205702.24f0d17a@thinkpad> Date: Fri, 12 Feb 2016 09:34:33 +0530 Message-ID: <87a8n6shf2.fsf@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Gerald Schaefer , "Kirill A. Shutemov" Cc: "Kirill A. Shutemov" , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton , Linus Torvalds , Michael Ellerman , Benjamin Herrenschmidt , Paul Mackerras , linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Martin Schwidefsky , Heiko Carstens , linux-s390@vger.kernel.org, Sebastian Ott Gerald Schaefer writes: > On Thu, 11 Feb 2016 21:09:42 +0200 > "Kirill A. Shutemov" wrote: > >> On Thu, Feb 11, 2016 at 07:22:23PM +0100, Gerald Schaefer wrote: >> > Hi, >> > >> > Sebastian Ott reported random kernel crashes beginning with v4.5-rc1 and >> > he also bisected this to commit 61f5d698 "mm: re-enable THP". Further >> > review of the THP rework patches, which cannot be bisected, revealed >> > commit fecffad "s390, thp: remove infrastructure for handling splitting PMDs" >> > (and also similar commits for other archs). >> > >> > This commit removes the THP splitting bit and also the architecture >> > implementation of pmdp_splitting_flush(), which took care of the IPI for >> > fast_gup serialization. The commit message says >> > >> > pmdp_splitting_flush() is not needed too: on splitting PMD we will do >> > pmdp_clear_flush() + set_pte_at(). pmdp_clear_flush() will do IPI as >> > needed for fast_gup >> > >> > The assumption that a TLB flush will also produce an IPI is wrong on s390, >> > and maybe also on other architectures, and I thought that this was actually >> > the main reason for having an arch-specific pmdp_splitting_flush(). >> > >> > At least PowerPC and ARM also had an individual implementation of >> > pmdp_splitting_flush() that used kick_all_cpus_sync() instead of a TLB >> > flush to send the IPI, and those were also removed. Putting the arch >> > maintainers and mailing lists on cc to verify. >> > >> > On s390 this will break the IPI serialization against fast_gup, which >> > would certainly explain the random kernel crashes, please revert or fix >> > the pmdp_splitting_flush() removal. >> >> Sorry for that. >> >> I believe, the problem was already addressed for PowerPC: >> >> http://lkml.kernel.org/g/454980831-16631-1-git-send-email-aneesh.kumar@linux.vnet.ibm.com >> >> I think kick_all_cpus_sync() in arch-specific pmdp_invalidate() would do >> the trick, right? > > Hmm, not sure about that. After pmdp_invalidate(), a pmd_none() check in > fast_gup will still return false, because the pmd is not empty (at least > on s390). Why can't we do this ? I did this for ppc64. void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, 0); + pmd_hugepage_update(vma->vm_mm, address, pmdp, ~0UL, 0); >So I don't see spontaneously how it will help fast_gup to break > out to the slow path in case of THP splitting. > >> >> If yes, I'll prepare patch tomorrow (some sleep required). >> > > We'll check if adding kick_all_cpus_sync() to pmdp_invalidate() helps. > It would also be good if Martin has a look at this, he'll return on > Monday. -aneesh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: aneesh.kumar@linux.vnet.ibm.com (Aneesh Kumar K.V) Date: Fri, 12 Feb 2016 09:34:33 +0530 Subject: [BUG] random kernel crashes after THP rework on s390 (maybe also on PowerPC and ARM) In-Reply-To: <20160211205702.24f0d17a@thinkpad> References: <20160211192223.4b517057@thinkpad> <20160211190942.GA10244@node.shutemov.name> <20160211205702.24f0d17a@thinkpad> Message-ID: <87a8n6shf2.fsf@linux.vnet.ibm.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Gerald Schaefer writes: > On Thu, 11 Feb 2016 21:09:42 +0200 > "Kirill A. Shutemov" wrote: > >> On Thu, Feb 11, 2016 at 07:22:23PM +0100, Gerald Schaefer wrote: >> > Hi, >> > >> > Sebastian Ott reported random kernel crashes beginning with v4.5-rc1 and >> > he also bisected this to commit 61f5d698 "mm: re-enable THP". Further >> > review of the THP rework patches, which cannot be bisected, revealed >> > commit fecffad "s390, thp: remove infrastructure for handling splitting PMDs" >> > (and also similar commits for other archs). >> > >> > This commit removes the THP splitting bit and also the architecture >> > implementation of pmdp_splitting_flush(), which took care of the IPI for >> > fast_gup serialization. The commit message says >> > >> > pmdp_splitting_flush() is not needed too: on splitting PMD we will do >> > pmdp_clear_flush() + set_pte_at(). pmdp_clear_flush() will do IPI as >> > needed for fast_gup >> > >> > The assumption that a TLB flush will also produce an IPI is wrong on s390, >> > and maybe also on other architectures, and I thought that this was actually >> > the main reason for having an arch-specific pmdp_splitting_flush(). >> > >> > At least PowerPC and ARM also had an individual implementation of >> > pmdp_splitting_flush() that used kick_all_cpus_sync() instead of a TLB >> > flush to send the IPI, and those were also removed. Putting the arch >> > maintainers and mailing lists on cc to verify. >> > >> > On s390 this will break the IPI serialization against fast_gup, which >> > would certainly explain the random kernel crashes, please revert or fix >> > the pmdp_splitting_flush() removal. >> >> Sorry for that. >> >> I believe, the problem was already addressed for PowerPC: >> >> http://lkml.kernel.org/g/454980831-16631-1-git-send-email-aneesh.kumar at linux.vnet.ibm.com >> >> I think kick_all_cpus_sync() in arch-specific pmdp_invalidate() would do >> the trick, right? > > Hmm, not sure about that. After pmdp_invalidate(), a pmd_none() check in > fast_gup will still return false, because the pmd is not empty (at least > on s390). Why can't we do this ? I did this for ppc64. void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pmd_hugepage_update(vma->vm_mm, address, pmdp, _PAGE_PRESENT, 0); + pmd_hugepage_update(vma->vm_mm, address, pmdp, ~0UL, 0); >So I don't see spontaneously how it will help fast_gup to break > out to the slow path in case of THP splitting. > >> >> If yes, I'll prepare patch tomorrow (some sleep required). >> > > We'll check if adding kick_all_cpus_sync() to pmdp_invalidate() helps. > It would also be good if Martin has a look at this, he'll return on > Monday. -aneesh